Efficient Large Language Models

"This project will explore more novel directions in improving the capability of LLMs in resource constrained settings. The general background to this is that LLMs are being used for two basic functions within a single model. They define a solid language model than understands how to express information in excellent human-readable format; they also provide a knowledge store that encodes real knowledge about many things.

These are both done together within a single framework. LLMs have universally taken the form of stacked transformer models. The basis for that construction is simply none other than that "it worked". There is no basis for expecting that this is in any way optimal, sensible or even considered.

In this project, we expect to consolidate a number of different directions: understanding LLM computation better, reducing LLM computation, exploring novel language model computational paradigms, factoring knowledge, semantics and syntactics, improving learning, and generally democratising LLMs: making them more amenable to use in individualised settings. A number of stages of this project will be:

1) Analysis and simplification of the mathematical structure of the transformer. 2) Distilling state space models from learnt transformer models. 3) Considering other approaches for distilling efficient LLMs: hashing, cascaded computation. 4) Considering decomposing LLMs into semantic, syntactic and knowledge components.

A significant challenge of all of these will be evaluation metrics within a context of limited compute capability for training models. Hence the first direction of this project will be to establish good evaluation metrics for this work."