1
$\begingroup$

Can we assume that a 7gb llm model knows 7gb of text info? (by 7gb i mean a 7 billion parameter model at Q8 quantization) or a (1.75 billion parameter model at full fp32 precision)

For example, a 70,000-word novel with no formatting would be a little over 0.4 MB. A 500-page novel in plain text will be 1 MB.

can we assume that a 7gb model has the info of approx 7k books?

Now there are more than one ways to write a book;) for something technical. it could be written to the point or the author could just beat around the bush.

So in the end it comes down to the quality of data.

Does anyone have expertise in information theory to answer this question in a deeper way? like is there deep level compression of data going on such that a 7gb llm "knows" or "can know" more than 7gb of information if trained on more data?

$\endgroup$
1
  • 1
    $\begingroup$ Comprehending a novel assumes that you can already read the language that the novel is written in, and you also have a bunch of assumed knowledge that you've acquired through your country's education system (or equivalent if self-taught) and existing in a human body and autonomously navigating the natural and human-created world for many years. No history book ever explicitly wrote down that whenever Abraham Lincoln was in Springfield, so was his left big toe. $\endgroup$
    – Pseudonym
    Commented May 21 at 3:05

2 Answers 2

2
$\begingroup$

A recent study "Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws" [1] examines how much knowledge an LLM can be trained to represent by training many language models of the GPT-2 and LLaMA architectures. The dataset used in the training is specifically constructed, so the intrinsic information within the dataset can be calculated precisely (more precisely, the entropy of the dataset construction method's randomness).

The authors show that the information capacity of an LLM increases linearly with the number of parameters, and more than two bits per parameter can be represented. Here, "two bits per parameter" is not a magic number but rather a proposed lower bound. The proportional constants are significantly different between the architectures and settings used in the experiments.

However, note that, an LLM can be trained to represent a such large information does not mean the LLMs used in practice have been trained to the same degree. In the experiments, LLMs are exposed to 1000 repetitions of the same information to achieve more than 2 bits per parameter capacity.

Moreover, one typically wants to train an LLM to be able to "reason" over the information contained in the dataset, not just memorizing the text as-is. It is an open problem how to achieve such goal efficiently. One approach is to use synthetic "textbooks" [2].

$\endgroup$
2
$\begingroup$

No. There is no such thing as a "7gb llm". You are probably thinking of models that are labelled as 7B -- for example, Llama 3 7B. The "7B" means the model has 7 billion parameters, not that it "knows 7gb of text info".

Representing the weights/parameters of the model probably takes about 4*7 billion bytes = 28GB, as each parameter takes about 4 bytes. There are various ways to quantize the parameters and store the quantized parameter in less than 28GB (at some loss in quality of output of the LLM).

It doesn't "contain text info", and it's not clear how one would define precisely what it means to "know 7gb of text info". In a likelihood, most such models have been trained on much more than 7 billion characters of text (not a fundamental requirement but just an empirical observation). There is no clear way to define what you mean by "has the info of n books", so I don't think that is answerable.

A LLM can be thought of as predicting the next word in a sequence in words, which is not exactly the same as compression, though it is certainly related.

$\endgroup$
2
  • $\begingroup$ By 7gb i meant a 7b model at Q8 quantization. but never mind. let's take your point that a 7b model in fp32 precision would take 28gb. can it be thought of as knowing 28 gb of text info? or more? $\endgroup$
    – Zenytt
    Commented May 20 at 23:09
  • 1
    $\begingroup$ @Zenytt, I already answered that. As I wrote, it's not clear how one would define precisely what it means to "know 7gb of text info". $\endgroup$
    – D.W.
    Commented May 21 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.