I am currently working my way through Language Models are Few-Shot Learners , the initial 75-page paper about GPT-3, the language learning model spawning off into ChatGTP.

In it, they mention several times that they are using 175 billion parameters, orders of magnitudes more than previous experiments by others. They show this table, for 8 models ranging from 125 million params to 175 billion params.

enter image description here

Then they say:

Table 2.1 shows the sizes and architectures of our 8 models. Here nparams is the total number of trainable parameters, nlayers is the total number of layers, dmodel is the number of units in each bottleneck layer (we always have the feedforward layer four times the size of the bottleneck layer, dff = 4 ∗ dmodel), and dhead is the dimension of each attention head. All models use a context window of nctx = 2048 tokens. We partition the model across GPUs along both the depth and width dimension in order to minimize data-transfer between nodes. The precise architectural parameters for each model are chosen based on computational efficiency and load-balancing in the layout of models across GPU’s. Previous work [KMH+20] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range.

I am not an expert in machine learning, I just know basic RNNs and how they work with just a few parameters and a few layers (I don't know, like 5 parameters and 5 layers max, it's been a while)? What are the things counted as parameters in this 175 billion parameter network? How does the network look with its 96 layers? How many nodes are there per layer sort of thing?

I am trying to understand this paper and eventually how ChatGPT works, and getting to section 2 so far, I haven't seen what you would use as inputs/parameters to such a large model. The ones you learn in school are tiny compared to this. Hoping for a little illumination on what could be going on.


1 Answer 1


The 175 billion parameters in the GPT-3 language model are values that are used by the model to make predictions about the next word or words in a sentence or piece of text. These parameters are essentially the weights that are applied to the input data in order to make the model's predictions. In a neural network, the parameters are the values that are learned and adjusted during the training process in order to minimize the difference between the predicted output and the desired output.

The GPT-3 model has 96 layers, which means that it is composed of multiple layers of neural networks. Each layer of the network is made up of a number of nodes, which are the individual processing units of the network. The number of nodes per layer can vary.

To use the GPT-3 model, you would need to provide it with some input data, such as a sentence or a paragraph of text. The model would then process this input using its 175 billion parameters and its 96 layers, in order to make a prediction about the next word or words that should come next in the text. The model's predictions would be based on the input data and its learned parameters, and it would be able to generate human-like text as a result.

  • $\begingroup$ You may want to add that even in the basic Neural Network, usually full connectivity between the nodes in subsequent layers is assumed, meaning that there are n x m parameters between two layers of sizes n and m. The question seemed to have misunderstood this when 5 layers only supposed to have 5 parameters. This should given an indication as to why the number of parameters can easily explode in large networks. $\endgroup$ Commented Dec 19, 2022 at 8:33
  • 3
    $\begingroup$ Did GPT-3 write this? $\endgroup$
    – benrg
    Commented Dec 19, 2022 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.