In it, they mention several times that they are using 175 billion parameters, orders of magnitudes more than previous experiments by others. They show this table, for 8 models ranging from 125 million params to 175 billion params.
Then they say:
Table 2.1 shows the sizes and architectures of our 8 models. Here nparams is the total number of trainable parameters, nlayers is the total number of layers, dmodel is the number of units in each bottleneck layer (we always have the feedforward layer four times the size of the bottleneck layer, dff = 4 ∗ dmodel), and dhead is the dimension of each attention head. All models use a context window of nctx = 2048 tokens. We partition the model across GPUs along both the depth and width dimension in order to minimize data-transfer between nodes. The precise architectural parameters for each model are chosen based on computational efficiency and load-balancing in the layout of models across GPU’s. Previous work [KMH+20] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range.
I am not an expert in machine learning, I just know basic RNNs and how they work with just a few parameters and a few layers (I don't know, like 5 parameters and 5 layers max, it's been a while)? What are the things counted as parameters in this 175 billion parameter network? How does the network look with its 96 layers? How many nodes are there per layer sort of thing?
I am trying to understand this paper and eventually how ChatGPT works, and getting to section 2 so far, I haven't seen what you would use as inputs/parameters to such a large model. The ones you learn in school are tiny compared to this. Hoping for a little illumination on what could be going on.