# Why GPT model is a higher order hidden markov model

I have read the GPT-1 paper, and my understanding is that it works as follows: $$U$$ is input tokens, $$h_0=UW_e+W_p$$, $$h_i=\text{transformer_block}(h_{i-1})$$ and the output is a probability vector $$P(u)=\text{softmax}(h_nW_e^T)$$, and the hidden states are the final latent layer before softmax, so the hidden states are $$h_n(U)$$? I do not really understand, how to formalise this as a hidden Markov chain similar to the definition in Wikipedia?

The statement that GPT is a higher order hidden Markov model should not be taken too seriously. It is intended as a slogan or intuition, not something that is intended to be be rigorously proven.

Let $$x_1,x_2,\dots,x_k$$ denote the words of the text, in order ($$x_1$$ is the first word, etc.).

GPT is a model for predicting the next word, given all prior words. In other words, it estimates $$p(x_i | x_1,x_2,\dots,x_{i-1})$$. Hopefully you can see how this resembles the formulation of a Markov model.

It is higher-order, because it uses all prior words to predict the next word, not just the previous few words.

• Thank you. It is about my summer research, so I think is should be taken seriously. I can see the higher order Markov model structure, but I do not understand the hidden state.
– 123
Jun 28, 2023 at 19:58
• @123, I am not convinced that there is any hidden state, and I am not convinced that you should take the "hidden" part of that statement that seriously. Without any context about where you have encountered that statement, I find it hard to say any more.
– D.W.
Jun 28, 2023 at 22:16