I have read the GPT-1 paper, and my understanding is that it works as follows: $U$ is input tokens, $h_0=UW_e+W_p$, $h_i=\text{transformer_block}(h_{i-1})$ and the output is a probability vector $P(u)=\text{softmax}(h_nW_e^T)$, and the hidden states are the final latent layer before softmax, so the hidden states are $h_n(U)$? I do not really understand, how to formalise this as a hidden Markov chain similar to the definition in Wikipedia?


1 Answer 1


The statement that GPT is a higher order hidden Markov model should not be taken too seriously. It is intended as a slogan or intuition, not something that is intended to be be rigorously proven.

Let $x_1,x_2,\dots,x_k$ denote the words of the text, in order ($x_1$ is the first word, etc.).

GPT is a model for predicting the next word, given all prior words. In other words, it estimates $p(x_i | x_1,x_2,\dots,x_{i-1})$. Hopefully you can see how this resembles the formulation of a Markov model.

It is higher-order, because it uses all prior words to predict the next word, not just the previous few words.

  • $\begingroup$ Thank you. It is about my summer research, so I think is should be taken seriously. I can see the higher order Markov model structure, but I do not understand the hidden state. $\endgroup$
    – 123
    Jun 28, 2023 at 19:58
  • $\begingroup$ @123, I am not convinced that there is any hidden state, and I am not convinced that you should take the "hidden" part of that statement that seriously. Without any context about where you have encountered that statement, I find it hard to say any more. $\endgroup$
    – D.W.
    Jun 28, 2023 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.