I am studying from the MARL textbook by Albrecht, Christianos and Schäfer. They define a stochastic game in Sec 3.3 as the multi-agent version of an MDP. In Fig 3.3 (pg 50) they give an intuition for why a stochastic game can be thought of as a sequence of states, where each state represents a non-repeated normal form game (i.e. a bandit problem but multi-agent).
The implication here is that a Nash equilibrium joint policy $\pi^\star$ for this game need only depend on the current state $s^t$, the policy doesn't need to map from the history of previous joint actions up to that point $h^t$ [call this proposition A]. I am struggling a bit with the intuition for prop A.
I understand why this is the case for an optimal policy in an MDP. There's only one agent, and the Markov dynamics imply that $R(a^t \mid s^t, h^t) = R(a^t \mid s^t)$. But in the multi-agent case, there is the additional nuance of signalling cooperation strategies based on past actions.
E.g., I can condition my policy in $s^t$ on whether the other player cooperated with me in $s^{t-1}$. This is after all the reason why the set of equilibria in a normal form game like Prisoner's Dilemma when the game is non-repeated is different from when the PD game is repeated infinitely. So why doesn't this logic apply to stochastic game with multiple states?
Is there some unstated condition for prop A to hold? For instance, that the stochastic game's state transitions should be a finite DAG with a terminal state? Does this then exclude games with cycles (even ones with a time discount factor $\gamma$)?
