When applied to domains that are episodic and have a "final" state but no final action, like a game, how does SARSA update the Q-values?

e.g. A game agent would receive this series: $$ s_0,a_0,r_0,s_1,a_1,r_1,.....s_{N-1},a_{N-1},r_{N-1},s_N $$ Based on the traditional definition, which updates the current Q-value using the *future Q-value, shouldn't it be impossible to apply SARSA, since there's no future action to use when plugging in the $(s_t, a_t, r_t, s_{t+1}, a_{t+1})$ values?

  • 1
    $\begingroup$ Welcome to Computer Science! Note that you can use LaTeX here to typeset mathematics in a more readable way. See here for a short introduction. $\endgroup$ – FrankW Oct 30 '14 at 5:53

You can think of the terminal state as a loop for every possible action where you receive zero reward. Once you enter the terminal state, not matter what action you choose, you will receive zero reward.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.