Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.

Say we have already computed values for the optimal $Q$-function: $Q^*(s, a) = \max\limits_{\pi} Q^\pi(s, a)$. Why exactly does $\pi(s) = \arg\max\limits_{a \in A} Q^*(s, a)$ constitute an optimal policy? This seems like an obviously true statement (it probably is) but I can't quite convince myself of its truth.

The reason I have doubts is that although $\pi(s)$ maximizes $Q^*(s, \cdot)$ and $\max\limits_a Q^*(s, a) = V^*(s)$, I don't see why $V^\pi(s) = Q^\pi(s, \pi(s))$ should equal $\max\limits_a Q^*(s, a)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.