consider the policy iteration algorithm for a finite state MDP. Suppose the initial policy is a stochastic policy. Now, can the optimal policy be deterministic after improvements ? Or, can we say that always the optimal policy will be a stochastic one ? Confused about this. Any ideas will be helpful. The reason I am asking this question is that in the absence of model i.e. when we need to need to use Monte Carlo methods then each of the improved policies must be a stochastic one to make sure action-value function estimates are near equal to the mean.

  • $\begingroup$ What have you tried? Have you tried looking at some small examples? If so, what happened in those examples? $\endgroup$
    – D.W.
    Jun 15, 2014 at 20:20
  • $\begingroup$ @D.W: we need to use just $\epsilon$-greedy algorithm. $\endgroup$ Jun 16, 2014 at 0:24
  • $\begingroup$ I can't understand what you mean by that, or how that responds to my questions. Please edit your question to provide more details on what you have tried -- and I suggest you look at some examples and see what happens with them. $\endgroup$
    – D.W.
    Jun 16, 2014 at 6:19

1 Answer 1


A stochastic policy can perfectly represent a deterministic policy by assigning probability 1 to a unique action.

Depending on how you are adjusting the policy it might take time to converge into a deterministic one, however it can get there in the limit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.