# basic doubt on policy iteration

consider the policy iteration algorithm for a finite state MDP. Suppose the initial policy is a stochastic policy. Now, can the optimal policy be deterministic after improvements ? Or, can we say that always the optimal policy will be a stochastic one ? Confused about this. Any ideas will be helpful. The reason I am asking this question is that in the absence of model i.e. when we need to need to use Monte Carlo methods then each of the improved policies must be a stochastic one to make sure action-value function estimates are near equal to the mean.

• What have you tried? Have you tried looking at some small examples? If so, what happened in those examples? – D.W. Jun 15 '14 at 20:20
• @D.W: we need to use just $\epsilon$-greedy algorithm. – RIchard Williams Jun 16 '14 at 0:24
• I can't understand what you mean by that, or how that responds to my questions. Please edit your question to provide more details on what you have tried -- and I suggest you look at some examples and see what happens with them. – D.W. Jun 16 '14 at 6:19