consider the policy iteration algorithm for a finite state MDP. Suppose the initial policy is a stochastic policy. Now, can the optimal policy be deterministic after improvements ? Or, can we say that always the optimal policy will be a stochastic one ? Confused about this. Any ideas will be helpful. The reason I am asking this question is that in the absence of model i.e. when we need to need to use Monte Carlo methods then each of the improved policies must be a stochastic one to make sure action-value function estimates are near equal to the mean.
A stochastic policy can perfectly represent a deterministic policy by assigning probability 1 to a unique action.
Depending on how you are adjusting the policy it might take time to converge into a deterministic one, however it can get there in the limit.