consider the policy iteration algorithm for a finite state MDP. Suppose the initial policy is a stochastic policy. Now, can the optimal policy be deterministic after improvements ? Or, can we say that always the optimal policy will be a stochastic one ? Confused about this. Any ideas will be helpful. The reason I am asking this question is that in the absence of model i.e. when we need to need to use Monte Carlo methods then each of the improved policies must be a stochastic one to make sure action-value function estimates are near equal to the mean.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ What have you tried? Have you tried looking at some small examples? If so, what happened in those examples? $\endgroup$– D.W. ♦Commented Jun 15, 2014 at 20:20
-
$\begingroup$ @D.W: we need to use just $\epsilon$-greedy algorithm. $\endgroup$– RIchard WilliamsCommented Jun 16, 2014 at 0:24
-
$\begingroup$ I can't understand what you mean by that, or how that responds to my questions. Please edit your question to provide more details on what you have tried -- and I suggest you look at some examples and see what happens with them. $\endgroup$– D.W. ♦Commented Jun 16, 2014 at 6:19
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
A stochastic policy can perfectly represent a deterministic policy by assigning probability 1 to a unique action.
Depending on how you are adjusting the policy it might take time to converge into a deterministic one, however it can get there in the limit.
