Reinforcement Learning - Q Learning

Question

I am having trouble understanding the following problem and Q learning in general.

What I know so far about Q learning is that Q-learning is a model free method, i.e., it doesn’t need to learn P(s’|s,a). Thus it can compare the expected utility of the available actions without the model of the environment.It will also eventually find a policy that is the maximum achievable. However, I am not able to use this knowledge to do this problem. The answer is given in red.

If we are to look at time t = 1, then the equation becomes the following: $$ Q_{t+1}(s,a)=(1-1/2)(1)+1/2[R(s,a,s^{'})+(1) max⁡Q(s^{'},a^{'})] $$ How do I proceed from here?

Answer 1

At t=1, you just took action a = d at state s = S and ended in s' = S again with 0 reward. So:

$Q(S,d) = (1 - 1/2)(1) + 1/2[0 + (1)(1)] = 1$ ($maxQ(s',a')$ is the maximum value you can get from state S in this case, which is 1)

So nothing changes. At t=2, it is exactly the same as t=1, so nothing changes again.

At t=3, we just took action p at s = S and ended on state s' = P with reward +1. Thus:

$Q(S,p) = (1 - 1/2)(0) + 1/2[1 + (1)(1)] = 1$ ($maxQ(s',a')$ here refers to $maxQ(P,a')$, the maximum value we can get from state P, which is 1.

So, update $Q(S,p)$ to 1.

At t=4, we just took action a = p at state s = P, ending in state s' = S with reward 0. Thus:

$Q(P,p) = (1 - 1/2)(0) + 1/2[0 + (1)(1)] = 1/2$

So, update $Q(P,p)$ to 1/2.

I think you can do t=5 and t=6 by yourself now, just note that maxQ at a terminal state is 0 (because there is no future reward to accumulate). If you can't, just tell me and I'll update the answer.