10
$\begingroup$

After reading of the google deepmind achievements on Atari's games, I am trying to understand the q-learning and q-networks, but I am little bit confused. The confusion arise in the concept of the discount factor. Brief summary of what I understand. A deep convolutional neural network is used to estimate the value of the optimal expected value of an action. The network has to minimize the loss function $$ L_i=\mathbb{E}_{s,a,r}\left[(\mathbb{E}_{s'}\left[y|s,a\right]-Q(s,a;\theta_i))^2\right] $$ where $\mathbb{E}_{s'}\left[y|s,a\right]$ is $$ \mathbb{E}\left[r+\gamma max_{a'} Q(s',a';\theta^-_i)\right|s,a] $$ Where $Q$ is a cumulative score value and $r$ is the score value for the action choose. $s,a$ and $s',a'$ are respectively the state and the action choose at the time $t$ and the state and the action at the time $t'$. The $\theta^-_i$ are the weights of the network at the previous iteration. The $\gamma$ is a discount factor that take into account the temporal difference of the score values. The $i$ subscript is the temporal step. The problem here is to understand why $\gamma$ does not depends on $\theta$.

From the mathematical point of view $\gamma$ is the discount factor and represents the likelihood to reach the state $s'$ from the state $s$.

I guess that the network actually learn to rescale the $Q$ according to the true value of $\gamma$, so why not letting $\gamma=1$?

$\endgroup$
6
$\begingroup$

The discount factor does not represent the likelihood to reach the state $s′ $from the state $s$. That would be $p(s'|s,a)$, which is not used in Q-Learning, since it is model-free (only model-based reinforcement learning methods use those transition probabilities). The discount factor $γ$ is a hyperparameter tuned by the user which represents how much future events lose their value according to how far away in time they are. In the referred formula, you are saying that the value $y$ for your current state $s$ is the instantaneous reward for this state plus what you expect to receive in the future starting from $s$. But that future term must be discounted, because future rewards may not (if $γ < 1$) have the same value as receiving a reward right now (just like we prefer to receive \$100 now instead of \$100 tomorrow). It is up to you to chose how much you want to depreciate your future rewards (it is problem-dependent). A discount factor of 0 would mean that you only care about immediate rewards. The higher your discount factor, the farther your rewards will propagate through time.

I suggest that you read the Sutton & Barto book before trying Deep-Q in order to learn pure Reinforcement Learning outside the context of neural networks, which may be confusing you.

$\endgroup$
  • $\begingroup$ Thanx for your answer, but I have still some doubts. I am thinking at loud. Imagine at every step you receive a score of $d$ and you have to pay $c$ to start play. How do I calculate the expected value? Well $$ Ev=\sum_{i=1}^{+\infty} \gamma^i d -c $$ because you are adding values of $d$ at different moments in the future, isn't it? $\endgroup$ – emanuele Aug 3 '15 at 12:16
  • $\begingroup$ Well, I will break even if $$d\frac{\gamma}{1-\gamma}=c$$ what is the correct value for $\gamma$? The correct value for $gamma$ is the value that allow me trade-off between the present and futures rewards and is $\gamma=p$. $p$ is the probability to survive at the step $t$ and that is the why $0\le \gamma \le 1 $. The check is $\frac{p}{1-p}=\tau$ where $\tau$ are odds to survive at every steps and is the expected life span. $\endgroup$ – emanuele Aug 3 '15 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.