0
$\begingroup$

From R.Sutton's book, the eligibility trace update rule is: $$ E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s) $$ I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to most recent states.

$\endgroup$
1
$\begingroup$

At least for gamma I can answer it generally without looking in the book:

Gamma is commonly used as the discount factor. That is a number in (0, 1) which defines how much of the reward you got in step i will be also given to step (i - n). Commonly, the reward is discounted exponentially (like gamma^n)

$\endgroup$
0
$\begingroup$

I was also wondering that, I think I understand it, but I'm having a difficult time proving it.

The principle of using eligibility is that over a large sum you get the same result going backward as you would if you went forward (but you get the additional bonus of being able to update online).

Concerning λ: You use λ the same way as with the forward looking TD(λ) - as a exponential coefficient in a geometric series .

Concerning γ : Looking at the expression for the change in value: $$ \delta_t(s) = R_{t+1} + \gamma*V_t(s_{t+1}) - V_t(s_t) $$ and $$ V_{t+1}(s) = V_t(s) + \alpha* \delta_{t+1}*e_{t+1}(s) $$ And you do that for EVERY step at EVERY time step, but there's no discount in δ nor in V. So, the future rewards get a discount only if γ is included in the E[s].

I'm also pretty sure that it makes all the other expressions cancel each other out in the sum (except for the rewards and the latest V[s]) , because for #lambda = 1 this should be the same as Monte Carlo:

$$ (\lambda = 1): V_{t+1}(s) = V_t(s) + \alpha* \delta_{t+1}*e_{t+1}(s)=V_t(s) + \alpha* \Sigma \gamma^nR_{n}(s) $$

But I can't prove that yet.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.