# Eligibility trace and the role of gamma and lambda

From R.Sutton's book, the eligibility trace update rule is: $$E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s)$$ I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to most recent states.

At least for gamma I can answer it generally without looking in the book:

Gamma is commonly used as the discount factor. That is a number in (0, 1) which defines how much of the reward you got in step i will be also given to step (i - n). Commonly, the reward is discounted exponentially (like gamma^n)

I was also wondering that, I think I understand it, but I'm having a difficult time proving it.

The principle of using eligibility is that over a large sum you get the same result going backward as you would if you went forward (but you get the additional bonus of being able to update online).

Concerning λ: You use λ the same way as with the forward looking TD(λ) - as a exponential coefficient in a geometric series .

Concerning γ : Looking at the expression for the change in value: $$\delta_t(s) = R_{t+1} + \gamma*V_t(s_{t+1}) - V_t(s_t)$$ and $$V_{t+1}(s) = V_t(s) + \alpha* \delta_{t+1}*e_{t+1}(s)$$ And you do that for EVERY step at EVERY time step, but there's no discount in δ nor in V. So, the future rewards get a discount only if γ is included in the E[s].

I'm also pretty sure that it makes all the other expressions cancel each other out in the sum (except for the rewards and the latest V[s]) , because for #lambda = 1 this should be the same as Monte Carlo:

$$(\lambda = 1): V_{t+1}(s) = V_t(s) + \alpha* \delta_{t+1}*e_{t+1}(s)=V_t(s) + \alpha* \Sigma \gamma^nR_{n}(s)$$

But I can't prove that yet.