Consider the following eqautions used in LSTM ( taken from Andrew ng's course on Sequential model)

In an LSTM model, LSTM Cell has three inputs at any time step t

  • Input($X_t , a^{(t-1)}, C^{(t-1)})$,

    Here $X_t$ is the input vector, $a^{(t-1)}$ is the previous hidden state and $ C^{(t-1)}$ is the previous cell state

Now the new cell state $c^t$ is given by the following formula :

$C^t = $ forget_gate * $C^{(t-1)} + $ update_gate* $\overline{C^t}$


If I swap the places of forget_gate and update_gate, I still get a valid $C^t$, So why are we multilyting the previous cell state with forget gate only and the current cell state with update gate only, what if Imultiply previous cell state with update gate ?

Edit : After swapping, the formula would look like this,

$C^t = $ update_gate * $C^{(t-1)} + $ forget_gate* $\overline{C^t}$

enter image description here

  • $\begingroup$ I don't understand what you mean by "swap the places". Can you show what the new formula would be after your swap, compared to what it was before the swap? $\endgroup$
    – D.W.
    Aug 24 '20 at 16:40
  • $\begingroup$ Your formula doesn't match the displayed equation ($C$ vs $\tilde{c}$.) $\endgroup$
    – D.W.
    Aug 24 '20 at 19:10

The two formulas are mathematically equivalent; the only change is that you have swapped the names of the variables (changes them to names that are less intuitive, compared to what effect they have), but that doesn't affect the behavior of the system.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.