How does the forget layer of an LSTM work?

Question

Can someone explain the mathematical intuition behind the forget layer of an LSTM?

So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.

Answer 1

Think of it like this: The cell state $h_t$ is a vector. The forget vector $f_t$ is used to choose which parts of the cell state to "forget". We update the hidden state with something like $c_t = f_t \circ c_{t-1}$ (it's actually more complicated, but let's start with that, to gain intuition). Suppose $f_t$ were a vector of 0's and 1's. In the coordinates where $f_t$ is 1, the value of $c_{t-1}$ would be copied over to $c_t$ (it's not forgotten). In the coordinates where $f_t$ is 1, $c_t$ is reset to zero and the value of $c_{t-1}$ is ignored (it's forgotten). So, the forget vector can be used to control in which positions we forget values from the previous cell state vector.

Now what remains is to figure out a way to choose a forget vector $f_t$. In general we might want to choose which positions to forget based on both the current input $x_t$ and the previous hidden state $h_{t-1}$. So, we should compute $f_t$ as some function of $x_t$ and $h_{t-1}$. Many choices of how to represent that function might be possible, but a LSTM chooses a specific function for this. In a LSTM, this is done by a single-layer fully-connected neural network. A single-layer fully-connected neural network concatenates all of the inputs, then multiplies them by a matrix, adds a bias, and feeds the result to an activation layer (in this case, sigmoid activation). So that's why the formula for $f_t$ looks the way it does: that formula is capturing what a single-layer fully-connected neural network does.