# RNN learning an iterative algorithm

How do I solve the following problem with a recurrent neural network (RNN)? What architecture should I use for the (conv)-RNN?

Let $s \in \mathbb{R}^N$ be a musical signal. We corrupt it with some white/pink noise $\omega$ to obtain $x= s+\omega$. We then create a conv-RNN with $N$ input and $N$ output neurons, we feed it with input $x^{(t)}=x$ and we train it to output the sequence $y^{(t)} = x -\frac{t^2}{100^2} \omega$ for $t = 0\ldots 100$, ie. some better and better approximations to $s$.

We repeat the same process with many different $s,w$ and we hope the resulting RNN will serve as a musical de-noiser.

De-noising is supposed to have a chance to work because it is quite suitable for convolution-RNNs, so even if we need a lot of neurons, the number of weights to learn shouldn't be too high.

Note this isn't only about this particular problem, I chose it because it is easy to generate training data the way I said, I will be interested in anything different but related (in particular anything about a conv-RNN trained to output better and better approximations to the solution). For example we could replace noisy musical signals by low-resolution pictures, and train the RNN to output higher and higher resolution version of the picture.

Edit - For now, I think the architecture of the RNN should be the following : Let $X = \text{spectrogram}(x)$ be the input, and each $X_{i,j}$ is a $16\times 16$ piece of $X$, the neighbor pieces overlap. Then the output of the $i,j$th piece of neurons at time $t+1$ is $$z_{i,j}^{(t+1)} = F(W,X_{i,j}, z_{i,j}^{(t)},z_{i\pm 1,j\pm 1}^{(t)})$$

where $W$ are the parameters of $F$ to be optimized. The output of the RNN at time $t$ is $Y^{(t)} = X-\sum_{i,j} z_{i,j}^{(t)} \delta_{i,j}$ (assembling all the pieces of spectrogram together, $z_{i,j}$ is supposed to contain some local estimation of the noise) and the error is $$E^{(t)} =\| Y^{(t)} -(X-\frac{t^2}{100^2} \Omega)\|^2, \qquad E = \sum_{t=1}^{100} E^{(t)}$$ $F$ is itself a 3 layer perceptron, so the parameters $W$ are the weights of all those $3$ layers, and the input weights tell how the neighbor pieces $z_{i\pm 1,j\pm 1}^{(t)}$ affect $z_{i,j}^{(t+1)}$, which we hope will let the information to propagate from a piece of neurons to the neighbor.

We update the parameters from something like $$W \leftarrow W - \eta \sum_{t=1}^{100} \frac{\partial E^{(t)}}{\partial W}$$

• a. I've edited it to try to focus it more. Does it correspond to what you want to ask? (I didn't see what the last two paragraphs have to do with the question at the top so I removed them to keep the question focused. If you want to ask about different problems, like denoising images, you can ask that separately. If you want to ask about a different approach, like genetic algorithms, you can also ask that separately) – D.W. Nov 14 '17 at 16:40
• b. What have you tried so far? Have you tried anything? What architectures have you considered? Why have you rejected them? Probably the only way to answer "what architecture will work well?" is to try some possibilities and see how well they work. These things are hard to predict - you have to run some experiments and see what you find. – D.W. Nov 14 '17 at 16:41
• c. You say you want to train the RNN to output the sequence $x-1^2/100^2 w$, $x-2^2/100^2 w$, $x-3^2/100^2 w$, ... when given $x$ as input. I don't see how that would work. The input to a RNN is typically a sequence of numbers, and the output of a RNN is a sequence of numbers. Your signals $x$ and $y$ are already sequences of numbers. I don't understand how a RNN would output a sequence of sequences. Have I misunderstood something about what you are asking? – D.W. Nov 14 '17 at 16:44
• @D.W. Sure it is hard to predict, that's why I'd like to find some litterature about (conv)-RNN trained to output better and better approximations to the solution (of any non-trivial problem !). Output a sequence of arrays means $N$ output neurons, and $t$ is the time index. I'd didn't run RNN experiments for now, I'm thinking about designing toy problems, RNN architecture, training data, not specifically about musical denoising (which I chose also because it is easy to construct training data the way I said in my question). – reuns Nov 14 '17 at 16:49
• @D.W. Is it clearer ? The idea is that ideally, to solve musical-denoising (and many other problems) we do it by applying iteratively $x \leftarrow x - \eta\ \text{Noise}(x)$ where $\text{Noise}(x)$ is a function estimating the "direction of noise" in the musical signal $x$. Many people tried to use NN to learn $\text{Noise}(x)$, but as far as I know, they didn't try using RNN to learn the whole sequence $x^{(t+1)} \leftarrow x^{(t)} - \eta\ \text{Noise}(x^{(t)})$, ie. the whole de-noising algorithm. – reuns Nov 14 '17 at 18:49