Reducibility and Artificial Neural Networks

Question

I have read (here and here ) about the computational power of neural networks and a doubt came up.

There is a way to reduce an ANN to another ANN (not taking into count the training algorithm) ? e.g. Reduce a Recurrent Neural Network to a Multilayer Perceptron, meaning that if I have a trained RNN, I can get a MP that maps the same inputs given to the RNN to the same outputs produced by the RNN.

And if exists an answer to the above question, we can show the equivalence between neural networks, e.g., all problems solved by an Multilayer Perceptron can be solved by a Recurrent Neural Network but the opposite is not true, i.e., $MP \subset RNN$ (I do not know if this is true, is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network $X$ that is more powerful than others, so, we can throw away all other neural networks because $X$ can solve any problem that other NN can. Is this reasoning correct ?

Thanks.

Answer 1

Not really. I respect what you're trying to achieve, but I don't think it's possible to achieve what you want, given our current level of knowledge of neural networks.

We already know that convolutional neural networks perform better for some problems, and fully-connected neural networks (what you call MP) work better for other problems. So you can't expect to find that one is always better than the other.

It has been proven that any computable function can be approximated arbitrarily well by some fully-connected neural network. However, the catch in this theorem is that the theorem doesn't tell us how large the neural network needs to be. We already know that sometimes a larger network is more accurate, but is also slower to train, so that's a pretty big caveat in the theoretical result.

And if you want to compare two different network architectures -- say, convolutional vs fully-connected -- then such a comparison won't be useful if it doesn't take into account the size of the network. If you need 1 billion parameters to make a fully-connected network work well, or 1 million parameters to make a convolutional network work well, are they equally good? No, you'll probably prefer the convolutional network. It's not at all clear how to get useful reductions between the two architectures that tells us anything about the size of the neural network.

So, no, you're probably not going to get some useful theory this way that says "you can throw away all other architectures other than type X".