How do neural networks create results like its inputs?

Question

I understand basic neural networks (input layer, hidden layers, output layer) and gradient descent learning. However I keep hearing about news talking about neural networks painting and making jazz music. That is puzzling me.

As I understand a neural network, you feed it the input data (i.e. a picture, music, ...) and it gives you some kind of classification on the result. But by reading those results it looks as if they were using the neural network to generate new results based on a training dataset. Because of that, I'm thinking the kind of neural networks they're referring to is essentially different from the ones I've studied, so I'd like to ask:

• What are the differences between classic neural networks and the ones used to run these experiments? What makes them fundamentally different so that they are able to generate results similar to its input?

• Do these results (generating results with a NN) have anything to do with deep learning? Or is deep learning just a better learning technique?

Answer 1

These are known as Autoencoders. As you said, these neural networks are trained to produce output that is similar to the input, rather than output a classification of some kind.

Internally, they do not differ much from other neural net designs. Simply, the expected output is the input (or a slight variant of the input), rather than a classification.

One difference however is often the level of depth. Networks designed to replicate sequences, like writing jazz music, need to remember a sequence throughout training and prediction. This is where deep learning succeeds better than other methods.

This paper briefly discusses one type of an auto-encoding network and how it's used to produce music. I think you will find the design is very similar to a traditional neural network.