Google DeepDream Elaborated

Question

I've seen a few questions on this site about Deep Dream, however none of them seem to actually speak as to what DeepDream is doing, specifically. As far as I've gathered, they seem to have changed the objective function, and also changed backpropagation so that instead of updating weights they update the input image.

I'm wondering if anyone knows exactly what Google did. They mention in one of their articles imposing Bayesian priors when they carry out their optimization, and with this I can imagine that getting the neural net to spit out an image for each label is not that difficult - we can simply set the label, and then optimize the input vector accordingly.

However, the interesting part of deep dream is that it does this per layer, and in this regard I'm not quite sure how it emphasizes the details in a per layer way.

Certainly, feeding in an image will give you values at each neuron, but then how can we use that information to exaggerate details in the original image? I've struggled to find any detailed write-ups about this.

References: Here vzn answered a similar question: https://cs.stackexchange.com/a/44857/49671

From that link, there is an implementation of Deepdream, here: http://auduno.com/post/125362849838/visualizing-googlenet-classes

Except it doesn't offer exaggeration of features as discussed here:http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html

Where they both show the visualization of particular classes, and particular layers, and say:

Instead of exactly prescribing which feature we want the network to amplify, we can also let the network make that decision. In this case we simply feed the network an arbitrary image or photo and let the network analyze the picture. We then pick a layer and ask the network to enhance whatever it detected.

Answer 1

The idea of DeepDream is this: pick some layer from the network (usually a convolutional layer), pass the starting image through the network to extract features at the chosen layer, set the gradient at that layer equal to the activations themselves, and then backpropagate to the image.

Why does it make sense? Intuitively, it amplifies the features that are maximally activated in the network. By backpropagating this gradient, we'll make an image update that will boost any of the existing activations. If there's a cat-like detector in the layer and the image contains some patch that looks a bit like a cat, DeepDream boosts this activation by updating this patch to be even more cat-like. As a result, DeepDream is trying to find cats and dogs everywhere in the image (ImageNet dataset has lots of dogs, so the network has many dog-related neurons).

If you look at the code, the key part is objective_L2 function that makes this: $dx = x$ and then backpropagates.