How to get a heatmap of pixel influences from a Convolutional Neural Network

Question

If I have a CNN model (let's say I'm using Tensorflow and Keras) that I've trained for a particular task. For example, let's say I've trained it to detect the difference between an apple and banana. For a given input image, is there a way to generate a heatmap overlaid on the original image that indicates which parts of that image influenced the network's decision?

I'm guessing that you'd have to use the chain rule to get the partial derivative of the loss function with respect to each pixel, and the normalize those values and heatmap them. Is there a library or existing method that does this?

How would I do this if I had a CNN mapping to predictions for more than two classes?

Answer 1

Yes, there are various proposals out there. Many of them work by computing a saliency map. If you search for "saliency map", you should be able to find multiple schemes for that.

Personal opinion: these methods give heatmaps that seem superficially plausible; but my sense is that they have major limitations. So don't set your expectations too high, or you may be disappointed.