If I have a CNN model (let's say I'm using Tensorflow and Keras) that I've trained for a particular task. For example, let's say I've trained it to detect the difference between an apple and banana. For a given input image, is there a way to generate a heatmap overlaid on the original image that indicates which parts of that image influenced the network's decision?
I'm guessing that you'd have to use the chain rule to get the partial derivative of the loss function with respect to each pixel, and the normalize those values and heatmap them. Is there a library or existing method that does this?
How would I do this if I had a CNN mapping to predictions for more than two classes?