1
$\begingroup$

If I have a CNN model (let's say I'm using Tensorflow and Keras) that I've trained for a particular task. For example, let's say I've trained it to detect the difference between an apple and banana. For a given input image, is there a way to generate a heatmap overlaid on the original image that indicates which parts of that image influenced the network's decision?

I'm guessing that you'd have to use the chain rule to get the partial derivative of the loss function with respect to each pixel, and the normalize those values and heatmap them. Is there a library or existing method that does this?

How would I do this if I had a CNN mapping to predictions for more than two classes?

Improve this question
$\endgroup$
1
$\begingroup$

Yes, there are various proposals out there. Many of them work by computing a saliency map. If you search for "saliency map", you should be able to find multiple schemes for that.

Personal opinion: these methods give heatmaps that seem superficially plausible; but my sense is that they have major limitations. So don't set your expectations too high, or you may be disappointed.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. Just curious, why do you think that they have major limitations? $\endgroup$ – Shrey Joshi Oct 18 '20 at 19:30
  • $\begingroup$ @ShreyJoshi, I wish I could, but I'm not sure how to articulate it very coherently (it's just an impression), and anyway it is only one person's opinion, which could be totally off-base. My sense is that they fall significantly short of what we might hope for, to have an interpretable and explainable model. Sorry that I don't know how to be useful on that. $\endgroup$ – D.W.♦ Oct 18 '20 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.