Evaluating the (Un)Trustworthiness of Saliency Maps in Medical Imaging

(Code available at: https://github.com/QTIM-Lab/Assessing-Saliency-Maps)

Background

Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image.  They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the  neural network makes. However, the utility and robustness of these saliency maps have not yet been rigorously examined in the context of medical imaging. This work examines four questions:

1. Are saliency maps good localizers?

2. Do saliency maps vary with model weight randomization?

3. Are saliency maps repeatable?

4. Are saliency maps reproducible?

Using the localization information available in two large public radiology datasets, the performance of eight commonly used saliency map approaches is examined (Gradients, Smoothgrad, Integrated Gradients, Smooth Integrated Gradients, GradCAM, XRAI, Guided-backprop, and Guided GradCAM).

Contribution to Data Science

Building off some initial work performed in this area [1, 2], this work develops a comprehensive framework to objectively examine the quality of saliency maps in four key areas: 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability and 4) reproducibility. 

Contribution to Healthcare Domain

This is the first work to corroborate the findings from [1] on medical images. The evaluation framework found that all eight saliency map techniques fail at least one of the criteria and are, in most cases, less trustworthy when compared to the baselines.   It is concluded that their usage in the high-risk domain of medical imaging warrants additional scrutiny and recommended that detection or segmentation neural networks be used instead if localization is the desired output of the network.

For more details of the project see Reference [3].

References

1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (pp. 9505-9515).

2. Arun, N. T., Gaw, N., Singh, P., Chang, K., Hoebel, K. V., Patel, J., ... & Kalpathy-Cramer, J. (2020). Assessing the validity of saliency maps for abnormality localization in medical imaging. arXiv preprint arXiv:2006.00063.

   • • First co-author

3. Arun, N. T., Gaw, N., Singh, P., Chang, K., Aggarwal, M., ... & Kalpathy-Cramer J. (In Review) Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. PNAS. medRxiv preprint: https://doi.org/10.1101/2020.07.28.20163899.

   • • First co-author