Neural networks achieve state-of-the-art performance in most applications they are applied in. However, they are not interpretable and thus, in some cases, they are not that reliable.
Randomized Input Sampling for Explanation of Black-Box Models tries to remove this issue by pointing out the regions of the image the model is focusing on while making its inference. It does that by creating multiple masks and occluding the image by those masks. The outputs of these occluded images are taken and the output probability of the necessary target class is considered. The mask which results in a lower probability thus occludes important features of the image while those which result in higher probability are covering non-important features. Based on the probabilities, the different masks are linearly weighted to obtain a final mask.
The mask gives a heatmap of the regions the model is focussing on. A higher value on the mask means that region of the image does not contribute very much to the final prediction. The model does not make any assumption on the architecture of the model and thus can be applied to a larger set of models and even when the weights of the model are not available.
The paper introduced a simple method to understand the heatmaps of neural networks just based on their outputs! The original paper is available at this link.