The project aims at developing mathematical methods for Explainable Artificial Intelligence (XAI). The emerging field of XAI seeks fundamental approaches that contribute to the interpretability and explainability of deep neural networks (DNNs) as opposed to practices where DNN is commonly used as an opaque box. The focus of the project is on specifying algebraic structures and formal techniques for performing analysis of deep learning models' behavior in biomedical image processing.