Research Activities

I am thirsty for knowledge, like logical puzzles and feel incredibly excited when new elegant theories open the door to solving practical problems. This is the reason I did my PhD, continued with postdoctoral training and, after a short excursion to the industry, decided to stay in academia. This page describes several research topics I have been recently working on and contributed to.

Visit our group webpage for more information on funded project and research activities: Embedded Information Processing - Research.

Understanding the Loss Landscape of Deep Neural Networks

Building efficient ensembles from trained deep models is a fundamental technique to improve prediction accuracy. Model ensembles are built in either the output space or in the weight space. While several related works show that output-space ensembling improves prediction accuracy and robustness, it requires a separate inference pass through each model, making it less interesting for resource-constrained applications. An alternative research direction investigates the means to build model ensembles in the weight space by averaging trained solutions. Vanilla averaging two SGD solutions fails: it leads to a low accuracy model due to a high loss barrier between two models trained independently from different seeds. However, there are cases when weight-space averaging is successful, e.g., if two models share a part of their optimization trajectory during training. We conjecture that the energy barrier between two independently trained SGD solutions can be removed if permutation invariance is taken into account and show how to do this in practice for a number of datasets and architectures.

Permutation conjecture
ICLR'22 and arXiv

Zero barrier with REPAIR
ICLR'23 and arXiv

REPAIR solves variance collapse
ICLR'23 and arXiv

Resource-efficient and Adaptive Deep Learning

Deep learning methods are increasingly employed to tackle complex tasks; however, the optimal selection of training data, architecture, and model capacity that align with hardware and energy constraints remains poorly understood. This research project aims to address this gap by exploring the trade-offs and optimization potential in this domain. Our focus also extends to developing deep learning models that demonstrate robustness, adaptability, and reconfiguration capabilities in out-of-distribution scenarios. Specifically, we concentrate on resource-constrained IoT devices and their autonomous operation on the edge, where data scarcity and non-stationary environmental dynamics pose unique challenges.

To accomplish our objectives, we analyze the impact of various factors such as model architecture, overparameterization, sparsity, data distributions, augmentation techniques, training regimes, hyperparameters, and regularization on generalization and robustness. Our research revolves around optimized models operating within resource-constrained environments, considering three key perspectives: (1) enhancing robustness against distribution shifts and engineering safe AI-based systems, (2) optimizing the efficiency of compressed models on embedded hardware, and (3) developing methods to adapt or reconfigure models for new tasks.

Subspace-configurable networks, configuration subspace D=2 (left), and achieved accuracy for rotation transformation (right), arXiv and Twitter paper

Subspace-configurable networks, D=8
arXiv and Twitter paper

Security, Privacy, Bias and Vulnerabilities of AI-based Systems

IoT sensor data is often collected in very private areas of human life and it is possible to derive sensitive information from it. For example, daily routine of a person or occupancy status of a flat can easily be predicted when looking at smart meter readings; wearable devices reveal the stress level of the person wearing it. However, many sensor data related applications would greatly benefit if the collected data could be shared or used to train machine learning models.

Privacy preserving machine learning is an active field of research, but so far IoT sensor data did not receive the necessary attention. In our work we try to bridge the gap between useful but private data and its applications, by researching methods on how to share and processes this data without sharing or leaking private information.

Adaptive, Predictive and Low-power Sensing for Clean Air

Low-cost environmental sensors and environmental models present interesting use-cases for testing, optimizing and improving machine learning models. For example, low-cost gas sensors may drift over time or their measurements may be affected by other environmental processes and environmental dynamics. Sophisticated machine learning models for accurate sensor calibration can help to compensate for these effects. On-device sensor data processing, however, may face severe resource constraints including processing power, memory and energy budget, that need to be taken into account. Another example is that some types of chemical gas sensors are power-hungry and machine learning can be used to replace actual measurements with high-quality predictions. Finally, sensor data coming from IoT sensors measuring gases and particle concentrations in the ambient air help to push the limits of today's air quality maps by extending the conventional networks of static high-quality measurement stations with dense IoT measurements. The challenge is to show that it is possible to build high-quality and high-resolution air quality maps using low-cost, less precise, low-resolution, less stable, noisy sensors. While successfully solving this challenge, we managed to improve the accuracy of air quality models by accounting for air pollution transfer, and used our models to understand the impact of COVID-19 lockdown measures on local air quality.

Air pollution model
LUR, PerCom'14 / PMC'15

Sensor calibration
SensorFormer, IoTJ'22

Tracking pollution transfer
TIP Air, CPD'21