Projects and Research

Posthoc analysis of safety of autonomous systems is a challenging problem. In situations where the system dynamics are well understood as an ODE with analytical feedback control laws, it is a challenging problem to analyze in practice. An added difficulty is when a deep neural network is used to compute the control input. In recent years there has been a proliferation of autonomous systems which fall into this category. Autonomous system which use neural networks in their decision making pipeline. RL based controllers have made their in roads in applications where an explicit model can be hard to build [link].

To address the above problem, we developed algorithms which can compute input-output ranges of neural networks for safety verification of closed loop cyber-physical systems. This relied on the fact a large range of physical systems, can be modeled as a hybrid automaton. Past research has produced tools like Flow* which can automate the analysis of physical dynamics. This meant that a careful encapsulation of the neural network in a sound fashion, can help us compute reach sets of the overall system. The details can be found here . We show how reach set computations can help produce set based simulation to guarantee safety.

But there is a catch ! In high-dimensional spaces like LIDAR, verification often becomes a hard property to capture due to lack of proper specifications. To this end, we developed a monitoring framework with statistical guarantees [link]. The statisitical guarantees that are obtained, are on the likelihood of an input point being sampled from an unknown region. But this analysis lacks the closed loop aspect of whether the system behavior is necessarily going to be affected when sampled from this unknown region. This has interesting ramifications on potential future work. Having explored these two ends of the spectrum, the natural next step is to develop techniques which can bridge verification and monitoring. In order to build safety monitors for cyber-physical systems with neural network components. Additionally a possible extension to recovery schemes under distribution shifts.

Trusted artificial intelligence is hard to achieve with current state of the art techniques of building machine learning models and inference systems. Apart from being data-hungry the size of models can be prohibitively large for being amenable to scrutiny in a post-hoc fashion. Informally, trust is an expression of confidence in an autonomous system's ability to perform an underspecified task. A large part of the progress in the last decade in the field of artificial intelligence has been driven by algorithmic improvements using specialized computer hardware. This has fuelled some stark progress in automated board games to protein folding, language translation to medical image analysis.

A possible future direction would be to extend machine-learning beyond simple data-driven pattern recognition and build avenues to integrate knowledge driven reasoning that includes context, physics and other background information. In each domain that can benefit from the advances in learning based components, there exists a gamut of expert knowledge built over centuries. These are often things like common-sense practices in each field, which are not so common once the parlance shifts. Yet machine-learning models do not have a mechanism to adapt and incorporate these domain specific information in a seamless manner. Such classical techniques have numerous advantages :

• The presence of rich abstractions that are grounded in the specific domain, and have their respective formalisms. Such formulations are often well supported in terms of tools and algorithms optimized for the specific domain. For instance, in case of robotics applications there is a slow progress towards using standard physics guided information to facilitate model-based reinforcement learning.

• They are often modular and composable. This allows for reuse, precision and automated analysis.

• This opens the doorway for formal specifications to capture system intent and verification technologies to be used. Making the system more resilient in safety critical situations against cyber attacks.

In this direction, we have developed a method which uses ideas from child-development to combine expert information with learning based system design. For instance, the sticky-mittens experiment [link] considers infants who have not yet learned to grasp objects. They give a subset of these infants mittens covered with Velcro hooks and allow them to play with toys fitted with Velcro loops, making it significantly easier for them to grasp these toys. Even if the Velcro is taken away, these babies learn how to grasp objects significantly faster than infants not exposed to this experience. In other words, enabling infants to explore unreachable parts of the state space helps guide them towards skills that are worth learning.

We introduce jumps in the state space of the system called option-templates in our paper. The agent in this case learns to use expert intervention before learning to implement it. This lead to orders of magnitude improvements in sample complexity ~100 in some cases. The next steps would be to develop techniques to combine such external symbolic information to build learning based autonomous systems.

Improving robustness of learning based models in a classification setting has been the sole purpose for a large section of machine learning literature. The brittleness of deep-learning based models often stand in its way to break into applications in high stakes setting like medicine. A well known issue is that images when subjected to small perturbations imperceptible to the human eye, can completely fool the classifier. Such perturbations can be constructed with full knowledge of the deep-learning model (adversarial setting), or by subjecting the images to naturally occurring perturbations like Gaussian noise [paper]. The former being the worst case scenario. We discovered similar behavior even in the case of 1-dimensional time series data such as electrocardiogram (ECG) models. In this setting the classification goal would be to predict different types of cardiac arrhythmia, from the time series signals. Naturally occurring perturbations in this setting happen due to things like muscle artefact noise and others. Muscle artefact noise occurs due to random twitching of muscles in the ECG measurement process. Muscle artefact isn't the only one out there. Other similar physiological perturbations expose the brittleness of DNNs, which can impair the classifier performance.

Before the advent of deep-learning based methods such life critical applications often used expert (physician) informed features which formed the inputs to the classifier. This was commonly known as feature engineering, and was well known in both medical images and other time series signals like ECG. Such features were often robust to noise. Current deep-learning based methods lack an avenue to integrate such pre-existing features into the data-driven learning framework. In our memory-classifiers paper , I developed a technique to combine the two worlds to obtain a more robust classifier without compromising performance. Mainly because we do not alter any of the existing optimizations algorithms or architecture choice in building this framework. This produced impressive results across different domains when compared to the benchmarks. Moving forward I plan to develop techniques which can use such sample points called memories to auto-clean inputs before being fed into a classifier. Classifiers often need to be updated in an incremental fashion as more data from different regions of the input domain are available. Memory-classifiers can be incrementally extended due to the natural separation it offers in the input space. In the future I would like to develop techniques which can create such easily editable classifiers without going through the full retraining pipeline.

This is a novel direction we discovered and we are just beginning to explore the level of flexibility this can offer. This can be extended to multi-modal pathways of learning as well, where different parts of the input can declare affinity to sub-components depending on its similarity metric.