Research

Multimodal learning where reliability matters

My research asks how multimodal AI can remain dependable when data, computation and access to modalities are constrained. These constraints arise together in medical imaging and industrial perception: a model must combine heterogeneous signals, generalise beyond its training environment and recognise when its predictions may be unreliable.

The programme connects four research themes. Methods for reliability guide how we learn; clinical and industrial problems expose the failures that matter; open datasets and benchmarks make those failures measurable.

1. Reliability under distribution shift

We study how to identify inputs outside a model’s training distribution and explain the evidence behind its predictions. Contributions include feature normalisation and loss reweighting for out-of-distribution detection (T2FNorm, ReweightOOD), neuron-level explanations (NERO), test-time augmentation (TTA-OOD), and evaluation of semantic understanding and hallucination in medical vision–language models (MedTech-Bench and gastrointestinal benchmarks).

2. Learning under resource constraints

We develop methods for institutions that cannot pool data, sites with missing modalities, scarce or noisy labels, restricted inference budgets, and languages with limited training corpora. CAR-MFL and ProMoE-FL address missing modalities in federated learning; related contributions include FedVG, Dimension Mixer and NepBERTa.

3. Multimodal perception in the field

We investigate perception across equipment, procedures and populations absent from the training data. Work spans surgical image segmentation, monocular depth and camera pose in endoscopy, guidewire reconstruction from bi-planar X-ray, and latent action modelling for surgical world models. Representative contributions include SurgTransfer, Guide3D, FetReg2021 and CholecTriplet2022.

4. Open datasets and evaluation

We build evaluation resources with clinical collaborators so that reliability claims can be tested against annotated data. These contributions include SAGE, MedTech-Bench, gastrointestinal hallucination benchmarks, Guide3D and the NepBERTa corpus. Dataset construction and evaluation are integral scientific contributions within the programme.

Clinical and industrial collaborations

Clinical work covers gastrointestinal endoscopy, fetoscopy, surgical video, non-contrast CT head and cardiac imaging, with collaborators in the United Kingdom, Nepal and Switzerland. At Fogsphere (Redev AI Ltd), I direct the research programme behind perception systems for safety-critical industrial environments, including energy, manufacturing, logistics, mining and construction.

Research foundations

Earlier work at Imperial College London and Normandie Université developed methods for face analysis, conditional generative models and large-scale metric learning. These contributions are included in the Publications page.