Current & Past Projects
Action Conditioned Video Generation
As part of my PhD thesis I am currently working on developing a generalised mathematical framework for action-conditioned video generation. We generalise our models from deterministic frameworks (like our models ACPNet 2023, ACVG 2024 to variational and diffusion-based networks: VG-LeAP and RAFI (our recent work in NeurIPS 2024, website) that incorporate the dynamics of the recording platform into the multi-step video generation process.
Advisor: Prof. Debasish Ghose
Industrial funding and collaborators: Intel, Nokia
Latent flow-based video prediction and generation
I have been working on developing novel deep learning frameworks that focus on decomposing the motion or the flow of the pixels from the static background for an improved and longer prediction of video sequences. Our work VANet showed that higher-order flow-maps can significantly improve the efficacy of deep visual prediction frameworks. Our most recent work aims at generalising latent flow maps to stochastic variational and diffusion frameworks. We have done detailed empirical studies on various datasets such as BAIR, KTH, KITTI, RoAM, SMNIST etc.
Advisor: Prof. Debasish Ghose
Robot Autonomous Motion (RoAM) Data Set
We have created a synchronised stereo image-action pair indoor robotic dataset using a custom-built Turtle-bots3 Burger robot. RoAM is currently the only publicly available dataset that provides stereo image pairs along with the synchronised motion the camera in scenarios where the camera is mounted on a moving platform, in this case, a Turtlebot3 robot. This dataset currently contains more than 300k image-action sequences of various human actions like walking, sitting, standing in various indoor environments containing stairs, lobby space, lounge etc and can be used by generative and diffusion-based video generation frameworks to benchmark improved generative architectures.
Advisor: Prof. Debasish Ghose.
Industrial funding and collaborators: Intel
Autonomous Navigation in Crowded Environments using AI/ML
Detection and tracking of human intent using multiple sensor data like camera and LiDAR with Gaussian Belief.
Advisor: Prof. Debasish Ghose
Project details coming up
Safe Control with Stochastic Control Lyapunov and Barrier Functions
I have been working on developing the mathematical framework with Control Lyapunov and Barrier functions for under-actuated stochastic systems with high relative degree that operates within desired safety bounds. Advisor: Prof. Debasish Ghose, Prof. Evangelos Theodorou
Xprize Pandemic Response Challenge 2021 (#AIforSocialGood)
As a member of the IISc-GCDSL team participating in the Xprize Pandemic Response challenge 2021, my primary responsibility was to design the learning based framework that generated the predictions and prevention of COVID-19 transmission. We are the only team from India to be selected for the final round (currently under evaluation) along with 47 other out of total 104 submissions from all over the world. Advisor: Prof. Debasish Ghose,
Mohamed Bin Zayed International Robotics Challenge (MBZIRC 2020)
As the student lead for the machine learning and computer vision group, I worked on developing computationally light and reliable visual servoing frameworks the mobile robots for various manipulation and navigation tasks, proposed by the challenge committee. We participated in the final challenge held at Adu Dhabi in 2020. Vision Team-Leader: Dr. Raghu Krishnapuram
Vision for Crisis Response
As a part of the initiative of AI for Social Good I have worked on designing a segmentation network for crisis response under the condition of natural disasters. Advisor: Prof. Debasish Ghose.
Deep Reinforcement Learning methods for hand-eye coordination
The end objective was to manipulate a 6 DOF Barret-Wam to a desired location using the Deep Q network. This project was done as part of my internship at TCS Innovation Labs, Bangalore. Project Mentor: Dr. Kaushik Das
Design of Robust Controller to Achieve Optimal Performance
Objective was to design a novel hybrid Sub-Optimal Sliding Mode Controller (SOSMC) for energy efficient navigation of autonomous mobile vehicles in unstructured environment. Up to 25\% improved efficiency in control effort was noted in the simulated environment for an autonomous underwater vehicle surveying on a 3D helical path. This work was part of my MTech thesis. Advisor: Dr. Sambhunath Nandy and Prof. Sankar Nath Shome