RESEARCH

From Generative Intelligence to Physical Intelligence

Generative AI is transforming how machines understand and generate information. The next frontier is to bring this intelligence into the physical world.

My Broader Research Vision: Human-Robot Interaction & Human-Centered Physical AI

I envision a future where embodied intelligent systems seamlessly perceive multimodal environments, comprehend context, objects, people, and actions, anticipate future states, and reason over long horizons. By remembering past experiences, continuously learning from interactions, and executing physical tasks, these systems will collaborate safely and naturally with humans. Ultimately, this will enable the deployment of robots across ubiquitous everyday environments—from households, grocery stores, and offices to critical settings like hospitals apart from various existing industrial environments.

These are some of my research outcomes with my students and research scholars published in the forums such as AAAI, NeurIPS, ICML, CVPR, ICRA , IROS etc. Feel Free to enjoy and contact me for any feedback!

Task-Oriented Grasping with Conditioning on Generative Examples ( Full paper Oral AAAI-2026):

Task-Oriented Grasping (TOG) is a challenging problem that requires an understanding of task semantics, object affordances, and the functional aspects of how an object should be held for a purpose. To address these challenges, we introduce GRIM (Grasp Re-alignment via Iterative Matching), a novel training-free framework for task-oriented grasping. Initially, a coarse alignment strategy is developed using a combination of geometric cues and the scoring of PCA-reduced DINO features. Subsequently, the full grasp pose associated with the retrieved memory instance is transferred to the aligned scene object and further refined against a set of task-agnostic, geometrically stable grasps generated for the scene object, prioritizing task compatibility. Compared to previous training-based methods, our approach is able to achieve high generalization with a few conditioning examples. (More details are available here). Publication details are available here.

Cognitive Planning for Object Goal Navigation using Generative AI Models(NeurIPS-2024, OWA workshop):

Recent advancements in Generative AI, particularly in Large Language Models (LLMs) and Large Vision-Language Models (LVLMs), offer new possibilities for integrating cognitive planning into robotic systems. In this work, we present a novel framework for solving the object goal navigation problem that generates efficient exploration strategies. Our approach enables a robot to navigate unfamiliar environments by leveraging LLMs and LVLMs to understand the semantic structure of the scene. To address the challenge of representing complex environments without overwhelming the system, we propose a 3D modular scene representation, enriched with semantic descriptions. This representation is dynamically pruned using an LLM-based mechanism, which filters irrelevant information and focuses on task-specific data. By combining these elements, our system generates high-level sub-goals that guide the exploration of the robot toward the target object. We validate our approach in simulated environments, demonstrating its ability to enhance object search efficiency while maintaining scalability in complex settings. (More details are available here). Publication details. Result Video.

Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge:

In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings. ( More details here). (Publication details).

UniTeam: Open Vocabulary Mobile Manipulation Challenge (winner of NeurIPS-2023 OVMM challenge):

This research introduces our UniTeam agent, an improved baseline for the Home Robot: Open Vocabulary Mobile Manipulation (OVMM) challenge. The challenge addresses the core tasks of navigating unfamiliar environments, manipulating novel objects, and recognizing open-vocabulary object classes to foster cross-cutting research in embodied AI across machine learning, computer vision, natural language, and robotics. In this work, we conduct a comprehensive evaluation of the provided baseline agent, identify key deficiencies across its skill stack, and significantly improve its overall performance. Notably, our enhancements target four key areas: perception (minimizing misclassifications), navigation (preventing infinite looping), picking (handling visibility changes), and placing (ensuring precise object positioning). Our UniTeam, securing top position, among around seventy teams worldwide, won the challenge, 37th NeurIPS, 2023. ( More details here). Publication details.

SplatR : Experience Goal Visual Rearrangement with 3D Gaussian Splatting and Dense Feature Matching:

In this research We present a novel approach that uses 3D Gaussian Splatting for experience goal visual rearrangement. Experience Goal Visual Rearrangement task stands as a foundational challenge within Embodied AI, requiring an agent to construct a robust world model that accurately captures the goal state. The agent uses this world model to restore a shuffled scene to its original configuration, making an accurate representation of the world essential for successfully completing the task. In this work, we present a novel framework that leverages on 3D Gaussian Splatting as a 3D scene representation for experience goal visual rearrangement task. Recent advances in volumetric scene representation like 3D Gaussian Splatting, offer fast rendering of high quality and photo-realistic novel views. Our approach enables the agent to have consistent views of the current and the goal setting of the rearrangement task, which enables the agent to directly compare the goal state and the shuffled state of the world in image space. To compare these views, we propose to use a dense feature matching method with visual features extracted from a foundation model, leveraging its advantages of a more universal feature representation, which facilitates robustness, and generalization. We validate our approach on the AI2-THOR rearrangement challenge benchmark and demonstrate improvements over the current state-of-the-art methods.( More details here). Publication details.

Figure :SplatR is an Embodied AI agent, that solves the experience goal rearrangement task by building a 3D Gaussian splat as a 3D scene representation. The agent initialized in the goal setting, collects observation and builds the Gaussian Splat to save the goal configuration. Reintroduced into the same world with shuffled object configuration, SplatR explores the scene and renders a consistent view from the Gaussian Splat. Changes in the scene are detected by the similarity between corresponding patchwise features extracted from DINOv2.

QUEST: Persistent Queries As Semantic Monitors for Drift Suppression in long horizon tracking:

Tracking points in videos is typically formulated as frame-to-frame correspon dence, where each point is matched locally to the next frame. While this works over short horizons, errors accumulate under articulation, occlusion, and view point change, leading to silent semantic drift that existing trackers cannot de tect or correct. In this work, we revisit long-horizon tracking from a monitoring perspective and introduce QueST, a monitoring-by-design framework that treats interaction-relevant entities as persistent semantic queries rather than transient point tracks. Instead of local propagation, each query attends globally over spa tiotemporal video features at every timestep, providing a stable semantic anchor across time. We further constrain query trajectories with lightweight 3D physical grounding, using geometric plausibility to suppress unbounded drift under occlu sion. We evaluate QueST on long-horizon articulated sequences from PartNet Mobility in SAPIEN and compare against RAFT-3D, CoTracker, and TAP-Net. QueST substantially reduces terminal drift achieving a 67.7% Absolute Point Er ror (APE) improvement over TAP-Net while better preserving identity over ex tended horizons. Our results show that embedding semantic monitoring directly into perception enables more reliable long-horizon tracking under distribution shift. (More details here). Publication details.

Robotic Grasp Manipulation Using Evolutionary Computing and Deep Reinforcement Learning:

Intelligent Object manipulation for grasping is a challenging problem for robots. Unlike robots, humans almost immediately know how to manipulate objects for grasping due to learning over the years. A grown woman can grasp objects more skilfully than a child because of learning skills developed over years, the absence of which in the present day robotic grasping compels it to perform well below the human object grasping benchmarks. In this paper we have taken up the challenge of developing learning based pose estimation by decomposing the problem into both position and orientation learning. More specifically, for grasp position estimation, we explore three different methods - a Genetic Algorithm (GA) based optimization method to minimize error between calculated image points and predicted end-effector (EE) position, a regression based method (RM) where collected data points of robot EE and image points have been regressed with a linear model, a PseudoInverse (PI) model which has been formulated in the form of a mapping matrix with robot EE position and image points for several observations. Further for grasp orientation learning, we develop a deep reinforcement learning (DRL) model which we name as Grasp Deep Q-Network (GDQN) and benchmarked our results with Modified VGG16 (MVGG16). Rigorous experimentations show that due to inherent capability of producing very high-quality solutions for optimization problems and search problems, GA based predictor performs much better than the other two models for position estimation. For orientation learning results indicate that off policy learning through GDQN outperforms MVGG16, since GDQN architecture is specially made suitable for the reinforcement learning. Based on our proposed architectures and algorithms, the robot is capable of grasping all rigid body objects having regular shapes. ( For more details please visit here). Publication details.

Generative model based robotic grasp pose prediction with limited dataset :

In the present investigation, we propose an architecture which we name as Generative Inception Neural Network (GI-NNet), capable of predicting antipodal robotic grasps intelligently, on seen as well as unseen objects. It is trained on Cornell Grasping Dataset (CGD) and attains a 98.87% grasp pose accuracy for detecting both regular/irregular shaped objects from RGB-Depth images while requiring only one-third of the network trainable parameters as compared to the existing approaches. However, to attain this level of performance the model requires the entire 90% of the available labelled data of CGD keeping only 10% labelled data for testing which makes it vulnerable to poor generalization. Furthermore, getting a sufficient and quality labelled dataset for robot grasping is extremely difficult. To address these issues, we subsequently propose another architecture where our proposed GI-NNet model is attached as a decoder of a Vector Quantized Variational Auto-Encoder (VQ-VAE), which works more efficiently when trained both with the available labelled and unlabelled data. The proposed model, which we name as Representation based GI-NNet (RGI-NNet) has been trained utilizing the various split of available CGD dataset to test the learning ability of our architecture starting from only 10% label data with the latent embedding of VQ-VAE to 90% label data with the latent embedding. However, being trained with only 50% label data of CGD with latent embedding, the proposed architecture produces the best results which, we believe, is a remarkable accomplishment. The logical reasoning of this together with the other relevant technological details have been elaborated in this paper. The performance level, in terms of grasp pose accuracy of RGI-NNet, varies between 92.1348% to 97.7528% which is far better than several existing models trained with only labelled dataset. For the performance verification of both the proposed models, GI-NNet and RGI-NNet, we have performed rigorous experiments on Anukul(Baxter) hardware cobot. Publication details here.

Robotic Grasp Detection By Learning Representation in a Vector Quantized Manifold:

For a robot to perform complex manipulation tasks, it is necessary for it to have a good grasping ability. However, vision based robotic grasp detection is hindered by the unavailability of sufficient labelled data. Furthermore, the application of semi-supervised learning techniques to grasp detection is underexplored. In this paper, a semi-supervised learning based grasp detection approach has been presented, which models a discrete latent space using a Vector Quantized Variational AutoEncoder (VQ-VAE). To the best of our knowledge, this is the first time a Variational AutoEncoder (VAE) has been applied in the domain of robotic grasp detection. The VAE helps the model in generalizing beyond the Cornell Grasping Dataset (CGD) despite having a limited amount of labelled data by also utilizing the unlabelled data. This claim has been validated by testing the model on images, which are not available in the CGD. Along with this, we augment the Generative Grasping Convolutional Neural Network (GGCNN) architecture with the decoder structure used in the VQ-VAE model with the intuition that it should help to regress in the vector-quantized latent space. Subsequently, the model performs significantly better than the existing approaches which do not make use of unlabelled images to improve the grasp. ( For more details please visit here)

Vision-Based Intelligent Robot Grasping Using Sparse Neural Network :

In the modern era of Deep Learning, network parameters plays a vital role in models efficiency but it has its own limitations like extensive computations and memory requirements, which may not be suitable for real time intelligent robot grasping tasks. Current research focuses on how the model efficiency can be maintained by introducing sparsity but without compromising accuracy of the model in robot grasping domain. More specifically, in this research two light-weighted neural networks have been introduced, namely Sparse-GRConvNet and Sparse-GINNet, which leverage sparsity in robotic grasping domain for grasp pose generation by integrating the Edge-PopUp algorithm. This algorithm facilitates the identification of the top K% of edges by considering their respective score values. Both the Sparse-GRConvNet and Sparse-GINNet models are designed to generate high-quality grasp poses in real-time at every pixel location, enabling robots to effectively manipulate unfamiliar objects. We extensively trained our models using two benchmark datasets: Cornell Grasping Dataset (CGD) and Jacquard Grasp ing Dataset (JGD). Both Sparse-GRConvNet and Sparse-GINNet models outperform the current state-of-the-art methods in terms of performance, achieving an impressive accuracy of 97.75% with only 10% of the weight of GR-ConvNet and 50% of the weight of GI-NNet, respectively, on CGD. Additionally, Sparse-GRConvNet achieve an accuracy of 85.77% with 30% of the weight of GR ConvNet and Sparse-GINNet achieve an accuracy of 81.11% with 10% of the weight of GI-NNet on JGD. To validate the performance of our proposed models, we conducted extensive experiments using the Anukul (Baxter) hardware cobot. ( For more details please visit here ).

Context-aware 6D Pose Estimation of Known Objects using RGB-D data:

6D object pose estimation has been a research topic in the field of computer vision and robotics. Many modern world applications like robot grasping, manipulation, autonomous nav igation etc, require the correct pose of objects present in a scene to perform their specific task. It becomes even harder when the objects are placed in a cluttered scene and the level of occlusion is high. Prior works have tried to overcome this problem but could not achieve accuracy that can be considered reliable in real-world applications. In this paper, we present an architecture that, unlike prior work, is context-aware. It utilizes the context information available to us about the objects. Our proposed architecture treats the objects separately according to their types i.e; symmetric and non-symmetric. A deeper estimator and refiner network pair is used for non-symmetric objects as compared to symmetric due to their intrinsic differences. Our experiments show an enhancement in the accuracy of about 3.2 %over the LineMOD dataset, which is considered a benchmark for pose estimation in the occluded and cluttered scenes, against the prior state-of-the-art DenseFusion. Our results also show that the inference time we got is sufficient for real-time usage. ( For more details and publication please visit here)

Designing effective power law-based loss function for faster and better bounding box regression:

Effective bounding box regression is essential for running any real-time object detection algorithm with acceptable accuracy. The currently available loss functions have issues like high computations, and sometimes they suffer from a subtle problem of plateau for non-overlapping bounding boxes, as the resultant bounding boxes are found to be far from the ground truth. In the present investigation, we have proposed a loss function with a new power-law term introduced in it for the normalized distance, which converges as fast as the Complete Intersection over Union (CIoU), but turns out to be computationally much faster than the Intersection over Union (IoU) and Generalised IoU (GIoU). The proposed function is simpler than CIoU. The incorporated power term has been optimized based on the corresponding computational time and on the sum of errors simulated for about multi-million cases, the details of which have been elaborated in the paper. The proposed Absolute IoU (AIoU)lossfunction has beensuccessfully implemented andtestedusingthestate-of-the-art object detection algorithms, such as You Only Look Once (YOLO) and Single Shot Multibox Detector (SSD) and is found to achieve significant performance improvement, using well-known metric Average Precision (AP), indicating the effectiveness of our approach

( Publication details)

Development of a robust cascaded architecture for intelligent robot grasping using limited labelled data:

Grasping objects intelligently is a challenging task even for humans and we spend a considerable amount of time during our childhood to learn how to grasp objects correctly. In the case of robots, we can not afford to spend that much time on making it to learn how to grasp objects effectively. Therefore, in the present research we propose an efficient learning architecture based on VQVAE so that robots can be taught with sufficient data corresponding to correct grasping. However, getting sufficient labelled data is extremely difficult in the robot grasping domain. To help solve this problem, a semi-supervised learning based model which has much more generalization capability even with limited labelled data set, has been investigated. Its performance shows 6\% improvement when compared with existing state-of-the-art models including our earlier model. During experimentation, It has been observed that our proposed model, RGGCNN2, performs significantly better, both in grasping isolated objects as well as objects in a cluttered environment, compared to the existing approaches which do not use unlabelled data for generating grasping rectangles. To the best of our knowledge, developing an intelligent robot grasping model (based on semi-supervised learning) trained through representation learning and exploiting the high-quality learning ability of GGCNN2 architecture with the limited number of labelled dataset together with the learned latent embeddings, can be used as a de-facto training method which has been established and also validated in this paper through rigorous hardware experimentations using Baxter (Anukul) research robot. ( for more details please visit here). For publication details please visit here.

Feature extraction and selection for emotion recognition from electrodermal activity:

Electrodermal activity (EDA) is indicative of psychological processes related to human cognition and emotions. Previous research has studied many methods for extracting EDA features; however, their appropriateness for emotion recognition has been tested using a small number of distinct feature sets and on different, usually small, data sets. In the current research, we reviewed 25 studies and implemented 40 different EDA features across time, frequency and time-frequency domains on the publicly available AMIGOS dataset. We performed a systematic comparison of these EDA features using three feature selection methods, Joint Mutual Information (JMI), Conditional Mutual Information Maximization (CMIM) and Double Input Symmetrical Relevance (DISR) and machine learning techniques. We found that approximately the same numbers of features are required to obtain the optimal accuracy for the arousal . ( More details are available here).

Robust real-time emotion detection system using CNN architecture:

As the human–robot interaction is catching eye day by day with the increase in need of automation in every field, personal robots are increasing in every area which may be coping needs of elderly people, treating autistic patients or child therapy, even in the area of babysitting the child. As robots are helping human being in all such cases, robots need to understand human emotion in order to treat human in a more customized manner. Predicting human emotion has been a difficult problem which is being solved over a decade’s time. In this paper, we have built a model which can predict human emotion from an image in real time. The network build is based on convolutional neural network which has reduced parameters by 90× from that of Vanilla CNN and also 50× from the latest state-of-the-art research carried out to the best of our knowledge. The network build is tested robustly on 8 different datasets, namely Fer2013, CK and CK+, Chicago Face Database, JAFFE Dataset, FEI face dataset, IMFDB, TFEID and custom dataset build in our laboratory having different angles, faces, backgrounds and age groups. The network achieves 74% accuracy which is an improved accuracy from the state-of-the-art accuracy with reduced computation complexity. ( For more details please visit here ).

Dynamic model of a gyroscopic wheel ( classic IEEE-IRCA ):

We develop a dynamic model of a gyroscopic wheel, an important component of Gyrover, a single-wheel robot developed at Carnegie Mellon University. The Gyrover robot consists of a single wheel, and is actuated through a spinning flywheel attached through a two-link manipulator at the wheel bearing. The flywheel can be tilted to achieve steering, and can be driven forwards and backwards to accelerate the robot. This paper focuses on developing a 3D model of the wheel part of the Gyrover. We first describe the Gyrover robot. We then develop the dynamic model of the wheel through the Lagrangian constrained generalized formulation. Finally, we implement the resulting equations of motion and present simulation results for the unactuated Gyrover in the different gravitational environments of Earth, the Moon, and Mars. ( For more details please visit here). Video of single wheel robot.

Analysis of actuation and dynamic balancing for a single-wheel robot ( Classic IEEE-IROS):

We develop a dynamic model of the steering and actuation mechanism of Gyrover, a single-wheel robot which can be considered as a single wheel, actuated through a spinning flywheel attached through a two-link manipulator at the wheel bearing and a drive motor. The spinning flywheel acts as a gyroscope to stabilize the robot, and at the same time it can achieve steering. We develop a dynamic model, investigate its motion equation, and nonholonomic constraints, and present a simulation study. The work is significant in understanding this type of dynamically stable but statically unstable system, and in developing automatic control of the system. . ( For more details please visit here).

Biologically inspired CPG based above knee active prosthesis (Classic IEEE -IROS):

The objective of the work presented here is to develop a low cost active knee prosthetic devices as real time embedded system which utilizes the available biological motor control circuit properly integrated with a Central Pattern Generator (CPG) aided control scheme. The approach is completely different from the existing Active Prosthetic devices, designed primarily as stand alone systems utilizing multiple sensors and embedded rigid control schemes. First we analyzed a fuzzy logic based methodology for offering suitable gait for an amputee, followed by formulating a suitable algorithm for designing a CPG, based on Rayleigh’s oscillator. Using the oscillator we presented a number of simulation results which showed the behavior of knee angles and hip angles and determined the stable limit cycles of the network, and compared them with the captured gaits of an individual. Subsequently, we presented a methodology about how to use CPG outputs for calculating the damping profile for controlling a prosthetic device called AMAL (Adaptive Modular Active Leg). ( For more details please visit here). (Videos of AMAL -1 AMAL-2, AMAL-3 here).

Development of Adaptive Modular Active Leg (AMAL) using bipedal robotics technology:

The objective of the work presented here is to develop a low cost active above knee prosthetic device exploiting bipedal robotics technology which will work utilizing the available biological motor control circuit properly integrated with a Central Pattern Generator (CPG) based control scheme. The approach is completely different from the existing Active Prosthetic devices, designed primarily as standalone systems utilizing multiple sensors and embedded rigid control schemes. In this research, first we designed a fuzzy logic based methodology for offering suitable gait pattern for an amputee, followed by formulating a suitable algorithm for designing a CPG, based on Rayleigh’s oscillator. An indigenous probe, Humanoid Gait Oscillator Detector (HGOD) has been designed for capturing gait patterns from various individuals of different height, weight and age. These data are used to design a Fuzzy inference system which generates most suitable gait pattern for an amputee. The output of the Fuzzy inference system is used for designing a CPG best suitable for the amputee. We then developed a CPG based control scheme for calculating the damping profile in real time for maneuvering a prosthetic device called AMAL (Adaptive Modular Active Leg).Also a number of simulation results are presented which show the stable behavior of knee and hip angles and determine the stable limit cycles of the network. ( For more details please visit here). For getting specification details of AMAL please visit here: version-1, version-2.

On Solving Inverse Kinematics of Redundant Robots Using Invertible Neural Networks with Ex-Post Density Estimation:

Learning inverse kinematics of humanoid and collaborative robots, which have inherent kinematic redundancy, is a challenging problem due to its multivalued nature. Since these robots hardly obey Pieper’s recommendation (Pieper and Roth 1969), solutions to the inverse kinematics problem cannot always be obtained analytically. Recently, Invertible Neural Networks (INNs) have found success in solving such illposed inverse problems. In this work, we empirically show that density constraints on the latent variables while training INNs could be replaced by an ex-post density estimation step. The advantage is twofold; the latent variables could have an arbitrarily complex distribution, and posterior mismatch is no longer an issue. Through experiments on learning the inverse kinematics of planar redundant serial robotic manipulators, we validate the efficacy of our approach. For more details please visit here.

Page updated

Report abuse