Visual Computing 

The main focus of this research theme is to build algorithms using computer vision, computer graphics and deep learning for :

•Exploring automation in content creation through AI

•Discovering novel interactive digital representation of a physical environment

•Re-rendered reality for Metaverse

To this end, we are solving various research problems which includes creating a 3D model of the surrounding world,  building a personalized digital 3D avatar of humans, draping 3D humans with 3D garments and animating them, making humans interact with other objects in the scene, and real-time rendering of the whole dynamic scene with the spatially varying lighting and reflectance. In what follows, I show some of our recent projects in our research lab.

3D Reconstruction

We are working on various applications of multi-view geometry which include Structure-from-Motion (SfM), Dense Reconstruction, Neural Reconstruction etc. 

2D/3D Conversational Human Face/Body

AI driven 2D/3D talking face animation from speech is a challenging problem due to multi-modal information, person identity preservation, lip sync, emotions. We are actively working on solving these problem for both 2D and 3D faces. We are also working on speech-driven full body gesture animation.

Inverse Rendering

For realistic augmented reality, we need to understand the indoor lighting to situate a virtual object in a scene. To this end, we are working on estimating 3D geometry and light from a single and group of images.

Embodied Intelligence

The main focus of this research is to endow higher level of intelligence to an embodied agent so that

1. They can see (3D spatial perception),  talk (generate dialogue) ,  move and reason to plan safely in a human environment autonomously.

2. Act socially and interact emphatically.

Exploration by a Robot

A cognitive robot needs to understand the environment automatically and therefore it has to explore an area by itself. We are developing neural exploration techniques, robust visual odometry, audio visual navigation algorithms etc.

Search and Find

Given an instruction like "Go to the sink" when the robot is in bedroom, the robot needs to search and such a searching needs contextual knowledge of the current observed scene and its relation to the destination object. This is goal driven biased exploration towards unknown location and therefore a difficult problem to solve. We use GCNN to learn the object-object relationships and utlise it to derive the relation of the currently observed object with the destination object given in a speech. This relation provides the intermediate sub-goals to reach and iteratively update the path of the robot to explore toward the target object.

Making Human Understand the Robot

In the above search and find, if the robots sees something which is not correlating with the instruction, then the robot faces ambiguity. It now need to convey back to human to resolve the doubt and resolve it through appropriate dialogue exchange with human.

Robot Understand the Human - Social Robots

Treating humans as dynamic obstacles, i.e. as non-interactive agents, leads to longer trajectory. This is because robot has to always plan conservative routes avoiding intrusion into their personal space. 

Learning to Plan - Object Rearrangement

Service robots are expected to perform daily-life support tasks in indoor home environments. Tidying up cluttered objects, which is one of the most common tasks aimed at supporting humans at home. The robot should

1.Adapt to the  environment

2.Handle observation uncertainty and recognition errors

3.Do simultaneous navigation and manipulation planning