This was one of the primary works towards my Master Thesis. Here is a link.

We are using Machine Learning techniques to animate any arbitrary character/agent dance to songs. The main contribution of the work is that our method does not require any motion capture data and can generate dance moves for a given character automatically.  Also, we use a physics-based controller and train the character to dance in physical environments, thereby avoiding various artifacts common with kinematic based controllers, such as foot sliding, unnatural transitions, etc.

We are training a Generative Adversarial Network to generate a corpus of dance moves through physical simulations. The character is then trained to interact with physical environments using Deep Reinforcement Learning, and then dance to the beats of songs while maintaining balance.  Finally, we construct a motion-graph for the character containing the full set of moves it has been trained on.

We are using various music information retrieval techniques to extract various features from the song, and then using a graph search method to generate choreography for the given song.

We published this at MIG 2021, and won best paper award!

We are training arbitrary characters to learn locomotion skills using Deep Reinforcement Learning. 

Our method does not require motion capture data, and proposes an automatic way of extracting the most important modes/ low-dimensional coactivations of a character given its anatomy. 

The central idea of our approach is actuating different joints of the robot with random motor commands for short periods of time. We are evaluating various methods such as PCA and Variational Autoencoders (VAE) to construct this low dimensional latent space, and then use it to train locomotion skills using DRL. Our results show considerably improved locomotion gaits for the character, which are otherwise difficult to achieve when trained in the high dimensional joint-action space.

This is my final project for the course CPSC 8420: Advanced Machine Learning, with Dr. Kai Liu.

Functional Magentic Resonance Imaging (fMRI) is a brain scan method that measures neural activity in the brain across a temporal axis. The data is represented as a 4D tensor (3D space + 1D time) with close to 1 million features per scan. With very limited number of images available compared to the number of features, learning useful representations of the data remains a challenging task with traditional Machine Learning approaches. 

In this project, we explore the idea of using Deep Learning based unsupervised training methods to obtain latent spatio-temporal embeddings of the data. Specifically, we use 3D Convolution-based Autoencoders to extract spatial features of the data, and an Attention-based Sequence to Sequence encoder-decoder recurrent network to capture the temporal characteristics. We then use these latent representations for supervised prediction of Autism Spectrum Disorder (ASD) using traditional machine learning techniques such as Kernel SVM, logistic regression, etc. 

We are working on a scalable movie recommendation system that can generate recommendations as well as reasonings for the same.  First, a movie knowledge graph represented as a multi-hypergraph network is constructed. Here, each node represents a movie, and different nodes are connected by hyperedges of various modalities such as cast, director, and plot-keywords.  The additional information is mined using TMDBpy.

The embedding of each movie is generated using a hypergraph convolution method, parameterized by a neural network. We aggregate local features of each movie with respect to its modalities (cast, genre, keywords, users) to generate movie embeddings. Similarly, user embeddings are generated by aggregating the information from adjacent movies and co-cited users.  A neighborhood sampling method is used for computation graph construction, similar to GraphSAGE, which allows the parameters to be trained in mini-batches and makes the proposed method scalable to large networks.  

The method is tested on the MovieLens-1M dataset, and it produces promising results with various commonly used evaluation metrics, such as Precision@K,  Recall@K,  F1@K, Hit@K, and NCDG@K.