Robotics
Robotics
Embedded Files
Build a generalist robot that operates in complex and unstructured environment
Traditional robotics systems operate in the "cage" environment, in which human operators maintain a safe distance to robots. However, with the increasing demand of AI-powered robots, the robots nowadays requires completing tasks in a human-centric environment, such as people's home. As a result, we will need to have an AI system that can reason, think, and interact with a complex and unstructured environment.
To enable generalization to complex and unseen tasks, We train vision language action (VLA) models and World Models (WM) with heterogeneous dataset such as real world teleop data, simulation dataset, and human videos.
Work in progress
To benchmark the progress of planning and reasoning in embodied multi-agent tasks, we develop Partnr and Habitat 3.0 with high-quality simulation realism and fast rendering.
M. Chang, G. Chhablani, A. Clegg, M. Cote, R. Desai, M. Hlavac, V. Karashchuk, J. Krantz, R. Mottaghi, P. Parashar, Tsung-Yen Yang, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In ICLR, 2024
X. Puig, E. Undersander, A. Szot, M. Cote, Tsung-Yen Yang, R. Partsey, R. Desai, A. Clegg, M. Hlavac, S. Min, et al. Habitat 3.0: A co-habitat for humans, avatars, and robots. In ICLR, 2023
To enable System 1 (fast and reactive), System 2 (planning and reasoning) robotics control stack, we build Adaptive Skill Coordination (ASC) algorithm for accomplishing long-horizon tasks such as mobile pick-and-place.
N. Yokoyama, A. Clegg, J. Truong, E. Undersander, Tsung-Yen Yang, S. Arnaud, S. Ha, D. Batra, and A. Rai. Asc: Adaptive skill coordination for robotic mobile manipulation. IEEE Robotics and Automation Letters, 2023
To enable robotics research in the community, we develop HomeRobot: an affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks.
S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, Tsung-Yen Yang, V. Jain, A. Clegg, J. Turner, et al. Homerobot: Open-vocabulary mobile manipulation. In CoRL, 2023
Finally, we develop a "digital" computer agent that solves programming issues via VSCode using self-learning approaches.
Work in progress
LLM AI Agents Trained with RL
LLM AI Agents Trained with RL
Build an AI agent that acquires knowledge by interacting with the world
Many autonomous systems, such as personalized robotic assistants, are required to interact with people. A natural way to communicate with robots is via natural languages. However, human languages are inherently complex: how autonomous systems can effectively understand humans' instructions and explore the environment safely?
We propose a computational model in the context of instruction following: the autonomous system is designed to directly reason about the structure of the instruction to learn robust and interpretable representations for language grounding.
Tsung-Yen Yang, Andrew S. Lan, Karthik Narasimhan. "Robust and Interpretable Grounding of Spatial References with Relation Networks." Findings of Conference on Empirical Methods in Natural Language Processing (Findings of EMNLP), 2020, paper
Tsung-Yen Yang*, Michael Hu*, Yinlam Chow, Peter J. Ramadge, Karthik Narasimhan. "Safe Reinforcement Learning with Natural Language Constraints." arXiv, 2020, paper, demo (*Equal contribution)
An AI agent with provable safety guarantees during deployment and training
Many autonomous systems, such as self-driving cars and industrial robots, are complex. In order to deal with this complexity, researchers are increasingly using reinforcement learning (RL) for designing control policies. However, there is one issue that limits RL's widespread deployment in the real-world system: how autonomous systems can learn safely?
We propose an algorithm for learning constraint-satisfying policies in the context of reinforcement learning with constraints: the autonomous system maximizes a reward while using projection onto constraint sets to ensure constraint satisfaction with provable performance guarantees.
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J. Ramadge. "Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies." International Conference on Machine Learning (ICML), 2021, paper
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J. Ramadge. "Projection-Based Constrained Policy Optimization." International Conference on Learning Representations (ICLR), 2020, paper, website
Time-series Data Analysis
Time-series Data Analysis
I develop a privacy-preserving technique using generative adversarial networks to learn informative representations without sacrificing the user's privacy. The approach is tested in the Massive Open Online Courses (MOOCs) dataset and shown to be effective.
Tsung-Yen Yang, Christopher Brinton, Prateek Mittal, Mung Chiang, and Andrew Lan. "Learning Informative and Private Representations via Generative Adversarial Networks." IEEE International Conference on Big Data (Big Data), 2018
I also collaborate with other researchers on machine learning for education.
Tsung-Yen Yang, Christopher G. Brinton, Carlee Joe-Wong, and Mung Chiang. "Behavior-based grade prediction for MOOCs via time series neural networks." IEEE Journal of Selected Topics in Signal Processing, 2017
Google Sites
Report abuse