Synergizing Reasoning and Decision-Making in Open-World Environments
A NeurIPS 2024 Workshop
12/15/2024 (Full day)
East Building - MTG 1-3 + S.FOY, Vancouver Convention Center
Assistant Professor (incoming), NYU Courant
Generative Simulators for Real-World Decision Making
Sherry is an incoming assistant professor of Computer Science at NYU Courant and a staff research scientist at Google DeepMind. Sherry works on generative modeling from internet-scale data coupled with decision making algorithms such as imitation learning, planning, and reinforcement learning. Her research UniSim: Learning Interactive Real-World Simulators has been recognized by the Outstanding Paper award at ICLR. Sherry received her PhD in Computer Science at UC Berkeley and her Bachelor's and Master’s degree in Electrical Engineering and Computer Science at MIT.
Assistant Professor, The University of Hong Kong
Scaling Multimodal Computer Agents
Tao Yu is an Assistant Professor of Computer Science at The University of Hong Kong and serves as Director of the XLANG Lab (as part of the HKU NLP Group). His main research interest is in Natural Language Processing. He completed his Ph.D. at Yale University and was a postdoctoral fellow in the UW NLP group at the University of Washington. His research aims to build LLM/VLM-based agents that transform (“grounding”) language instructions into code or actions executable in real-world environments, including databases, web applications, and the physical world etc,. Tao is the recipient of the Google Research Scholar Award and the Amazon Research Award.
Senior Research Scientist, Google DeepMind
What's Missing for Agentic Foundation Models?
Ted Xiao is a Senior Research Scientist at Google DeepMind working on robot learning. His research agenda focuses on scaling robot learning in the real world, with a particular focus on approaches that can leverage internet-scale foundation models and methods that improve with more experience. Prior to joining Google DeepMind, Ted received his B.S. and M.S. in Electrical Engineering and Computer Science from UC Berkeley, where he was advised by Professor Claire Tomlin.
Assistant Professor, University of Washington
Social Reinforcement Learning
Natasha Jaques is an Assistant Professor of Computer Science and Engineering at the University of Washington, and a Senior Research Scientist at Google DeepMind. Her research focuses on Social Reinforcement Learning in multi-agent and human-AI interactions. During her PhD at MIT, she developed techniques for learning from human feedback signals to train language models which were later built on by OpenAI’s series of work on Reinforcement Learning from Human Feedback (RLHF). In the multi-agent space, she has developed techniques for improving coordination through the optimization of social influence, and adversarial environment generation for improving the robustness of RL agents. Natasha’s work has received various awards, including Best Demo at NeurIPS, an honourable mention for Best Paper at ICML, and the Outstanding PhD Dissertation Award from the Association for the Advancement of Affective Computing. Her work has been featured in Science Magazine, MIT Technology Review, Quartz, IEEE Spectrum, Boston Magazine, and on CBC radio, among others. Natasha earned her Masters degree from the University of British Columbia, undergraduate degrees in Computer Science and Psychology from the University of Regina, and completed a postdoc at UC Berkeley.
Partner Researcher Manager, Microsoft Research
John Langford studied Physics and Computer Science at the California Institute of Technology, earning a double bachelor’s degree in 1997, and received his Ph.D. from Carnegie Mellon University n 2002. Since then, he has worked at Yahoo!, Toyota Technological Institute, and IBM‘s Watson Research Center. He is also the primary author of the popular Machine Learning weblog, hunch.net and the principle developer of Vowpal Wabbit. Previous research projects include Isomap, Captcha, Learning Reductions, Cover Trees, and Contextual Bandit learning.
Assistant Professor, Stanford University
Structured Representations for Human-Centered Embodied AI
Jiajun Wu is an Assistant Professor of Computer Science and, by courtesy, of Psychology at Stanford University, working on computer vision, machine learning, and computational cognitive science. Before joining Stanford, he was a Visiting Faculty Researcher at Google Research. He received his PhD in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology. Wu's research has been recognized through the Young Investigator Programs (YIP) by ONR and by AFOSR, the NSF CAREER award, paper awards and finalists at ICCV, CVPR, SIGGRAPH Asia, CoRL, and IROS, dissertation awards from ACM, AAAI, and MIT, the 2020 Samsung AI Researcher of the Year, and faculty research awards from J.P. Morgan, Samsung, Amazon, and Meta.
Researcher, Shanghai Artificial Intelligence Laboratory & The University of Sydney
Building AI Society with Foundation-Model Agents
Zhenfei Yin is a Ph.D. candidate at the University of Sydney, working alongside Professor Wanli Ouyang. He is also affliated with the Shanghai AI Lab, collaborating with Dr. Jing Shao. His research centers on the goal of developing foundation-model agents in both the virtual and physical worlds. Before his doctoral studies, he worked at SenseTime's AGI Research Group on developing multi-modal foundation models, guided by Dr. Junjie Yan and Professor Xiaogang Wang.