XRNeRF
Second Workshop on Advances in
Radiance Fields for the Metaverse
CVPR Workshop 2024
June 18, 2024 8:30 AM
Seattle Convention Center, Seattle
Overview
A longstanding problem in computer graphics is the realistic rendering of virtual worlds. Generation of highly realistic 3D worlds at scale is an important piece of the Metaverse puzzle. However, creating such worlds and the content inside it can be costly and time consuming.
In 2020, new techniques were developed in neural volume rendering, also known as NeRF (Neural Radiance Fields), which brought an explosion of new work that has direct applicability to the future metaverse. In CVPR 2023, there were more than 100+ published papers aimed towards improving the fidelity, efficiency, scalability and application of RFs. With the recent development of Gaussian Splatting, we are also seeing alternative approaches to neural networks driving advancements as well. We believe that these techniques represent some of the most viable solutions to address the growing content needs of Metaverse.
There have been many recent advances in Radiance Fields (RF- NeRF, Gaussian Splatting, etc.) that have enabled them to be a strong content generation tool. Some of these advances include but are not limited to a) ability to represent arbitrary scenes including unbounded scenes at city scale, b) ability to run on mobile devices, c) generative RF through natural language, d) spatio-temporal RF of videos, and e) stylizing and editing RF. This workshop is an opportunity to showcase work on RF that expands upon key areas that further the Metaverse development.
The aim of this workshop is to bring industry innovators and academic leaders in the world to discuss the problems, applications, and general state of this technology. Specifically, we would like to cover recent advances in radiance fields that focus on three areas that need significant gains for the metaverse: scale, efficiency, and fidelity.
Call for Speakers
To help drive the development of the metaverse, we are looking for speakers around these key areas associated with RF (Radiance Fields):
Generative RF: Being able to generate objects by just textual cues (or audio) is going to enable large amounts of virtual objects and scenes in Metaverse. DreamFusion, Dreambooth, Zero123 have shown this to work on individual objects.
Efficient/Real-time RF: NeRF has been trained to run on standard mobile devices. Recent RF techniques such as Gaussian Splatting can render real-time radiance fields. This level of efficiency can allow for the rendering of these worlds on low-compute devices in the future, such as future AR/VR devices which are significantly limited by their compute. It is also exciting to see RF being generated from limited viewpoints.
Dynamic RF: RF of videos is an exciting area of research - achieving spatio-temporal consistency still remains a challenge. Dynamic RF can enable realistic simulation inside Metaverse.
Stylization of RF: Ability to edit RF through large language models such as InstructNeRF2NeRF can unlock exciting metaverse applications for human avatars.
Speakers
Jon Barron
Senior Staff Research Scientist, Google Research
Richard Newcombe
VP, Research Science, Meta
Shubham Tulsiani
Assistant Professor, Carnegie Mellon University
Lingjie Liu
Assistant Professor at the University of Pennsylvania
Jonathon Luiten
Research Scientist, Meta
Peter Hedman
Senior Research Scientist, Google
Georgios Kopanas
Research Scientist, Google
Organizers
Fernando De la Torre (CMU)
Fernando De la Torre is an Associate Research Professor at Carnegie Mellon University. He is the author of more than 200 peer-reviewed publications in top conferences and journals in the topic of computer vision and machine learning. He served as associate editor of IEEE Transactions on Pattern Analysis and Machine Intelligence, and regularly acts as an area chair for ECCV, CVPR and ICCV. He founded FacioMetrics, which was acquired by Facebook in 2016. At Facebook he led the efforts in developing the technology for facial feature tracking, person segmentation and other real-time on device technology for people augmentation in Messenger, Instagram, Facebook and Portal.
Angela Dai (Technical University of Munich)
Angela Dai is professor at Technical University of Munich, leading the 3D AI Lab. Her research focuses on attaining a 3D understanding of the world around us, capturing and constructing semantically-informed 3D models of real-world environments. This includes 3D reconstruction and semantic understanding from commodity RGB-D sensor data, leveraging generative 3D deep learning towards enabling understanding and interaction with 3D scenes for content creation and virtual or robotic agents.
She received her PhD in computer science from Stanford in 2018 and her BSE in computer science from Princeton in 2013. Her research has been recognized through a ZDB Junior Research Group Award, an ACM SIGGRAPH Outstanding Doctoral Dissertation Honorable Mention, as well as a Stanford Graduate Fellowship.
Peter Vajda (Meta Reality Labs)
Peter is a Research Manager in computer vision at Meta. Before joining Meta in 2014, he was Visiting Assistant Professor in Professor Bernd Girod’s group in Stanford University, Stanford, USA. He was working on a personalized multimedia system and mobile visual search. He received M.Sc. in Computer Science from the Vrije Universiteit, Amsterdam, Netherlands and an M.Sc. in Program Designer Mathematician from Eötvös Loránd University, Budapest, Hungary. He completed his Ph.D. with Prof. Touradj Ebrahimi at the Ecole Polytechnique Fédéral de Lausanne (EPFL), Lausanne, Switzerland, 2012.
Daeil Kim (Meta Reality Labs)
Daeil is an engineering manager leading synthetic data efforts at Meta. Before joining Meta in 2021, he was the CEO / Founder of AI.Reverie, a startup focused on developing a platform for synthetic data generation for a variety of real world computer vision problems and before this he was an ML scientist at the New York Times. He received his Ph.D in computer science from Brown University in 2014, where he focused on scalable machine learning algorithms for Bayesian nonparametric models with Erik Sudderth. He has published several papers at NeurIPS, ICML, AISTATS, and his academic work also spanned the areas of statistical neuroimaging techniques with a focus on neuro-psychiatry with publications in Neuroimage, Human Brain Mapping, and several others.
Aayush Prakash (Meta Reality Labs)
Aayush is an engineering manager who leads the machine learning team within the synthetic data organization at Reality Labs, Meta. His group works on problems at the juncture of machine learning, computer vision and computer graphics. They tackle challenges in domain adaptation, neural rendering and other sim2real problems for mixed reality. Before joining Meta, he was the head of machine learning at synthetic data startup, AI Reverie. Prior to this, he worked at Nvidia where I spent 6 years on synthetic data research in computer vision. While at Nvidia, his group delivered some of the prominent works in synthetic data creation. He graduated with a B.Tech in E&ECE from Indian Institute of Technology (IIT) Kharagpur, India, in 2010, and MASc in Computer Engineering from University of Waterloo, Canada, in 2013.