About

From 2021 to 2023, I worked at Waymo Research Team, an R&D department lead by Dragomir Anguelov. At Waymo, I focused on developing cutting edge technologies on camera-based scene understanding, city-scale reconstruction, and sensor simulation, using deep generative models and neural fields. During this period, most work I co-led/contributed to have been given back to the open-sourcing community through dataset releases (Block-NeRF and neural assets) and open challenge (4D camera segmentation). Prior to this, I worked at R&D Team between 2019 and 2021, Uber Advanced Technologies Group (or Uber ATG Toronto), in close collaboration with Raquel Urtasun. At Uber, I mainly contributed to research on simulation, 3D reconstruction,  and behvior prediction in the self-driving domain. 

I received my PhD degree from the University of Michigan in 2019. At UofM, I focused on cutting-edge research topics lie in the intersection of deep generative models and representation learning with structured and multimodal data, under the supervision of Honglak Lee. Prior to graduate school, I received my Bachelor's degree from ACM Honored Class (交大ACM班), Shanghai Jiao Tong University in 2014. I was a visiting undergraduate student at Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University (Singapore) in 2014, advised by Junsong Yuan. I also spent time working as an undergraduate research student at MOE-MSR Joint Key Lab of Intelligence Computing (BCMI Lab), advised by Liqing Zhang.

I have also spent my time interning at Adobe Research (San Jose, 2017) and Google Brain & X (Mountain View, 2016, 2017, 2018, & 2019). I feel very honored to receive Adobe Research Fellowship and Google PhD Fellowship which support my PhD research.

[Google Scholar] [Github] [Twitter] [Linkedin]

General Research Interests

My primary research interests lie in the intersection of deep representation learning and generative modeling with structured and multimodal data. More specifically, my research has been focused on three related topics: (1) controllable visual content generation and manipulation, (2) self-supervised learning of object shape and motion, and (3) scalable and robust perception simulation for robotic applications. Besides, I have a broad range of interests in developing deep generative models for vision, graphic, and robotic applications. 

For a high-level summary of my research agenda, please see the following talk slides:

* PhD dissertation talk: "Learning Deep Controllable and Structured Representations for Image Synthesis, Structure Prediction, and Beyond", June 2019.

* Invited talk at KAIST School of Computing Colloquium: "Learning Deep Controllable and Structured Representations for Visual Content Synthesis and Manipulation", March 2021.

* "Learning Deep Controllable and Structure Representations for Autonomous Driving Simulation", more recent work.

Announcement

* 11/2023: Left Waymo after more than 2.5 years with the Research team. Look forward to new adventures!

* 03/2023: Two papers were accepted by CVPR 2023, including our GINA-3D work on 3D asset synthesis in the wild for autonomous driving.

* 07/2022: Two papers were accepted by ECCV 2022, including our large-scale video panoptic segmentation benchmark [link][video].

* 04/2022: Check out our CVPR 2022 work Block-NeRF, the largest neural scene representation to date. We released a subset of data captured in the SF Mission Bay neighborhood for the purpose of replication. Please download our dataset here.

* 02/2022: Co-host ICLR 2022 Workshop on Socially Responsible Machine Learning.

* 09/2021: Check out our CoRL 2021 work on multi-sensor adversarial robustness for self-driving: Multi-Sensor Adv.

* 06/2021: Our CVPR 2021 paper on camera sensor simulation for self-driving has been nominated for the best paper candidate.

* 05/2021: Honored to be one of the CVPR 2021 outstanding reviewers.

* 02/2021: Check out our CVPR 2021 work on sensor simulation for self-driving: GeoSim (camera simulation) and S3 (human modeling and animation).

* 01/2021: After a wonderful journey with Uber ATG Toronto, I've headed for new adventures!

* 06/2020: Check out Pytorch implementations of our ECCV 2020 work on Github (PT2PC and SemanticAdv).

* 10/2019: Honored to be one of the top NeurIPS 2019 reviewers.

* 09/2019: I joined Uber Advanced Technologies Group (ATG) as an AI research scientist.

* 05/2019: I volunteered for AI4All summer program at the University of Michigan in Summer 2019.

* 03/2019: Check out the Pytorch implementation of our NeurIPS 2018 paper on Github.

* 01/2019: Check out the TensorFlow implementation of our ECCV 2018 paper on Github.

* 11/2018: Check out the Project page of our NeurIPS 2018 paper.

* 08/2018: Recipient of Adobe Research Fellowship and Google PhD Fellowship. I really appreciate the support from my PhD advisor, committee members at the UofM, collaborators in academia and industry.

* 07/2018: Check out the Project page of our ECCV 2018 paper.

* 01/2018: Check out the Project page of our ICRA 2018 paper.

* 07/2017: Check out the TensorFlow implementation of our NeurIPS 2016 paper on Github.

* 12/2016: Check out the Torch implementation of our NeurIPS 2016 paper (unsupervised learning of 3D shape generation) on Github.

* 10/2016: Check out the Torch implementation of our ECCV 2016 paper (conditional image generation) on GitHub.

Education & Research Experience (full in Mandarin)

(03/2021-- 11/2023) Research Scientist@Waymo, Mountain View

(09/2019 -- 01/2021) AI Research Scientist@Uber ATG, San Francisco

(06/2018 -- 04/2019) Student Researcher Intern@Google Brain, Mountain View 

(06/2017 -- 09/2017) Research Intern@Adobe Systems Inc, San Jose

(11/2016 -- 06/2017) Research Intern@Google Brain, Mountain View

(08/2014 -- 07/2019) PhD@University of Michigan, Ann Arbor

Undergraduate Research Projects

Professional Activities

* Conference Reviewer: NeurIPS (2016 -- 2023), ICML (2018 -- 2024), ICLR (2018 -- 2023), CVPR (2019 -- 2024)

ICCV (2019, 2021 & 2023), ECCV (2020 & 2022), WACV (2019 -- 2020), SIGGRAPH (2020 -- 2021), ICRA 2021, IROS 2021, CoRL (2018 & 2022)

* Journal Reviewer

    - IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

    - IEEE Transactions on Image Processing (TIP)

    - IEEE Transactions on Multimedia (TMM)

    - IEEE Robotics and Automation Letters (RA-L)

Invited Talks

(06/2023) CVPR 2023 workshops, Vancouver, Canada

(06/2022) CVPR 2022 workshops, New Orleans, US

(03/2021) Tech Talk@KAIST, Daejeon, Korea

(02/2021) Tech Talk@Cruise Automation, San Francisco, US

(01/2021) Tech Talk@Adobe Research, San Jose, US

(01/2021) Tech Talk@Waymo Research, Mountain View, US

(03/2020) Tech Talk@Stanford Univ, Palo Alto, US

(02/2020) Tech Talk@Uber AI, San Francisco, US

(08/2019) Tech Talk@Didi Lab, Mountain View, US

(04/2019) Tech Talk@Google AI, Mountain View, US

(03/2019) Tech Talk@Facebook AI Research, Menlo Park, US

(03/2019) Tech Talk@Waymo Research, Mountain View, US

(02/2019) Tech Talk@Uber ATG, Toronto, Canada

(01/2019) Tech Talk@Adobe Research, San Francisco, US

(01/2019) Tech Talk@Nvidia Research, Santa Clara, US

Honors & Awards

(04/2018) Google PhD Fellowship in Machine Learning (8 selected world-wide) [News from Google AI Blog]

(02/2018) Rackham Predoctoral Fellowship

(12/2017) Adobe Research Fellowship (10 selected in USA) [News from Adobe Research Blog]

(12/2012) Academic Excellence Scholarship (Second-class) of Shanghai Jiao Tong University in Year 2011-2012

(11/2012) Scholarship from Kai Yuan in Year 2011-2012

(04/2012) Scholarship from Tung OOCL in Year 2010-2011

(12/2011) Academic Excellence Scholarship (Second-class) of Shanghai Jiao Tong University in Year 2010-2011

(Year 2008, 2009, & 2010) First Prizes in National Olympiad in Informatics in Provinces

Selected Publications (see full list)

* GINA-3D: Learning to Generate Implicit Neural Assets in the Wild

Bokui Shen, Xinchen Yan, Charles R. Qi, Mahyar Najibi, Boyang Deng, Leonidas J. Guibas, Yin Zhou, Dragomir Anguelov

To appear: In Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023

[paper][arxiv][project page][video][talk]

* Waymo Open Dataset: Panoramic Video Panoptic Segmentation

Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Yukun Zhu, Liang-Chieh Chen, Henrik Kretzschmar, Dragomir Anguelov

In European Conference on Computer Vision (ECCV), Tel-Aviv, Israel, 2022

[paper][arxiv][website][video][code]

* Block-NeRF: Scalable Large Scene Neural View Synthesis

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar

In Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, US, 2022 (Oral Presentation)

[paper][arxiv][project page][dataset]

* GeoSim: Photorealistic Video Simulation with Geometry-Aware Composition for Self-Driving

Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Siva Manivasagam, Shangjie Xue, Ersin Yumer, Raquel Urtasun

In Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, US, 2021 (Best Paper Candidate)

[paper][supp][arxiv][project page][video]

* S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling

Ze Yang, Shenlong Wang, Siva Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, Raquel Urtasun

In Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, US, 2021

[paper][supp][arxiv][video]

* PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

Kaichun Mo, He Wang, Xinchen Yan, Leonidas J. Guibas

In European Conference on Computer Vision (ECCV), Glasgow, UK, 2020

[paper][arxiv][project page][code]

* SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing

Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, Bo Li

In European Conference on Computer Vision (ECCV), Glasgow, UK, 2020

[paper][arxiv][project page][code]

* Learning Hierarchical Semantic Image Manipulation through Structured Representations

Seunghoon Hong, Xinchen Yan, Thomas Huang, Honglak Lee 

In Advances in Neural Information Processing Systems (NeurIPS), Montreal, Canada, 2018

[paper][arxiv][project page][code]

* MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics

Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, Honglak Lee

In European Conference on Computer Vision (ECCV), Munich, Germany, 2018

[paper][supp][arxiv (full version)][project page][code]

* Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations

Xinchen Yan, Jasmine Hsu, Mohi Khansari, Yunfei Bai, Arkanath Pathak, Abhinav Gupta, James Davidson, Honglak Lee

In Proceedings of International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018

[paper][arxiv][project page][patent]

* Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, Honglak Lee

In Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain, 2016

[paper][arxiv (full version)][project page]

Torch Implementation: [data][code]

TensorFlow Implementation: [code]

Note, if you want to use your own camera matrix, please refer to the section " with your own camera" here.

* Generative Adversarial Text-to-Image Synthesis

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee

In Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, USA, 2016

[paper][supplementary material]

Torch Implementation: [code]

* Attribute2Image: Conditional Image Generation from Visual Attributes

Xinchen Yan, Jimei Yang, Kihyuk Sohn, Honglak Lee

In European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 2016

[paper][arxiv][project page]

Torch Implementation: [code]

* Learning Structured Output Representation using Deep Conditional Generative Models

Kihyuk Sohn, Xinchen Yan, Honglak Lee

In Advances in Neural Information Processing Systems (NeurIPS), Montreal, Canada, 2015. 

[paper][supplementary material][code]

Teaching Assistant

(09/2011 - 01/2012) Teaching Assistant of C++ Programming (Fall 2011)

(03/2012 - 06/2012) Teaching Assistant of Data Structures (Spring 2012)

(09/2012 - 12/2012) Assistant Lecturer of Math Foundations in Computer Science (Fall 2012)