Text, Camera, Action!
Frontiers in Controllable Video Generation
Workshop @ ICML 2024, Vienna

The past few years have seen the rapid development of Generative AI, with powerful foundation models demonstrating the ability to generate new, creative content in multiple modalities. Following breakthroughs in text and image generation, it is clear the next frontier lies in video. There has recently been remarkable progress in this domain, with state-of-the-art video generation models rapidly improving, generating visually engaging and aesthetically pleasing clips from a text prompt.

One challenging but compelling aspect unique to video generation is the various forms in which one could control such generation: from specifying the content of a video with text, to viewing a scene with different camera angles, or even directing the actions of characters within the video. We have also seen the use cases of these models diversify, with works that extend generation to 3D scenes, use such models to learn policies for robotics tasks or create an interactive environment for gameplay. 

Given the great variety of algorithmic approaches, the rapid progress, and the tremendous potential for applications, we believe now is the perfect time to engage the broader machine learning community in this exciting new research area.  The first ICML workshop on Controllable Video Generation (CVG) seeks to bring together a variety of different communities: from traditional computer vision, to safety and alignment, to those working on world models in a reinforcement learning or robotics setting. 

We are accepting submissions on the following research areas:

• Text-to-video models

• Action-controllable video models

• Style transfer and video editing

• Camera pose control and 3D models

• Methods to address safety, bias, ethical and copyright considerations.

With the following applications:

• Video generation and editing.

• Interactive experiences and games.

• World models for agent training, robotics and autonomous driving.