Continuity Editing for 3D Animations

This work has been partly done during my PhD thesis at Inria Rennes from 2010 to 2013. We initiated this project in 2011, with Mathieu Chollet's M.S. internship, co-supervised by Dr Marc Christie (Inria Rennes) and Dr Rémi Ronfard (Inria Grenoble), and with whom I have closely collaborated on this work. It has then been taken over and studied more extensively during Quentin Galvane's PhD thesis, co-advised by Dr Marc Christie and Dr Rémi Ronfard, and whith whom I have also collaborated on this work. This was during my Postdoc at Inria Grenoble in 2013-2014. These different steps of this work lead to 1 poster (ACM SCA 2011) and 3 publications (ICIDS 2011, WICED 2012, AAAI 2015).

Preliminar work: A Computational Model of Film Grammar and Film Editing (chapter 4.3 of my PhD thesis)

Film editing is the art of composing multiple shots into a coherent movie that (i) reveals important actions in the scene, (ii) respects continuity of screen composition and (iii) keeps the viewer interested. We propose an optimization framework for selecting shots and cuts while the narrative unfolds, based on a relatively simple scoring scheme driven by working practices of film and television [Murch86,Thom93]. Building upon an existing cinematography framework [Lino 2010], we designed an automated camera planner which generates virtual camera candidates using cinematic idioms. We then cast the problem of film editing as selecting a path inside an editing graph which consists of a collection of evolving film takes (a take is a continuous sequence of images from a given camera) and precisely deciding when to cut in and out of film takes. A node of this graph represent the shooting, during a fragment of time, of the story through a given candidate camera viewpoint, and an arc represents a transition between a camera candidate at time i and a camera candidate at time i+1; transitioning to the same camera candidate means that we extend the current shot, and transitioning to another candidate camera means that we make a cut and begin a new shot. In contrast to related work, we also account for a precise enforcement of pacing (rhythm at which cuts are performed) through a well-founded model of shots durations (a log-normal law).

We extend the system to enable users to select preferred shots at preferred moments in the movie. These key shots are used as constraints (mandatory nodes in the path) and our process automatically computes appropriate sub-paths in-between these key-shots.

In this work, we introduce a novel framework for virtual cinematography and editing which adds an evaluation function to previous approaches. Preliminary results demonstrated that our approach was efficient in separating correct from incorrect shot sequences in complex narratives with many actors and actions, and was thus appropriate for future research in film-mediated interactive storytelling.

This work has been partly funded by the European Commission under grant agreement IRIS (FP7-ICT-231824).

Publications:

• Lino, Chollet, Christie, Ronfard. Automated Camera Planner for Film Editing Using Key Shots. In ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 2011.
• Lino, Chollet, Christie, Ronfard. Computational Model of Film Editing for Interactive Storytelling. In International Conference on Interactive Digital Storytelling, 2011.
• Christie, Lino, Ronfard. Film Editing for Third Person Games and Machinima. In Workshop on Intelligent Cinematography and Editing, 2012.

Continuity Editing as a Path Search under a Semi-Markovian Hypothesis

As presented in our preliminar work, we encode the continuity editing process as a search for the optimal path through an editing graph. We then propose an optimization method which makes use of dynamic programming to minimize, under a semi-Markovian hypothesis, the errors made along our three criteria: (1) the quality of the shots (with respect to the unfolding actions), (2) the respect of continuity editing rules and (3) the respect of a cutting rythm (still a log-normal law).

The inputs of this system are (i) an ordered list of durative world events, and (ii) a list of precomputed rushes from different cameras filming the scene. The output of our system is a movie, described as an edit decision list (EDL) defined as a sequence of shots. To do so, our method procedurally annotates the screen projection of relevant on-screen content (screen positions, motions, visibility of main and secondary characters or objects) through rushes. Our editing algorithm is further driven by the proper conveyance of important actions and by the enforcement of both the visual continuity and of an input cutting rhythm. Note that, our method has the advantage that it could equally been applied to any set of raw footage (using any unedited rushes, from either real cameras, virtual cameras, or both), as soon as one provides the same set of annotations on these rushes.

Our method has been validated through a user study on the perception of an edit quality, in function of the method used to edit, from the same set of footage, a short movie. We compared our method both to degraded versions of our algorithm (removing one component, (1) (2) or (3)), and to the original sequence of the movie. This study involved 21 participants, and demonstrated that each of our three criteria significantly improve the perceived editing quality, and that the perceived quality of an edit that was made by an expert filmmaker is not significanlty better than an edit that was computed by our method.

This work has been partly funded by the French Research Agency (ANR) through the Chrome Project, and by the European Research Council through the Expressive grant.

Publication:

• Galvane, Ronfard, Lino, Christie. Continuity Editing for 3D Animations. In AAAI Conference on Artificial Intelligence, 2015.

Illustrations and videos