Autonomous drone Cinematography

tools and techniques to transform drones into cinematographers

Are you ready to create the next blockbuster film? Imagine exploring new perspectives and pushing the boundaries of your artistic vision like never before. With our innovative solutions, you'll have the power to test artistic effects and easily capture breathtaking shots. We offer various approaches to make drones autonomous when filming, revolutionizing the way you bring your cinematic visions to life. Say goodbye to limitations and hello to limitless creativity. Let's redefine filmmaking together.

We're excited to introduce CinemAirSim [1], an innovative extension of the renowned drone photorealistic simulator, AirSim. What sets CinemAirSim apart is its integration of drones equipped with cinematic cameras and an extended API, allowing real-time control over all parameters. From focal length and focus distance to aperture settings, CinemAirSim empowers users with the ability to manipulate various cinematographic properties seamlessly. This breakthrough feature opens up a world of possibilities for both researchers and cinematographers alike. With CinemAirSim, users can experiment with different lens configurations and develop advanced control algorithms to enhance drone autonomy within a photorealistic environment. 

Utilizing CinemAirSim as our foundation, we proudly introduce CineMPC [2] – an advanced algorithm designed to autonomously control filming drones with unparalleled precision. This work was first presented as a conference paper [3] and then extended as a journal paper, published in the journal Transactions on Robotics (T-RO) [2]. Leveraging a non-linear Model Predictive Control (MPC) technique, CineMPC orchestrates both the intrinsics and extrinsics of the cameras embedded within the drones, all with the singular goal of capturing the perfect shot autonomously. This groundbreaking algorithm optimizes the position of targets within the frame, adjusts the depth of field, and aligns shots with artistic and compositional guidelines drawn from cinematographic literature. With CineMPC, the art of filmmaking reaches new heights, effortlessly blending technical expertise with artistic vision to deliver cinematic excellence.

Building upon the foundations laid by CineMPC, we present our latest innovation: CineTransfer [4]. This advancement in cinematographic technology allows users to effortlessly imbue their footage with the distinctive style of a single example video. Through sophisticated neural networks, CineTransfer extracts stylistic elements from the focus and composition of the input video, discerning key features such as the positioning of the main subject on the screen. Leveraging the dynamic capabilities of our drones, originally developed for CineMPC, CineTransfer seamlessly transfers the acquired style to capture new footage in a similar or entirely different scene. With CineTransfer, the boundaries of cinematic creativity are boundless, enabling filmmakers to infuse their productions with unparalleled depth and character effortlessly.

References

[1] Pueyo, P., Cristofalo, E., Montijano, E., & Schwager, M. (2020). CinemAirSim: A Camera-Realistic Robotics Simulator for Cinematographic Purposes. IROS 2020 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9341066, https://ieeexplore.ieee.org/abstract/document/9341066/

[2] Pueyo, P., Dendarieta J., Montijano, E., Murillo, A. C., & Schwager, M. (2024). CineMPC: CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene Composition. Accepted to T-RO v40. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10398502, https://ieeexplore.ieee.org/abstract/document/10398502

[3] Pueyo, P., Montijano, E., Murillo, A. C., & Schwager, M. (2021). CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography. Accepted to ICRA 2022. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9811827,  https://ieeexplore.ieee.org/abstract/document/9811827

[4] Pueyo, P., Montijano, E., Murillo, A. C., & Schwager, M. (2021). CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example. Accepted to ICRA 2023. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10342280,  https://ieeexplore.ieee.org/abstract/document/10342280

Source code

Cinemairsim: https://github.com/ppueyor/CinemAirSim
CineMPC: https://github.com/ppueyor/CineMPC_ros