Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks

Overview

We demonstrate Sim-to-Real, zero-shot transfer to a real world quadrotor drone using liquid neural networks. Our liquid network policies, trained on single-step maneuvers collected from only indoor, photorealistic-simulated flight, generalize to multi-step hikes onboard a real hardware platform in outside environments. This work presents a sample efficient method to improve generalization and robustness to distribution shifts in Sim-to-Real visual quadrotor navigation tasks.

Training Dataset (Videos + Velocity Labels): https://zenodo.org/records/10966588

Gaussian Splatting Capture of Real-World Environment: https://zenodo.org/records/10723785

GitHub: https://github.com/alexquach/multienv_sim

Gaussian Splatting (GS), indoor environment, with single-step maneuvers

The Gaussian Splatting environment is used to render images and the PyBullet simulator is used to calculate dynamics. The task is to approach the target, then turn left if the ball is red-colored or turn right if the ball is blue-colored. Notably, we use a grainy simulation with cloudy artifacts for generating training data to demonstrate generalization ability. More details on hybrid simulation environment, task description, and trajectory design can be found in Section IV.

Video of first 20 examples of the total dataset (n=600)

In-Distribution + Two-step Hike: GS, indoor environment

Liquid (left) video shows the ability of liquid policies to complete a composition of two targets, while the LSTM (right) policies struggle. 

Videos of 10 examples from the total evaluation (n=100). Quantitative GS-to-GS results at Table I in Section V.C.2

GS-to-Real: Transfer to Real World, indoor environment:

Videos show the Liquid policies locking into the target with more stability than the LSTM policy. Turning before approaching is a common mistake with the LSTM policy.

Videos of 8 examples from the total evaluation (n=40). Quantitative GS-to-Real results at Table II in Section V.C.3

Additional indoor examples, showing two-step hikes.

GS-to-Real + Generalization: Transfer to Real World, outdoor environment:

Video shows effective transfer of Liquid policies to a different scene (outdoors).

Videos of examples from the total evaluation (n=40). Quantitative GS-to-Real + Generalization results at Table III in Section V.C.4

Additional outdoor examples, where shadows are cast over the targets.

GS-to-Real + Generalization + Five-step Hike: Transfer to Real World, outdoor environment:

(Same as overview video)

Only PyBullet simulation environment, with single-step maneuvers

Only the PyBullet simulator is used to render and calculate dynamics. More details on hybrid simulation environment, task description, and trajectory design can be found in Section IV.

Video of first 20 examples of the total dataset (n=600)

In-Distribution + 100-Step Hike: PyBullet environment

Our Liquid model successfully achieves the 100-step hike, but the LSTM model fails after the 3rd step. Both videos are at the same speed, demonstrating the relative hesitation of the LSTM model compared to the Liquid model.

Video of the best 100-step hike for both the Liquid and LSTM models (2nd Run in Fig. 5 in Section Section V.C.1).