TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation

Qualitative Results on the Validation Sets

For each validation sequence, we show:

The four cameras
Ground truth semantic maps (ground and ceiling), showing the class with the highest probability
Predicted semantic maps (ground and ceiling), showing the class with the highest probability
Probability heat map for each semantic class
Ground truth elevation maps (ground min, ground max, and ceiling)
Predicted elevation maps (ground min, ground max, and ceiling)

All videos are played at 1x speed (best played in full screen)

TerrainNet-RGB+Stereo-TA

RGBD_RECURRENT_fleet3_sm_cactusrun_1_Tue_Nov__2_15-27-36_2021_utc-bev-val_DistroA.mp4

On-trail, high-speed driving

avg speed 7.8 m/s, max speed 13.4 m/s

RGBD_RECURRENT_tuesday_hill_12_2022-06-07-19-16-44-bev-val_DistroA.mp4

Off-trail, extreme elevation changes

avg speed 3.7 m/s, max speed 7.6 m/s

RGBD_RECURRENT_fleetcrl_sm_wash_5_Wed_Nov__3_17-55-42_2021_utc-bev-val_DistroA.mp4

Off-trail, rocky terrain

avg speed 2.8 m/s, max speed 7.4 m/s

RGBD_RECURRENT_tuesday_valley_2_wash_2022-06-07-18-39-56-bev-val_DistroA.mp4

Off-trail, valley

avg speed 4.2 m/s, max speed 6.7 m/s

RGBD_RECURRENT_tuesday_hill_9_2022-06-07-18-24-58-bev-val_DistroA.mp4

Off-trail, hill

avg speed 5.7 m/s, max speed 10.9 m/s

Real robot experiment

We tested TerrainNet under extreme weather conditions: steep slopes covered with deep snow, with scattered bushes and grass. The vehicle was able to complete a 1.1 km run with 2 interventions, both of which are caused by wheel slips. The maximum speed of the vehicle is 7m/s, The average speed is 3m/s.

The video on the right shows the 3rd persion view of the vehicle driving autonomously with TerrainNet and MPPI.

SnowThirdPerson_DistroA.mp4

This video shows what how TerrainNet works with planning and control on the real vehicle (best played in full screen)

The left status panel shows the vehicle's current status. You can see how the speed of the vehicle is regulated based on the semantics and terrain elevation.
The window in the middle shows the four camera feeds and the costmap. The red line points to the direction of the next goal, and the "tentacles" are the current mppi rollouts.
The window on the right shows the current elevation estimation relative to the vehicle's base.
The challenge of driving in deep snow is heavy wheel slip (we only use wheel odometry on the vehicle for state estimation). TerrainNet is able to predict a consistent local costmap even with significant odometry drift.

tour_camp4p2_DistroA.mp4