In Halfcheetah, the primary risk to the agent is flipping over on its back and getting stuck. The policy with small standard deviation adopts a gait which hugs the ground, preventing it from flipping over, while the policy with large standard deviation performs a risky behavior which frequently tips up into the air, eventually causing it to fail.

In Hopper, the policy with low standard deviation of the post-update return distribution performs a stable, well-balanced, upright gait. The policy with the high standard deviation corresponds to an unstable curved gait -- we see this policy eventually trip and fall.

In Ant, the agent incurs the risk of flipping over and causing termination. The policy on the right goes too fast without appropriately balancing against flips. This causes the mean of the post-update return distribution to be the similar.

In walker, the agent has to walk without falling. The agent on the left has a more upright gait, while the agent on the right leans backwards to make potentially wider steps but being unsafe in a way that lets it fall at the end of the trajectory.