tl;dr 

Reinforcement learning from images is difficult because of high-dimensional observations. Behavioral distance based algorithms learn state-representations from images where behaviorally similar states are grouped together. This can help downstream policy learning. 

BeigeMap representations preserve local metric structure of such distances, instead of trying to match all distances globally. This drop-in modification improved policy performance in our experiments.