Full paper: https://arxiv.org/abs/1812.10687

Example out-of-distribution images cast onto the training distribution:

Simulated images:

Real images:

Additional encoding insights:

We found an image's NLL carries little additional information relevant to the task of avoiding collisions given unusual images at test time. Indeed Figure 9 shows there is considerable overlap between both in-distribution and out-of-distribution image NLLs, meaning it is difficult to even detect (let alone handle) out-of-distribution inputs using NLL information alone. Indeed we found this information is often misleading.

To gain some insight into how out-of-distribution images are encoded, we visualize the latent space z in two dimensions (Figure 10) using t-SNE [Maaten et al. 2008]. To do so, we first computed the latent z Gaussian distributions for both train and test images. Second, for each latent distribution, we drew 10 samples. Third, we inputted all samples into t-SNE, such that t-SNE does not know which samples are train and which are test. Finally, we show the differences of 1) the training data used to train the VAE, 2) additional in-distribution data, and 3) out-of-distribution data. Figure 10 shows that even though most out-of-distribution images have latent values that are unlikely under the latent encoding of training images, they rarely have values that were never used by the training distribution. Thus the VAE loss function (5) encountered most out-of-distribution latent values during training, and fit them to decode to a reasonable in-distribution image.