Learning Adaptive Sampling and Reconstruction for Volume Visualization

Sebastian Weiss, Mustafa Işık, Justus Thies, Rüdiger Westermann

TVCG 2020

Abstract

A central challenge in data visualization is to understand which data samples are required to generate an image of a data set in which the relevant information is encoded. In this work, we make a first step towards answering the question of whether an artificial neural network can predict where to sample the data with higher or lower density, by learning of correspondences between the data, the sampling patterns and the generated images. We introduce a novel neural rendering pipeline, which is trained end-to-end to generate a sparse adaptive sampling structure from a given low-resolution input image, and reconstructs a high-resolution image from the sparse set of samples. For the first time, to the best of our knowledge, we demonstrate that the selection of structures that are relevant for the final visual representation can be jointly learned together with the reconstruction of this representation from these structures. Therefore, we introduce differentiable sampling and reconstruction stages, which can leverage back-propagation based on supervised losses solely on the final image. We shed light on the adaptive sampling patterns generated by the network pipeline and analyze its use for volume visualization including isosurface and direct volume rendering.

The basic idea is visualized above:

First, a low-resolution image is rendered, typically of 1/4 or 1/8 the resolution
A first neural network estimates an importance map,
which is then used to sample a fixed number of rays.
A second neural network reconstructs the dense output image from these sparse samples.
The networks are trained end-to-end using supervised losses on the output images.

But enough text, here are some comparisons on test datasets:

PDF: doi.org/10.1109/TVCG.2020.3039340
Preprint: arxiv.org/abs/2007.10093
Source Code: github.com/shamanDevel/AdaptiveSampling
Citation:

@article{weiss2020adaptivesampling,
title={Learning Adaptive Sampling and Reconstruction for Volume Visualization},
author={Sebastian Weiss and Mustafa Işık and Justus Thies and Rüdiger Westermann},
year={2020},
journal={IEEE Transactions on Visualization and Computer Graphics},
volume={},
number={},
pages={1-1},
doi={10.1109/TVCG.2020.3039340}}
}