RayNet

RayNet was proposed by Paschalidou et al. [1] as a framework combining strengths of probabilist models [2,3,4] and neural networks [5,6] to reconstruct 3D models from a collection of images and their camera poses. RayNet integrates (see figure below) a convolutional neural network that learns view-invariant feature representations with an Markov network that explicitly encodes the physics of perspective projection and occlusion. The authors trained RayNet end-to-end using empirical risk minimization, and evaluate it on real-world datasets. The code is available in GitHub.

References

1. Paschalidou, D., Ulusoy, O., Schmitt, C., Van Gool, L., & Geiger, A. (2018). Raynet: Learning volumetric 3d reconstruction with ray potentials. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3897-3906).
2. Ulusoy, A. O., Geiger, A., & Black, M. J. (2015, October). Towards probabilistic volumetric reconstruction using ray potentials. In 2015 International Conference on 3D Vision (pp. 10-18). IEEE.
3. Osman Ulusoy, A., Black, M. J., & Geiger, A. (2016). Patches, planes and probabilities: A non-local prior for volumetric 3d reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3280-3289).
4. Ulusoy, A. O., Black, M. J., & Geiger, A. (2017, July). Semantic multi-view stereo: Jointly estimating objects and voxels. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4531-4540). IEEE.
5. Hartmann, W., Galliani, S., Havlena, M., Van Gool, L., & Schindler, K. (2017). Learned multi-patch similarity. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1586-1594).
6. Ji, M., Gall, J., Zheng, H., Liu, Y., & Fang, L. (2017). Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2307-2315).