Differentiable Ray Tracing

We know that rendering a photorealistic image from a 3D scene includes information from the camera, geometry, material, and light source.

Reversing a rendering is important since we often want to reconstruct 3D scenes from images and render them again, say, in robotics, VR, or lighting design in films.

Both direct and reverse rendering have great potential for deep learning so that the neural network can understand the 3D information of the images. For this, we need to compute the rendering gradient.

The scene function f(x,y) contains the information of radiance passing through the pixel. Phi is the set of scene parameters.

Let's view the problem in 1D. Triangles are modeled as indicator functions. The x, y-axes are the pixel coordinate and pixel color.

The pixel color is the integral of all colors passing through the pixel. To compute its gradient, the derivatives enter the integral (Leibniz's theorem) and differentiate f(x,y). A key observation is that f(x,y) is non-differentiable along the edges. Informally, its derivatives are Dirac delta functions. This is linked to the fact that we can approximate indicator functions by smooth functions preserving the integral.

As all discontinuities occur at the edges, f(x,y) can be broken into smooth parts using indicator functions.

Using Leibniz's rule the gradient enters the integral, then applying the chain rule to the expression, we get two components.

In yellow, the expression estimates the original integral, except by replacing the f(x,y) by its gradient, which can be calculated through automatic differentiation.

The blue part computes the integral over the edges of triangles containing the Dirac delta functions.

Coarea formula is used to eliminate the Dirac delta function and the integration domain becomes the edges.