In recent literature there are two lines of work in volume rendering. One is using the traditonal discrete representation techniques such as triangle meshes or voxel grids. The second is using multilayer perceptron networks to represent the 3D shape/scene, wherein the shape/scene is encoded in the weights of the network and the 3D spatial location is mapped to an implicit representation of the shape.

Representing 3D shapes using Neural Networks

In recent works, 3D shapes were represented as level sets by optimizing deep networks that map (x,y,z) co-ordinates to signed distance functions. One such network is the "Occupancy Network" where the 3D surface is represented as the continuous decision boundary of a deep neural network classifier. However, these techniques are limited to simple shapes with low geometric complexities that result in oversmooth rendering. The proposed 5D radiance field can represent more complex and high-resolution geometry.

Synthesising novel views and rendering

Given a large set of images, interpolation techniques can be used to to generate novel views. With less number of images, computer vision techniques have to be used to synthesise novel views. One approach is to use mesh-based methods such as differentiable rasterizers or pathtracers. Another approach is using volumetric representations such as voxel grids. Recent methods have used a combination of deep networks, CNN and voxel grids to synthesise novel views.

These methods have produced impressive results, but are limited due to poor time and space complexity due to discrete sampling. This limitation is overcome by representing the volume (shape/scene) within the weights of a fully connected neural network. This produces higher quality renderings at a fraction of the storage of volumetric methods.

• During our testing we found out a multitude of facts about NeRF which are listed below.

• Reflections can be captured by NeRF due to its very properties, since it calculates the behavior of light for every viewing angle it learns how reflections work in the environment.

• Fully converged scenes require input images from all directions to fully model the scene, this is because the algorithm can't figure out how something looks from a direction that it has no data on.

• Convergence for real world scenes is typically not as good as convergence from a synthetic scene, this is likely because of slight imperfections that are made while making the data set, as well as the fact that it is just as likely that not as many images were added to the data set in the real world versus the synthetic scenes.

• NeRF also does not work for dynamic scenes in its current implementation, after all it has to learn from static images, and since they are not all taken at exactly the same time there are differences if there is motion in the image. This could possibly be mitigated by having many cameras all take an image at once, but that is impractical for most situations.

In the future we plan to enable users to add, remove, and/or replace objects in the 3D volumetric scene generated by NeRF. We hope to achieve this by using Tiny Object Loader to add 3D objects to the scene. Removing objects will be done by modifying the NeRF network to include an object label which when detected would make the labelled portion of the image 'see-through' for all intensive purposes. To replace objects in the scene, a combination of the two approaches will be used, first the object is removed, then another object will be placed in its centroid allowing for a fully modifiable scene for use in applications ranging from VR to home improvement.