Flying with Photons: Rendering Novel Views of Propagating Light

Summary

Flying with Photons: Rendering Novel Views of Propagating Light 1 University of Toronto 2 Vector Institute 3 Stanford University ECCV 2024, Oral Presentation This synthesized flythrough shows a pulse of light traveling through a coke bottle. The whole event spans roughly 3ns, which is 100 million times less than it takes us to blink. Light can be seen scattering off liquid, hitting the ground plane, focusing on the cap and reflecting to the back. We present an imaging and neural rendering technique that seeks to synthesize videos of light propagating through a scene from novel, moving camera viewpoints. Our approach relies on a new ultrafast imaging setup to capture a first-of-its kind, multi-viewpoint video dataset with picosecond-level temporal resolution. Combined with this dataset, we introduce an efficient neural volume rendering framework based on the transient field. This field is defined as a mapping from a 3D point and 2D direction to a high-dimensional, discrete-time signal that represents time-varying radiance at ultrafast timescales. Rendering with transient fields naturally accounts for effects due to the finite speed of light, including viewpoint-dependent appearance changes caused by light propagation delays to the camera. We render a range of complex effects, including scattering, specular reflection, refraction, and diffraction. Additionally, we demonstrate removing viewpoint-dependent propagation delays using a time warping procedure, rendering of relativistic effects, and video synthesis of direct and global components of light transport. To render transient videos from novel viewpoints, we cast rays into the scene and use a neural representation to learn two fields: a conventional density field and the transient field: a mapping from a 3D point and 2D direction to a high-dimensional, discrete-time signal that represents the time-varying radiance at ultrafast timescales. That is, for every ray sample, the neural representation returns a transient ...