Summary:

My research focuses on understanding the intrinsic structure of the space of light rays as the most general representation for visual information. Understanding this structure enables us to tailor both image acquisition devices and image processing algorithms to the image understanding task at hand, thus optimally facilitating the recovery of information about the world.

Plenoptic Video Geometry.
	Nowadays, cameras and computing resources become cheaper and smaller by the month which enables us to capture video sequences using hundreds of cameras at the same time. These camera arrangements should not be modeled as sets of discrete cameras anymore, but one should study directly the continuous space of all light rays as described by the plenoptic function and interpret this multitude of video sequences as spatio-temporal samples in the space of light rays. By extending conventional signal processing techniques to the space of time-varying light rays, I study the Plenoptic Video Geometry.
	Plenoptic video geometry decribes the geometric and differential structure of all the visual information that a moving imaging sensor can possibly capture. If we relate this structure to the motion of the sensor as well as to the spatio-temporal structure of the world that is observed, we can show that for a polydioptric camera, that is a generalized camera that captures a multi-perspective subset of the space of light rays, the 3D motion estimation problem becomes scene independent and is thus much simpler to solve then when using conventional pinhole cameras.
Polydioptric Camera Design.
	It has been shown that conventional pinhole cameras are often not the optimal imaging device for a given task. Based my analysis of the Plenoptic Video Geometry I define a coordinate system on the space of cameras, where the position of a camera design is determined by the relationship between the design parameters of the camera and the ease and accuracy by which we can solve a given task. Each coordinate system then suggests guidelines for the design of novel imaging sensors that are optimal for the task of 3D motion estimation, dynamic 3D photography, object tracking and many others.
	The framework of camera design above allows us to design cameras that are specifically tailored to the task at hand. Specifically, we study the properties of polydioptric cameras, that are generalized cameras that capture a multi-perspective subset of the space of light rays, and their implementations, e.g. arrays of closely spaced pinhole cameras.
Spatio-temporal models from video.
	Along another line of work, I am also studying how to represent and extract the spatio-temporal structure of complex non-rigidly deforming objects from many (discretely or continuously spaced) viewpoints. Some initial results can be found here, where from synchronized and calibrated multi-view video sequences I reconstructed a talking head using spatio-temporal stereo information and multi-resolution subdivision surfaces which is then rerendered from novel view points.
Tracking objects in video sequences.
	Using a combination of statistical and geometrical methods I track and analyze the motion of objects in video sequences captured by moving vision platforms.