Doctoral Research at the University of Maryland, College Park (1997  2004) Summary:
My research focuses on
understanding the intrinsic structure of the space of light rays as the
most general representation for visual information. Understanding this
structure enables us to tailor both image acquisition devices and image
processing algorithms to the image understanding task at hand, thus
optimally facilitating the recovery of information about the world. 

Nowadays,
cameras and computing resources become cheaper and smaller by the month
which enables us to capture video sequences using hundreds of cameras
at the same time. These camera arrangements should not be modeled as
sets of discrete cameras anymore, but one should study directly the
continuous space of all light rays as described by the plenoptic
function and interpret this multitude of video sequences as
spatiotemporal samples in the space of light rays. By extending
conventional signal processing techniques to the space of timevarying
light rays, I study the Plenoptic Video Geometry.


Plenoptic
video geometry decribes the geometric and differential structure of all
the visual information that a moving imaging sensor can possibly
capture. If we relate this structure to the motion of the sensor as
well as to the spatiotemporal structure of the world that is observed,
we can show that for a polydioptric camera,
that is a generalized camera that captures a multiperspective subset
of the space of light rays, the 3D motion estimation problem becomes
scene independent and is thus much simpler to solve then when using
conventional pinhole cameras. 


It
has been shown that conventional pinhole cameras are often not the
optimal imaging device for a given task. Based my analysis of the
Plenoptic Video Geometry I define a coordinate system on the space of cameras,
where the position of a camera design is determined by the relationship
between the design parameters of the camera and the ease and accuracy
by which we can solve a given task. Each coordinate system then
suggests guidelines for the design of novel imaging sensors that are
optimal for the task of 3D motion estimation, dynamic 3D photography, object tracking and many others. 

The
framework of camera design above allows us to design cameras that are
specifically tailored to the task at hand. Specifically, we study the
properties of polydioptric cameras,
that are generalized cameras that capture a multiperspective subset of
the space of light rays, and their implementations, e.g. arrays of
closely spaced pinhole cameras. 


Along
another line of work, I am also studying how to represent and extract
the spatiotemporal structure of complex nonrigidly deforming objects
from many (discretely or continuously spaced) viewpoints. Some initial
results can be found here, where from synchronized and calibrated
multiview video sequences I reconstructed a talking head
using spatiotemporal stereo information and multiresolution
subdivision surfaces which is then rerendered from novel view points. 


Using
a combination of statistical and geometrical methods I track and
analyze the motion of objects in video sequences captured by moving
vision platforms. 
