Reliable 3D Eyeball Sphere Model Fitting through Pupil Size Estimation before Unprojection

Video-based eye tracking system reveals where the subject is looking at using the pupil position in the eye camera image and the calibration result that maps a given pupil position to a certain point in the scene. Traditional video-based eye tracking systems usually define the pupil position in the eye camera using the pupil’s relative position with respect to the “glint,” the strong IR reflection spot created by the cornea (the bright spot at the center of Fig 1). Those systems usually have complicated hardware, strict setup requirements, and high demand for power because they need a strong IR light source and require the camera, the light source, and the subject’s eye to be placed at a specific angle such that the camera can capture a strong glint.

In recent years, Pupil Labs developed a new 3D eye tracking system that does not use the glint to describe the pupil position. It unprojects the pupil ellipses from many video frames into the 3D space to get the normal vectors of the pupil in different video frames. Then, it reconstructs a 3D eyeball sphere model in the eye camera reference frame (shown by the green circle). After that, it calculates where the eye looks at by unprojecting the pupil ellipse (shown by the red ellipse) onto the eyeball sphere and calculating the vector from the eyeball center position to the unprojected pupil center. Using the calibration result, this system maps where the eye looks at in the eye camera reference frame into the scene camera reference frame.

1. From the camera image, we can extract the pupil region and fit an ellipse (the red ellipse in Fig 3) to it.

2. Most people's pupils have the shape of a circle. As shown by this paper from Safaee-Rad et. al.,  if we know the camera intrinsic matrix, the equation for the ellipse of the projected circular feature, and the size of the circular feature, we can unproject the ellipse into the 3D space and know the center position and the normal vector of the circular feature. Here, we unproject the pupil assuming an arbitrary constant size.

3. The unprojection in step 2 produces 2 circles, so we need to decide which circle corresponds to the actual pupil in the 3D space. To do that, for each pair of circles, we project their centers and normal vectors back onto the camera image. After that, the projected centers and normal vectors become the same line described by two pairs of a point and a vector. The two vectors should be pointing at opposite directions.

4. If we do step 1,2, and 3 for many video frames, we get a lot of lines on the 2D camera image. Since the pupil always rotates around the eyeball center, by finding the intersection of those lines using RANSAC, we can roughly estimate the eyeball center position in 2D space. Since the pupil circle normal vector should always point away from the eyeball center, we select the circle and normal vector pair that corresponds to the projected vector which points away from the 2D eyeball center. This allows us to know the actual pupil orientation in many video frames.

5. Since the pupil rotates around the eyeball center, we can intersect the pupil lines (the line represented by the pupil circle center and the pupil circle normal vector) using RANSAC in the 3D space to find the 3D pupil center position (Fig. 4). Assuming the eyeball is tangent to the pupil circles at their centers, we can fit an eyeball sphere model.

6. After getting the 3D eyeball sphere model, we can unproject the pupil circle center onto the 3D eyeball sphere to get the gaze vector, which is the vector from the 3D eyeball center to the 3D pupil center. Through calibration results, we can map this gaze vector into the world reference frame.

The problem of Pupil Labs algorithm exists in step 2: "Here, we unproject the pupil assuming an arbitrary constant size." The pupil size of humans can change a lot depending on external lighting conditions and internal physiological conditions (Fig 5). If we assume the pupil size is constant, then the unprotected pupil circle can be all over the place (assumed size < real size, closer than actual; assumed size > real size, farther than actual), making the eyeball center estimation unstable (Fig 6) if the user does not freeze the eyeball model. Besides, it's hard for the user to know what's the right time to freeze the eyeball model such that the user can get the best tracking results. Therefore, to reliably estimate the 3D eyeball sphere model, we need to estimate the pupil size before unprojecting them to calculate the eyeball center.

To estimate the pupil size, we can use the outer edge of the iris. Iris refers to the colorful part of the eye that surrounds the pupil (Fig 7). It contains the muscle that controls the size of the pupil. The outer iris edge and the pupil is roughly concentric in the 3D space for most people. Besides, the size of the iris outer edge circle does not change, which makes it a perfect reference for estimating the pupil size.

Here, due to the single camera size ambiguity, we cannot know the true size of the pupil in each frame. What we really want to know is the relative size of the pupil with respect to the outer iris edge circle in each video frame. If we can get the relative size correct, we can reliably estimate the 3D eyeball center. 

To get the relative pupil size, we can first unproject the pupil ellipse assuming an arbitrary size. Then, we can find the pixels on the eye camera image that corresponds to the outer iris edge circle and unproject those points onto the plane where the pupil circle is. By calculating the distance from the unprojected points and the pupil circle center, we can then calculate the relative size of the pupil.

If we assume the eyeball radius is 10.39 mm, which is the Pupil Labs' default value, then the 3D eyeball center position is [3.2265    2.1426   37.9450] (in mm). This coordinate is given in the 3D camera reference frame, whose origin is at the camera, x axis goes right with respect to the image, y axis goes down with respect to the image, and z axis goes forward with respect to the image. In Fig 11, the projected 3D eyeball sphere is shown by the blue line. 

The reason for the eyeball sphere model being smaller than the actual eyeball is that Pupil Labs’ algorithm assumes the fitted 3D eyeball is tangent to the pupil circles at the pupil center, but in real eyes, the pupil is within the eyeball itself. Just like Fig 12, a perfectly fitted eyeball model should be this blue circle, which is well within the eyeball. Again, this shows that it’s almost impossible for new users to correctly freeze the eyeball model unless the user understands the pupil labs’ algorithm well.