Geometrically and Radiometrically Undistorted Projection onto Nonplanar Surfaces

Geometrically and Radiometrically Undistorted Projection onto Nonplanar Surfaces

Abstract - Some critical issues for geometrically- and radiometrically- undistorted projection onto nonplanar surfaces are introduced and an integrated framework (under development) for dealing with the issues is proposed.

Direct-projected (or projection-based) augmented reality approaches have been gaining attention recently. Projection made it possible to use 3-D real and large objects as displays and freed from discomforts incidental to wearing a device such as HMD. However, DirectAR usually suffers from the following problems:

1. Geometric distortion - the projection may be geometrically distorted due to the non-planar surface
2. Viewpoint-ignorant projection - projection cannot be seen to user as intended because the position of the projector is not same as that of the user's viewpoint
3. Radiometric distortion - the projection may be modulated by the surface color
1. Uneven projection - the projected area may not have uniform brightness when the projection is obliquely headed for the surface

Geometric Registration

Geometric registration indicates that projection is exactly overlaid on the surface without geometric distortion. To do so, the projectors should be calibrated and the surface geometry should be known first. We use a modified version of the well-known Zhang's calibration method for calibrating projectors and a linear triangulation method for recovering the geometry of projection surface. The surface is triangularly represented using the recovered points on the surface because the surface is assumed to be piece-wise planar. Finally, the projection images are patch-wise prewarped using homography to be undistorted.

Figure 1. Radiometric compensation of a color image (monkey) and a gray image (lena). The right-top images show that the projection color was distorted due to the texture of projection surface. There is no color distortion in the left-bottom images by compensating the projector input images. The right-bottom images show the change of pixel values of projector input images for compensating the color distortion.

Our radiometric compensation method is so fast that we can project the color-compensated video frames onto a colored screen in realtime. In dynamic environments, the process of analyzing the surface color should be performed in realtime. A method for resolving this problem was presented in CVPR 2005. However, the method is unavailable on geometrically-changing surfaces. We are trying to combine the method with our realtime geometry estimation method.

Viewer-Dependent Projection

The projection cannot be seen to user as intended because the position of the projector is not same as that of the user's viewpoint. In AR-assistant surgery system of Fig. 2, the system visualizes the 3D position of tumor and additional information on the surface of human body. In the viewpoint of the projector, the direction of the projection should be ①. However, it is clear that the direction of the projection should be changed into ② when considering user's viewpoint.

Figure 2. AR-assistant surgery system considering user's viewpoint.

Intensity-Compensated Projection

When the projection is obliquely headed for the nonplanar surface, the area of the projection has uneven brightness as shown in Fig. 3. We developed a geometry-based method for making the projected area have even brightness. The intensity of projection is dependent on the angle between the projection vector and the surface normal vector. The intensity of the projection image is increased or decreased according to the angle. The target brightness is obtained by the average brightness of the projection area.

Figure 3. Uneven projection. The projector lights the nonplanar surface in the left side. The projection is not uniform although an unicolored projection is applied to the surface.

Figure 4. Intensity-compensated projection in a color image (cat) and a gray image (lena). Left images: a projector lights the cylindrical surface on the left side and thus the left part of the image is brighter than the right part of the image. Right images: after compensation, the brightness of the whole image became similar.

Figure 5. Geometrically- and radiometrically- undistorted projection onto nonplanar (static) surface. All the component methods were combined. First row: 3D target object (three blocks), nonplanar colored screen, without any compensation, magnification of the third image. Second row: after only geometric correction and after fully compensated projection when user's viewpoint is located on the left side of the screen, after only geometric correction and after fully compensated projection when user's viewpoint is located on the right side of the screen. Third row: magnification of the images of second row.

Further development

• Realtime geometric calibration: In case of dynamic projection surface, the process of estimating the surface geometry should be performed in realtime. There are some methods using structured light in the literature. However, they cannot be applied to augmented reality because the structured light projected together with augmented reality image may visually distort the augmented reality image. Thus, we are developing a novel method for estimating the dynamic surface geometry. In our method, the structured light is not visible externally but detectable using a simple image processing algorithm internally. Therefore, the invisible structured light can be applied to augmented reality applications for recovering the dynamic surface geometry without frame loss. The details are not mentioned here because of not being published.
• Radiometric enhancement: Radiometric calibration can make the augmented projection look as intended. However, the original contrast cannot be enhanced. Contrast enhancement method of the augmented projection is of great importance, which combines a hue- and saturation- preserving histogram equalization method with radiometric calibration methods. Refer to [3] for the details.

Publication

1. Hanhoon Park, Gap-Chul Kang, Moon-Hyun Lee, Sang-Jun Kim, and Jong-Il Park, "Direct-Projected Augmented Reality Considering User's Viewpoint," Proceedings of International Meeting on Information Display and Exhibition (IMID) 2005, vol. 1, pp. 748-751, Seoul, Korea, July 2005
2. Hanhoon Park, Moon-Hyun Lee, Sang-Jun Kim, and Jong-Il Park, "Surface-Independent Direct-Projected Augmented Reality," Lecture Notes in Computer Science 3852, Computer Vision, Proceedings of ACCV'06, pp. 892-901, Jan. 2006
1. Hanhoon Park, Moon-Hyun Lee, Sang-Jun Kim, and Jong-Il Park, "Contrast Enhancement in Direct-Projected Augmented Reality," Proceedings of IEEE International Conference on Multimedia & Expo (ICME) 2006, pp. 1313-1316, Toronto, Canada, July 2006