Augmented Reality

Virtual Ads Insertion by Scene Analysis for Augmented Reality

Augmented Reality (AR) is getting closer to real world consumer applications. The user expects the augmented content to better comprehend and enjoy the real scene, like the sightseeing, sports game and work place. One of its applications is ads insertion, also being a category of virtual content insertion (VCI). The basic concept consists of identifying specific places in the real scene, tracking them and augmenting the scene with the virtual ads.

Specific region detection relies on scene analysis. For some typical videos, like sports game (soccer, tennis, baseball, volleyball etc.), a playfield constrains the player’s action region and also makes easier to find a good place for ads insertion. So playfield modeling is applied to extract the court area and the standard model for court size is used to detect the specific region, like soccer center circle and goalmouth, tennis and volleyball court etc. For building view, the façade looks appropriate to post ads. The modern building shows structured visual elements, like parallel straight lines with repeated window patterns. So it makes sense to estimate vanishing points to determine the orientation of the architecture. Then the rectangular region from two groups of parallel lines can be used for ads insertion.

Camera calibration is to identify the camera parameters when the scene is captured. Based on that, the virtual ads image can be transformed to the detected region for insertion with perspective projection. Registration is important to accurately align the virtual ads with the real scene by visual tracking. The visual tracking method can be either feature-based or region-based, extensively discussed in computer vision field. Sometimes GPS or information from other sensors (inertial for the camera) can be used together to make tracking much more robust. The failure in tracking may cause jittering and drifting which cause bad viewing impression for users.

The virtual-real blending should take into account the difference in contrast, color and resolution to make the insertion seamless for the viewers. Apparently, it is easier to adapt the virtual ads to the real scene. Color harmonization is an efficient method to realize this goal. Below is the diagram of how an ads insertion system works in the video proposed by us.

Figure 1: Ads insertion from the camera view in Augmented Reality.

Figure 2: Diagram of ads insertion in the video by scene analysis.

1. Ads insertion above the soccer goalmouth bar

Figure 4: Court model fitting & tracking of the tennis for insertion (w/o color harmonizstion).

3. Ads Insertion on the building facade

Figure 5: Vanishing point estimation, rectangular region detection/tracking, ads insertion.

4. Video Demos

Ads insertion for building facade 1 Ads insertion for building facade 2

Ads insertion for building facade 3 Ads insertion for building facade 4