Edge-Based Markerless Real-Time 3D Tracking of Rigid Objects (2007)


Method for real-time 3D object tracking. During the tracking process, the algorithm is continuously projecting the 3D model to the current frame by using the pose estimated in the previous frame. Once projected, some control points are generated along the visible edges of the object. The next pose is estimated by minimizing the distances between the control points and the edges detected in the image.


The implemented model-based tracking system follows a recursive scheme. This means that it repeats the same cycle frame after frame. This cycle consists of obtaining 2D evidences of the displacement of the model edges, using the pose calculated in the previous frame, and estimate the 3D translation and rotation made by the object between consecutive frames.

Thus, the result of the iteration is a 3x4 transformation matrix V = [ R | t ] (composed by a 3x3 rotation matrix and a 3x1 translation vector), called the motion matrix, that transforms the pose calculated in the previous frame into the pose of the object in the new frame.

In order to obtain the 2D data, a CAD model of the object is employed. In each frame, the model is projected by the pose calculated in the previous frame and some control points along its edges are calculated. A visibility test is done for each control point. Points that are occluded by the model are discarded for the next steps.For each of the visible control points, a one-dimensional search is done in order to find an edge in the image. At the end, the system knows a relation between the 3D coordinates of the control points, their 2D projections and the 2D translation of the points made between consecutive frames.


  • Barandiaran, J., Borro, D., Basogain, X., and Izkara, J.L., "MobileAugmented Reality for Providing Guide in Maintenance Tasks", Poster Contribution of the Laval Virtual, 9th International Conferenceon Virtual Reality 2007 (VRIC 2007). Laval, France. April 18-22, 2007.
  • Barandiaran, J., Borro, D., "Edge-Based Markerless 3D Tracking of Rigid Objects", 17th International Conference on Artificial Reality and Telexistence (ICAT 2007), pp. 282-283. Esbjerg, Denmark. November 28-30, 2007. pdf

Software used

  • VideoMan: Video capture and display
  • OpenGL: 2D and 3D graphics
  • OpenCV: Edge detection and other images processing algorithms
  • Levenberg-Marquardt nonlinear least squares algorithms in C/C++