We prepared an offline Blender demo of our project. We setup 5 randomly placed cameras in a scene, and placed a 3D human model that walks on a path. We calibrated the cameras using our multi-camera calibration module. After that, we took 250 frames from 5 cameras, with the human subject walking around the scene. Then we fed these frames into our 2D human detection module and produced 2D keypoints. Using corresponding 2D keypoints in our multi-camera calibration module, we were able to track the 3D motion of the subject.

Although we haven’t had time to try on a real world multi-camera calibration setup with synchronized cameras (which is a really big problem on its own), we have implemented proof-of-concept virtual tests with various scenarios. We will show how the variation of different parameters (cameras’ field of view (fov) values, locations, orientations, number and number of points) affect the results. Because some parameters do not affect the result, we kept them static. (e.g. aspect ratio will always be 16 / 9). Also we always added some noise to 2D projections, to see how the algorithm works on noisy data. The following approach was used to test our solution:

Create m virtual cameras

a. Different field of view degrees f: 30° and 60°

b. Place the cameras on random locations in a sphere with a radius of 50 meters.

c. All cameras looking at the origin [0,0,0]

Create n random points in bounding box
Generate 2D point correspondences for each camera
Add random pixel noise in range ±x to the correspondences
Calibrate the cameras
Compute the RMS errors for reprojection, triangulation, camera location, and camera rotation
Repeat the steps 4-6 100 times and take the average of errors
Repeat the steps 4-7 with different pixel noise ranges

Plots for All Cases

Noise-Free Setup

Evaluation

Looking at case 1 and case 2, only difference is cameras’ focal points, which were parametrized by fov (field of view) angles, which is 30° in case 1, and 60° in case 2. Almost all the time, test case 1 with 30° fov gives more accurate results. We interpreted this difference as, since the same pixel noise in a tighter field of view sweeps a larger cone, these results are understandable.

If we compare test case 3 (10 points) with case 1 (100 points) and case 5 (1000 points), we see that using 10 points is not sufficient, since a more accurate result can be achieved by 100 or 1000 points. Except for reprojection error, 100 and 1000 points does not incur much difference. As can be seen from plots, reprojection error in the case of 1000 points, is slightly lower than error in case of 100 points.

To see how the number of cameras affects the accuracy of our solution, we compared case 1 with case 4. In case 1 we have 9 cameras, while there are 18 cameras in case 4. After a certain noise value which is about 2.0 pixels, if we increase the number of the cameras, reprojection error will also increase, instead of decreasing. That is to say, if noise is expected to be in a range we know, there is no need for more cameras for more accurate results.

Impact

Since our system can find real-world 3D positions of points from frames obtained by multiple cameras, it is appropriate to be used as an effective tracking system. Once the tracking system has been calibrated, it can begin working on a wide variety of specialized applications. Following required modifications to our basic application, it will be possible to follow any discernible moving item across large spaces and use the data obtained for a specific purpose. During tracking, using multiple cameras provides significant advantages since it can tackle difficulties that stereo-camera systems can't, such as picture instabilities caused by a foreign item blocking the target object's image. Furthermore, even when there are no barriers in the road, it would improve accuracy.

When our system is converted into a product, it will provide a wide flexibility to the user. That is, if only one camera is kept fixed by knowing its location, the location of all other cameras can be changed by the user to create the desired configuration. To illustrate its convenience with a scenario, assume that a large area is to be processed with many cameras. In order to establish a tracking system with cameras in a certain configuration, the position and camera angle of the point where all the cameras are located would have to be laboriously measured. However, with the solution we have developed and consequent user-friendly extensions, the information of all the cameras can be automatically determined by simple user operations, completing the camera calibration.

Additionally, unlike the traditional methods, in the solution we implemented, the cameras do not have to see all of the 3D points, only some overlap between camera pairs is required.

Future Directions

Our solution assumes that we know the intrinsic parameters of cameras. Although this assumption makes the calibration process easy, if we also predict camera intrinsic parameters, using similar procedures to those we have explained before, any camera can be used for calibration.

Instead of 8-point Algorithm, Gold Standard Algorithm can be implemented for fundamental matrix estimation for better robustness to noise. If the intrinsics are known, we can also directly estimate the essential matrix using Nister’s 5-point Algorithm. Furthermore, the triangulation method we use assumes no noise present in the measurements. Different triangulation methods that are robust to noise can be implemented.

Using a tracking algorithm for people who were detected once, would improve the performance by keeping tracking at times detection algorithm misses in some frames. Also it is possible to tag them with the said improvement, and accordingly various actions can be taken.

Page updated

Google Sites

Report abuse