1. Stereo Image Processing

Objectives:

- Stereo Matching using Induced Symmetry (SymSTEREO);

- Stereo Rangefinder (the independent estimation of depth along a scan plan);

- Piecewise-planar (PPR) reconstruction of 3D environments;

- Visual odometry and dense PPR using plane primitives (structure and motion from plane primitives);

- Speeding up Stereo-Scan using parallel computer architectures (GPU/CUDA);

- A fully functional real-time parallel 3D reconstruction pipeline;

- Acquisition of large stereo datasets in urban environments from vehicles (cars and drones);

- Processing of long sequences acquired by a stereo camera pair mounted on conventional vehicles (for

reconstruction of urban scenarios);

- A new metric for detecting planes from a monocular sequence;

- 3D modeling of man-made environments from a single image.

3D Reconstruction

Achievements: A

• A new family of stereo cost functions that measure symmetry instead of photo-similarity for associating pixels across views: SymSTEREO (Stereo Matching using Induced Symmetry);

• Fig. 1 - SymStereo: Stereo matching using induced symmetry.
• Stereo range finder (SRF) based on SymSTEREO for detecting and reconstructing plane surfaces from stereo. This is the first work that discusses and benchmarks the concept of SRF (Stereo RangeFinder---the independent estimation of depth along a scan plan), providing the theoretical ground for the work in piecewise planar reconstruction;

• Fig. 2 - (Top) Experimental setup: The top camera C1 points down, the bottom camera C2 points up, and the LRF is positioned between the cameras. (Bottom) Experimental results on profile cut estimation showing how stereo vision can replace a LRF.
• A new method for carrying Piecewise Planar Reconstruction (PPR) from a calibrated stereo pair. The approach is based in the SymStereo/SRF framework and the experimental results clearly show that it outperforms competing methods;

• A new method for object pose estimation using mirror reflections to provide an accurate and practical solution for the extrinsic calibration of mixtures of color cameras, LRFs, and depth cameras with non-overlapping Field-of-View (FOV). The method is able to calibrate any possible sensor combination as far as the setup includes at least one color camera.

Visual Odometry

Achievements:

• Stereo-Scan: New approach for simultaneously carrying visual odometry and dense PPR using plane primitives. This approach is able to detect scene planes, compute the relative camera motion based on plane detections (visual odometry), and densely reconstruct the scene;

• Large-scale sequential stereo-scan: We built on the Structure-from-Motion (SfM) pipeline to obtain a more robust pipeline that includes a new approach for plane segmentation. This new pipeline was tested in long sequences acquired by a stereo pair mounted on a conventional car driven through the streets of Coimbra.
  • Dataset acquisition: stereo sequences acquired in urban areas of the city of Coimbra, using the vehicle (car) with a stereo pair mounted so that it points forward and sideways. This new dataset includes a closed loop within the city of Coimbra.

Fig. 3 - Stereo pair pointing forward (left) and sideways (right) for dataset acquisition.

• Reconstruction from a moving vehicle:

• Investigation about how affine correspondences (ACs) constrain the two-view geometry, and development of new algorithms for plane segmentation and visual odometry from ACs that compare favorably with respect to methods relying in point matches.

Real-time Dense Reconstruction

Achievements:

• A new GPU-based parallel program and hardware platform that perform the acceleration of symmetry-based dense stereo matching (SymStereo algorithm) for generating 3D volumes;
  • A new procedure to Solve the Uncapacitated Facility Location (UFL) problem using fast fusion moves;
  • A fully functional real-time parallel 3D reconstruction pipeline that uses dense stereo based photo-symmetry. The LogN variant of the SymStereo framework was exploited to achieve superior results for images with slanted surfaces, which are of particular interest for areas of computer vision, such as the processing of big data sets for urban scene reconstruction intensively used by players like Google. The output results obtained by refining distinct matching cost and aggregation parameters for SymStereo are analyzed, targeting the most suitable combinations for slant dominated images. In we propose a parallelization strategy for the presented pipeline, based on multiple GPU devices, orchestrated by a master CPU;

Fig. 1 - Parallel pipeline

Table 1 – Examples of 3D reconstruction using dense stereo algorithm

• Single- and multi-GPU parallel solution that performs symmetry-based sparse stereo matching (the logN variant of the SymStereo algorithm) used in Piecewise Planar Reconstruction (PPR);
• A GPU-accelerated message passing algorithm (max-sum) for solving the Uncapacitated Facility Location (UFL) problem used in Piecewise Planar Reconstruction (PPR) (we propose single- and multi-GPU solutions);

• A website was built to describe the Piecewise Planar Reconstruction (PPR) from plane primitives (will be available online at: http://montecristo.co.it.pt/PPR_Rec/);

• SymStereo (logN) and UFL sequential (CPU) and parallel code (GPU / CUDA) will be available online at: http://montecristo.co.it.pt/PPR_Rec/;
• Dataset of stereo images acquired in urban areas of the city of Coimbra, using the vehicle and setup shown in the Fig. XXXXa;
• Acquisition of a new dataset of stereo images captured in urban areas of Helsinki, Finland, using a drone with a setup mounted as shown in Fig. 2.

Fig.3 - Camera setup to mount in drone

Fig. 4 - Acquisition setup and sensors used in the acquisition of images from the top of the Torni’s bulding, the highest building in Helnsinki. The goal is to develop and test a setup that is capable of acquiring images captured from a drone flying over the city.

Publications

• Michel Antunes and J. P. Barreto, “SymStereo: Stereo Matching using Induced Symmetry,” International Journal of Computer Vision, pp. 1–21, September 2014
• Michel Antunes, J. P. Barreto, U. Nunes, “Piecewise Planar Reconstruction using Two- Views,” International Journal Image and Vision Computing, Elsevier, 2016.
• Ricardo Ralha, G. Falcão, J. Amaro, V. Mota, M. Antunes, J. Barreto, U. Nunes, “Parallel refinement of 3D slanted scenes using dense stereo induced from symmetry” (submitted to an Intern. Journal)
• C. Raposo, J.P. Barreto, and U.Nunes, Extrinsic Calibration of Multi-Modal Sensor Arrangements with Non-Overlapping Field-of-View (to be submitted to an Int. Journal)
• Carlos Graça, Carolina Raposo, J.P. Barreto, U. Nunes and Gabriel Falcão, Accelerating UFL and SymStereo pipelines for picewise planar-based 3D reconstruction (to be submitted to an Int. Journal)
• C. Raposo, M. Antunes, and J. P. Barreto, “Piecewise-Planar StereoScan: Structure and Motion from Plane Primitives,” European Conf. on Computer Vision (ECCV’14), 2014
• Vasco Mota, G. Falcão, M. Antunes, J. P. Barreto, U. Nunes, Using the GPU for fast symmetry-based dense stereo matching in high resolution images, IEEE Int. Conference on Acoustics, Speech, and Signal Processing (ICASSP14), Florence, Italy, May 2014
• R. Ralha, G. Falcão, J. Andrade, M. Antunes, J. P. Barreto, U. Nunes, Distributed dense stereo matching for 3D reconstruction using parallel-based processing advantages, IEEE Int. Conference on Acoustics, Speech, and Signal Processing (ICASSP15), May 2015

Awards

• João P. Barreto e Gabriel Falcão. "UrbanScan: fast 3D modeling of urban scenes using a new stereo vision approach", Google Faculty Research Award from Google Inc., January 2014;
• Gabriel Falcão (PI). The University of Coimbra has been elevated to "GPU Research Center" from NVIDIA, January 2015.

MSc Theses

• R. Ralha, Real-time symmetry-based dense stereo matching for 3D reconstruction, MSc dissertation (supervisors: G. Falcão and J.Barreto), FCTUC, February 2015;
• V. Mota, Using Fast Fusion Moves to Solve the Uncapacitated Facility Location (UFL) Problem: Applications in Computer Vision, MSc dissertation (supervisors: J.Barreto and G. Falcão), FCTUC, February 2015.

Technical Reports

• João Marcos, “SymStereo Acceleration, Smart Lines Detection and Stereo Datasets Acquisition of Distinct Challenging Urban Scenarios”, Technical Report, 2014;
• Carlos Graça, “Fast Parallel SymStereo and UFL (message passing - max-sum) algorithms applied to Piecewise Planar 3D reconstruction using hybrid CPU + GPU and multi-GPU architectures”, Technical Report, December 2015.

Invited Talks

• G. Falcão, “GPUs and parallel computing towards 3D reconstruction” under the participation at the international workshop on “Interactive 3D models from photos”, from the Architectural Democracy Group (http://www.aibeo.com/#!architectural-democracy-research/c1m2d) at the University of Aalto, in Helsinki, Finland, September 2015.