Embedded Files
The MEMEX project promotes social cohesion through collaborative, heritage-related tools that provide inclusive access to tangible and intangible cultural heritage and, at the same time, facilitate encounters, discussions and interactions between communities at risk of social exclusion. These tools will empower communities of people with the possibility of welding together their fragmented experiences and memories into compelling and geolocalised storylines using new personalized digital content linked to the pre-existent European Cultural Heritage (CH) (See Paris story video example). MEMEX focuses on three core reusable technologies:
1. Knowledge Graph: Creating new infrastructure for geolocalised Cultural Heritage to reason on.
2. Localization: Computer vision based automatic localization of users and objects.
3. Storytelling trough augmented reality: Assisted story creation and visualization using advanced AR technologies.
As part of the CNRS-I3S Robot Vision team, our role in this project is to provide more accurate geolocalization of images taken by smartphones or hand-held cameras, where the GPS is partially available or absent. Particularly, the proposed solution involves the fusion a visual SLAM approach with GPS measurements in order to correct the global coordinates tag of each individual taken image. Such images are uploaded to the Mapillary server for further post-processing such as segmentation and 3D reconstruction. A wearable acquisition system has been developed to be used by an agent in public scenarios to register 360 images of the environment. Particularly, 360 dual-fisheye cameras have been used. The estimated trajectory by the visual SLAM approach is then fused with GPS trajectory by Kalman filtering to correct the geotagging of each 360 panoramic image.
Figure 1. Overview of geolocalization framework. Omnidirectional images are tracked using the EUCM in a variant of ORBSLAM. The estimated trajectory is fused with the GPS coordinates measured for each image. Corrected GPS coordinates are employed for geotagging the equirectangular images associated to each omnidirectional image.
video_360_mission1_paris_gopromax_ver_tetu.mp4
Figure 2. Wearable 360 acquisition system. It consist on a pair of dual-fisheye cameras (GoPro MAX) in a vertical stereo configuration.
One of the applications is to maintain valid GPS measurements at indoor scenarios. For this purpose, classic ORBSLAM has been extended to 360 monocular images where a better performance w.r.t. monocular fisheye V-SLAM approaches has been observed. The opensource code can be found here.
indoor-outdoor-transitions-i3s-mapillary_sQZmbS4y.mp4The acquisition system has been updated to register imagery from perspective cameras (smartphone OnePlus9) where 4 stereo pairs staring at front, back, left and right where used to perform SLAM and update the Mapillary server. Synchronized imagery (8 Perspective cameras + 2 Omnidirectional images + IMU + GPS) is available online for benchmarking purposes in []
sequence_4_sync.mp4Page updated
Google Sites
Report abuse