Object PErception and Reconstruction with deep neural Architectures

Pentru varianta în română, apăsați aici. 

Grant: 178PCE/2021, PN-III-P4-ID-PCE-2020-0788

Contracting authority: UEFISCDI

Grant value: 1.198.032 RON

Coordinating institution: Romanian Institute of Science and Technology

Supported by: Oltenia Museum, Restoration and Conservation Lab, History and Archaeology Section 

Implementation period: 04.01.2021-31.12.2023

Abstract

The OPERA project puts forward a fundamental theoretical tripartite framework at the intersection between two frontline areas in computer science: computer vision and machine learning. The technological pipeline will consist of object perception – semantic image inpainting – 3D object reconstruction. Its factual effectiveness will be investigated end-to-end on a real test case scenario coming from another frontier domain, the preservation of the cultural heritage. The importance of the problem is therefore twofold. From the computational viewpoint, this research is exploratory as it allows the application, modification and creation of new deep learning techniques for the specific, delicate task of historical artefact analysis and computational modelling. From the cultural perspective, the three directions are essential for an adequate restoration, while semantic inpainting and 3D object reconstruction are also imperative for the consistent efforts towards the digitization of historical assets. It has been already acknowledged that archaeology exposes the limits of current computer vision techniques, since artefacts are broken, corroded and noisy. Conversely, there is an admitted need of theoretical approaches for understanding the past from its remains found in the present. 

Obtained results

Semantic inpainting and 3D reconstruction

The OPERA project aimed to show how artificial intelligence (deep learning) can offer support towards the restoration of deteriorated cultural heritage assets and exposition of a digital counterpart for the objects.

A deep learning regression model can first estimate the chemical composition at the surface of the object from microscope images. Subsequently, the present corrosion compounds can be delineated and identified through a semantic segmentation model. At this point, the expert can also consider the output from this virtual assistant and proceed with the appropriate treatment for the piece.

Once the chemical repair is achieved, the artistic completion must be performed. At this second stage, a semantic inpainting technique can offer different plausible completions to the expert. Based on this agreed output, a 3D generative model can virtually produce a replica of the object.

Discover how deep learning models guide restoration processes, offering a glimpse into the future of preservation: