The aim of the HERON project is the development of digital twin (DT) technology for smart home applications. DT is a digital representation of a physical entity (including geometric, functional, and usage models) that has the following characteristics: being ultra-realistic and fully consistent with the physical entity, continuous and automatic update to be able to adapt to changes, semantic content to support intelligent decision making, and implementation in a decentralized structure to collaborate with other DTs. The principle of HERON is the development of DT based on three entities: 1) a smart home, 2) assistive robotic devices, and 3) elderlies or people with reduced mobility. State-of-the-art artificial intelligence (AI) and computer vision solutions, and 3D robotics simulators are used to implement this objective. Such a DT can have applications in the optimization of robots and smart home settings. For example, finding the optimal number and configuration of sensors especially when new robots or users are introduced. Learning of edge cases (rare situations) for the safety of human users by pushing those events from simulation to real-world and vice-versa, thanks to DT for enabling a bi-directional link between simulation and real-world, etc.

In the first part of the project, a prototype of a DT was developed. The steps of DT creation are defined as follows. The smart home is scanned using a hand-held RGB-D camera and a SLAM algorithm to build a reconstruction of it. A modeling software is used to construct the CAD models of the home and objects inside it from the scan. The CAD models (augmented with different properties, for example, being static or movable) and a robot model are imported into a robotic simulator. A proof-of-concept is developed to enable semantic understanding of objects. A set of experiments were performed to compare a Turtlebot's behavior (such as traversed trajectories and the generated 2D maps) in the actual and simulated DT environments. For more information please refer to our paper.

Dissemination:

The following is a summary of in-progress works, publications, and technological development contributions. The project source codes and associated resources are available online.

• Digital Twin Driven Smart Home.

A proof of concept of a DT-driven smart home powered by several packages from ROS and Gazebo. It includes aspects of 3D representation, robot navigation, and semantics.

Url: github.com/alirezaasvadi/DTSH/

Associated publications: A. Asvadi, A. Mitriakov, C. Lohr, P. Papadakis, Digital Twin Driven Smart Home: A feasibility study, (work in progress).

• Cubic marker-based 3D reconstruction.

An open-source pipeline for the 3D reconstruction of an indoor environment using cubic fiducial markers and RGBD images acquired from a single RGB-D camera.

Url: github.com/alirezaasvadi/CM3DR

Associated publications: A. Asvadi, C. Lohr, P. Papadakis, 3D Reconstruction Using a RGBD Camera and Cubic Fiducial Markers, (work in progress).

Project perimeter: The HERON was 18 months project (started on 08 December 2020 until 03 June 2022). Alireza Asvadi, a R&D Engineer (CDD), was the main force working on this project under the supervision of Panagiotis Papadakis and Christophe Lohr. The research was done in RAMBO team of Lab-STICC (UMR 6285) in the IMT Atlantique Bretagne-Pays de la Loire, Brest, France. The Experiment'HAAL (Human Ambient Assisted Living) platform was used for the development of experiments in the project.

Sponsors: The project was jointly funded by the Brittany Region under the program SAD 2019 (Strategy for Durable Development), the Department of Finistere under the program ARPE (Aid to Emergent Research Program), and the Chaire M@D (Maintien en Domicile).