Tracking in Egovision
for Applied Memory
A research project funded by the Italian Ministry of University and Research
Project Goal
Computer vision algorithms for wearable devices like smart glasses and AR headsets can offer user-centric experiences by understanding the user's surroundings and aiding in tasks. Current methods typically focus on understanding short-term, category-level object interactions from a static perspective, whereas humans have long-range, instance-based, and opportunistic interactions with objects. To emulate the way of human understanding, the TEAM project aims to develop a computer vision system able to 1) discover important objects to be tracked and highlight their relationships with other objects, 2) track the discovered objects in an instance-based long-term fashion, 3) use information about the detected objects and associated object tracks to form symbolic, high-level and compact memories describing past user interactions, 4) exploit such memories to carry out downstream tasks. This system will comprise an object interaction discovery module, a long-term visual object tracking module, and a memory formation module. The goal is to create technology for long-term user-object interaction understanding, particularly valuable in healthcare for cognitive assessment and training using wearable cameras.
Technologies
User-Related Object Discovery
The user-object interaction discovery component aims to detect important objects and their interactions, by incorporating information about actions, object categories, and relationships between objects.
Long-Term Object Tracking
The object tracking module focuses on long-term tracking of detected objects, using first-person vision cues and merging single with multiple object tracking methodologies to efficiently locate and monitor several objects of interest.
Memory Forming
This module focuses on creating semantic memories from detected and tracked interactions and relationships, by exploring innovative neural object representation methods.
Downstream Tasks
The system aims to use formed memories to address future downstream tasks answering questions of the kind “where were the keys last seen?” (“The keys were last seen in your bag.”).
Research Units
University of Udine
University of Catania
Project Grant
The project is funded by the Italian Ministry of University and Research (MUR) through the PRIN 2022 PNRR measure.