Computer vision algorithms for wearable devices like smart glasses and AR headsets can offer user-centric experiences by understanding the user's surroundings and aiding in tasks. Current methods typically focus on understanding short-term, category-level object interactions from a static perspective, whereas humans have long-range, instance-based, and opportunistic interactions with objects. To emulate the way of human understanding, the TEAM project aims to develop a computer vision system able to 1) discover important objects to be tracked and highlight their relationships with other objects, 2) track the discovered objects in an instance-based long-term fashion, 3) use information about the detected objects and associated object tracks to form symbolic, high-level and compact memories describing past user interactions, 4) exploit such memories to carry out downstream tasks. This system will comprise an object interaction discovery module, a long-term visual object tracking module, and a memory formation module. The goal is to create technology for long-term user-object interaction understanding, particularly valuable in healthcare for cognitive assessment and training using wearable cameras.

The project is funded by the Italian Ministry of University and Research (MUR) through the PRIN 2022 PNRR measure.