The EXTRA EYE project aims to decode human interactions in diverse environments. We plan to combine first-person and third-person views to provide a holistic understanding of human's activities and behavior. EXTRA EYE will rely on three modules: a First-Person View (FPV) module which aims to understand take/release user-object interactions from egocentric vision, a Third-Person View (TPV) module which processes exocentric signals captured from fixed cameras, accompanied with an aggregation unit that also integrates information from the other components to perform context-specific behavioral analysis, and an Object Tracking (OT) module which supports the former two components by tracking objects of interest from both the third- and first-person points of view. The system is will be useful in many scenarios in which human behavior is largely determined by the way in which people interact with the objects available in the scene. As a clear application context for the development and test of EXTRA EYE, we will consider retail stores as goal-driven, interaction-rich scenarios. The technologies emerging from this project aim to offer a new perspective on human behavior understanding from Computer Vision, informing the construction of systems able to user experiences in a variety of environments.

The project is funded by the Italian Ministry of University and Research (MUR) through the PRIN 2022 measure.