Smart Cart
Unity, Meta quest, YOLO, MobileNet, NCNN
Embedded Files
Team
Fullstack Programmer : Enzo LEBLANC-GALBERT
ML Programmer : Raphaël TAN
Engine
Unity 6 (C#, HLSL)
Platform
MR (Meta Quest)
Main things I did
Training
Creation of databases from video
Training of YOLO model from already existing database with augmentations
Training of MobileNet V3 model from my own database with augmentations
Inference
Realtime inference on Unity in true async using NCNN
Algorithm to remove / avoid inference on false positives
Queries to OpenAI for product recommendations
Optimisation of token use with smart prompts and avoiding requests if recommendation can be computed (calories, eco score...)
Product filtering (Allergies..)
Device
Running the pipeline in realtime on the Meta Quest 3s using the camera feed as input
Contextual UI for product information
Description
Smart Cart transforms the Meta Quest 3 into an intelligent shopping assistant.
Select a specific mode like Budget, Vegan, or Eco-friendly. Simply dictate your grocery list out loud and validate it. As you walk through the store, the headset tracks and analyzes products on the shelves in real-time.
Smart Cart will pinpoint the best options directly in your field of view, allowing you to stop reading labels and just grab and go.
The app takes the video feed from your meta quest and extracts the exact barcodes of the products on the shelf, allowing for some very accurate suggestions! Allergies, tastes, everything can be filtered!
Page updated
Google Sites
Report abuse