In 2024, for my Bachelor's degree capstone project at PES University, I worked on building a multimodal human-robot interaction system that lets users control a robot using hand gestures and spoken commands, without any hardware sensors.

The core idea was to combine two recognition pipelines: one for gestures, one for speech, and have them work together in real time. For gesture recognition, I used MediaPipe to extract pose and hand landmarks from video frames, feeding those into a custom Transformer-based classifier trained to recognize five gestures - thumbs up, thumbs down, left swipe, right swipe, and stop. The model was trained on a Kaggle dataset of 695 videos and validated on a custom dataset of unseen users recorded across different age groups and genders.

For speech, my teammates integrated OpenAI's Whisper model to transcribe live audio and extract navigation commands from free flowing speech. We also explored using Llama 2 as an alternative command mapper, which handled more natural phrasing like interpreting "come forward by 10 meters" as a structured navigation command, something basic keyword extraction approaches struggled with.

We used OpenAI Gym's Taxi environment to simulate human-robot interaction with actions and spoken commands.