Embedded Files
Research Team
Research Team
Noah Emmert
Joseph Sanuk
Demetrius Hullum Scott
Ryan Hardek
Graduate Mentor: Nadim Mahmud
Advisor: Dr. Vaskar Raychoudhury
Abstract/Introduction
People who are blind or have low vision (BLV) face daily challenges navigating unfamiliar or complex environments safely and independently.
Existing camera-based systems often fail due to privacy concerns, poor lighting, or occlusions.
mmWave radar addresses these due to their cost effectiveness and ability to be used in low-light conditions.
By utilizing an Intel RealSense camera during training, highresolution RGB-D data can be cross-referenced with radar point clouds to automate ground-truth labeling.
Radar signals have the ability to be processed into situational awareness maps that interface with a Large Language Model (LLM)
Proposed System
Proposed System
We propose an end-to-end, portable navigational system that integrates mmWave radar and an LLM to provide real-time, privacy-preserving guidance for blind/low vision users.
Our sytem:
Establishes spatial correspondence between the TI IWR1843BOOST radar and Intel RealSense D435i camera using a buffering-immune temporal alignment and a four-stage robust extrinsic solver.
Utilizes high-resolution RGB-D data from the RealSense camera exclusively during the training phase to automate ground-truth labeling for radar point clouds via OneFormer semantic segmentation.
Uses Radar signals that are processed by a Dynamic Graph Convolutional Neural Network (DGCNN) with temporal attention, utilizing 20 engineered features to classify environments into structure, floor, and human categories.
Integrates a mobile NVIDIA Jetson Orin Nano Super, translating 3D situational awareness maps into intuitive spoken directions via an Ollama-integrated Llama 3.2 model.
Methods
Sensor Calibration: A two-stage pipeline established spatial alignment between sensors. Temporal alignment utilized a buffering-immune, frame-number timebase and an affine transformation to reconcile independent clocks. Extrinsic calibration derived a rigid-body transform by correlating radar cluster centroids with AprilTag-guided depth back-projections of a corner reflector, using a four-stage robust solver (RANSAC and IRLS) to achieve sub-10 pixel reprojection error.
Data Acquisition and Pre-processing: Using a Texas Instruments IWR1843BOOST radar and DCA1000EVM board, raw ADC data was captured at 10Hz and converted to 3D point clouds via a vectorized CFAR detector. For training, an Intel RealSense D435i provided synchronized RGB-D data for automated ground-truth labeling using OneFormer. A physical Z-filter was applied to eliminate multipath ghost returns below the floor.
Model Architecture and Feature Engineering: The system employs a KPCONV architecture.. Each point uses 20 features, including base 3D coordinates and Doppler velocity, "soft structural" features (height above floor and surface angles), and engineered discriminators like range-normalized elevation and aggregate Doppler statistics to separate static structures from moving humans.
Real-time Inference and System Integration: Deployed on an NVIDIA Jetson Orin Nano Super, the system uses a three-threaded architecture:
Recorder Thread: Captures continuous 3-second radar sessions.
Main Thread: Processes ADC data to point clouds and executes DGCNN inference for structure, floor, and human classification.
LLM Worker Thread: Aggregates predictions into spatial sectors and uses Llama 3.2 3B via Ollama and Piper TTS to generate spoken navigation prompts.
Results/Findings and Discussion
Results/Findings and Discussion
Calibration validation overlay showing mmWave radar detections projected onto the synchronized camera feed. Each colored marker represents a radar-detected point mapped into the camera's field of view, with track IDs and range estimates confirming that the radar-to-camera extrinsic calibration accurately aligns radar measurements with the corresponding physical objects in the scene.
This is an example frame of our system overlayed with radar points that splits the video into 3 directions and provides navigational feedback based on the location of nearby people and objects.
Conclusions and Future Study
Conclusions and Future Study
Our model successfully integrated temporal attention and engineered features, such as range-normalized elevation, to achieve a 72.4% balanced accuracy in distinguishing between floor, human, and structural classes.
By utilizing mmWave radar for real-time inference, the system demonstrated reliable obstacle detection and spatial mapping that is immune to lighting constraints and privacy concerns
We produced a functional, end-to-end prototype on the NVIDIA Jetson Orin Nano Super that maintains a continuous loop of data capture, AI classification, and LLM-driven natural language guidance.
The two-stage calibration pipeline achieved a sub-5 cm mean correspondence error, validating that radar detections can be accurately mapped to the physical world for safe wheelchair navigation
NACE Career Readiness Competencies
NACE Career Readiness Competencies
Critical Thinking: We performed diagnostic analyses to solve class issues.
Technology: We engineered an end-to-end technical stack on the NVIDIA Jetson Orin Nano Super, integrating TI mmWave radar hardware, DGCNN deep learning models, and Llama 3.2 LLMs.
Professionalism: We maintained a comprehensive technical record of the development history.
Teamwork: We collaborated on a multimodal sensor platform integration to ensure synchronized operation between hardware recording, AI processing, and user feedback.
Page updated
Report abuse