CVPR 2024 Tutorial
Contactless AI Healthcare using Cameras and Wireless Sensors
Date and time: June, 2024 (physical + virtual)
Zoom link: To be announced
Abstract
Understanding people and extracting health-related metrics is an emerging research topic in computer vision that has grown rapidly recently. Without the need of any physical contact of the human body, cameras have been used to measure vital signs remotely (e.g. heart rate, heart rate variability, respiration rate, blood oxygenation saturation, pulse transit time, body temperature, etc.) from an image sequence of the skin or body, which leads to contactless, continuous and comfortable heath monitoring. The use of cameras also enables the measurement of human behaviors/activities and high-level visual semantic/contextual information leveraging computer vision and machine learning techniques. Understanding of the environment around the people is also a unique advantage of cameras compared to the contact bio-sensors (e.g., wearables), which facilitates better understanding of human and scene for health monitoring. In addition to camera based approach, Radio Frequency (RF) based methods for health monitoring have also been proposed, using Radar, WiFi, RFID, and acoustic signals. Radar based methods mainly use Doppler/UWB/FMCW radar for health monitoring. They can obtain high monitoring accuracy for different applications such as sleeping staging and posture estimation. By using off-the-shelf WiFi device, for example WiFi RSS and CSI data from commodity WiFi NIC, we can monitor breathing and heart rates for single and multiple persons. For acoustic based vital sign monitoring, the speaker and microphone of smartphones are used to build sonar based sensing system to monitor breathing and heart rates. The rapid developments of computer vision and RF sensing also give rise to new multi-modal learning techniques that expand the sensing capability by combining two modalities, while minimizing the need of human labels. The hybrid approach may further improve the performance of monitoring, such as using the camera images as beacon to gear human activity learning for the RF signals. The contactless monitoring of camera and RF will bring a rich set of compelling healthcare applications that directly improve upon contact-based monitoring solutions and improve people’s care experience and quality of life, called “AI health monitoring”, such as in care units of the hospital (patient monitoring), sleep/senior centers (sleep disorders like apnea), assisted-living homes (baby and elderly care), telemedicine and e-health, fitness and sports, driver monitoring in automotive, etc.
Speakers
Wenjin Wang
Southern University of Science and Technology
Daniel McDuff
Google Research
Xuyu Wang
Florida International University
Schedule
Part 1: Camera based physiological measurement and its clinical trials in China
(1) Measured vital signs and activities for healthcare
a. Pulse rate: different measurement principles; RGB and IR setups (multi-wavelength camera, multi-camera system, RGB2NIR system, low-light cameras, etc.); core algorithms for pulse signal extraction; solutions to improve robustness
b. Respiration rate: different measurement principles (motion based, PPG based); core algorithms for respiratory signal extraction (PixFlow); limitations and challenges
c. Blood oxygen saturation: different multi-wavelength settings (red-IR, full-IR, visible); core algorithms for SpO2 signal extraction; solutions to improve robustness (reduce parallax between cameras, better wavelength selection)
d. Health-related body signs: baby comfort/discomfort/pain monitoring, sleep posture detection, patient motion actigraphy, etc.
(2) Clinical trials in China
a. ICU deteriorated patient monitoring using CCTV cameras
b. NICU pre-term infant monitoring
c. Lung rehabilitation analysis based on imaging
d. Contactless vital signs based sleep staging in sleep center
e. Baby and elderly care
(3) Q & A (10 min)
Part 2: Deep Learning for Camera Physiological Measurement
1. Introduction to neural approaches to camera physiological measurement.
a. Datasets and preprocessing
b. Network architectures and loss functions
c. Evaluation and the state-of-the-art results
2. Supervised vs. self-supervised learning.
a. Self-supervised learning techniques.
b. Data augmentation and synthetics.
3. What is next for deep learning for camera physiological measurement? Unsolved challenges for the field.
a. Designing and evaluating personalized models
b. On device performance.
Q&A (10 min)
Part 3: Wireless based vital sign monitoring and activity monitoring
(1) Radar based vital sign monitoring
a. Radar Techniques: Continuous-wave (CW) radar, Frequency-Modulated Continuous-Wave (FMCW) Radar, Ultra Wide-band (UWB) radar; Commodity Radar Vital Sign Devices including TI mmWave FMCW Radar and XeThru X4 UWB radar.
b. Radar based vital sign monitoring and activity recognition applications: breathing and heart rate monitoring, apnea detection, activity recognition using deep Learning.
(2) Acoustic based vital sign monitoring with smartphones
a. FMCW based vital sign monitoring with acoustic signal: acoustic signal based breathing monitoring on smartphones with FMCW.
b. CW based vital sign monitoring with acoustic signal: acoustic signal based breathing monitoring on smartphones with CW and the detail Sonarbeat system.
(3) WiFi based vital sign monitoring
a. WiFi system: WiFi techniques; WiFi received signal strength (RSS) and channel state information (CSI).
b. WiFi based vital sign monitoring: WiFi CSI for breathing and heart rate monitoring, WiFi CSI values for monitoring multi-person breathing beats, robust breathing monitoring with CSI amplitude and phase difference.
(4) Q & A (10 min)