How Does Camera-Based Physiological Sensing Work?

As the heart beats, the periodic change of blood flow can be captured by optical sensors, such as a pulse oximeter clipped to a person’s fingertip measuring the photoplethysmogram, or PPG. The blood flow also leads to subtle color changes on the surface of a person's face. The light that reflects off of a person's face is modulated by these subtle color fluctuations in the skin's tissue. The strength of these fluctuations is typically strong on the skin tissue of a person's face. Hence, the light that reflects off of a person's face and into a camera's sensor carries pulsatile information in it, thereby enabling remote PPG (rPPG, or sometimes known as imaging PPG or video PPG) as a non-contact way to track physiological activity by analyzing face videos. The primary goal of rPPG methods is to isolate pulsatile from non-pulsatile information in the skin pixels of a video (such as camera noise and specular reflection).

Building reliable video-based physiological monitoring

The main purposes of our Video-Physio study are threefold. First, the study will evaluate the accuracy of a video-based photoplethysmography (PPG) for measuring physiological conditions such as heart rate (HR) and respiratory rate (RR), and inferring other physiological information such as potentially the blood oxygen levels and electrocardiogram (ECG) signals from videos. Second, the study will elucidate sensing factors and algorithmic factors affecting the performance of rPPG based physiological sensing and inference, and their relations with those obtained by contact-based sensors and devices. Third, the study will examine the relationship between the health conditions of interests (such as potential early signs of the COVID-19 infections or other related illness) and the following data: the physiological data extractable from video alone, the vital sign data by established contact-based sensors/devices alone, and these two combined.

Privacy protection under video-based physiological monitoring

While physiological monitoring from face videos may lead to many fruitful opportunities for improving healthcare monitoring, the use of such data does not come without its own challenges and risks. Face videos are often considered sensitive information since they contain identifiable appearance features. Our lab is developing tools to protect identity privacy in the context of video-based physiological monitoring. We are conducting a survey to understand users' privacy preferences under video-based physiological monitoring. Through this research, we aim to create a more secure video monitoring system in which everyone feels comfortable.

Join our studies -- Together, we can advance science and technology!

See more information on the tab "Join Our Studies" (under "More").

Talks & Tutorials

Tutorials at WIFS'21 on Physiological Forensics and CVPR'22.

Dr. Wu's Talk in Spring 2021 for the UMD Faraway Fourier Talk (FFT) series

Peer-Reviewed Publications: (** more updates coming)

[1] Qiang Zhu, Chau-Wai Wong, Chang-Hong Fu, and Min Wu, “Fitness heart rate measurement using face videos,” IEEE International Conference on Image Processing (ICIP’17), Beijing, China, 17–20 Sep. 2017. [IEEE Xplore] [Postprint] [Slides] [Demo Videos: Elliptical Machine, Treadmill]

[2] Qiang Zhu, Mingliang Chen, Chau-Wai Wong, and Min Wu, "Adaptive Multi-Trace Carving for Robust Frequency Tracking in Forensic Applications," in IEEE Trans. on Info. Forensics and Security, vol. 16, pp. 1174-1189, 2021, doi: 10.1109/TIFS.2020.3030182. (See also an earlier version at Asilomar'18, [IEEE Xplore] [Poster])

[3] Mingliang Chen, Qiang Zhu, Min Wu and Quanzeng Wang, "Modulation Model of the Photoplethysmography Signal for Vital Sign Extraction," in IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 4, pp. 969-977, April 2021, doi: 10.1109/JBHI.2020.3013811.

[4] M. Chen, X. Liao, and M. Wu: “PulseEdit: Editing Physiological Signal in Facial Videos for Privacy Protection,” IEEE Trans. on Info. Forensics and Security, accepted 2021, to appear 2022, DOI 10.1109/TIFS.2022.3142993.

[5] Z. Lazri, Q. Zhu, M. Chen, M. Wu, and Q. Wang: “Detecting Essential Landmarks Directly in Thermal Images for Remote Body Temperature and Respiratory Rate Measurement with a Two-Phase System,” accepted by IEEE Access (open access), March 2022, to appear.

Additional publications and preprints:

J. Mathew, X. Tian, C-W. Wong, S. Ho, D.K. Milton, and M. Wu: “Remote Blood Oxygen Estimation from Videos Using Neural Networks,” submitted for journal publication, under revision. Preprint available through <https://arxiv.org/pdf/2107.05087>.

... more to come.

Patent Applications:

[1] Min Wu, Chau-Wai Wong, Qiang Zhu, Chang-Hong Fu, and Jiahao Su, “Heart rate measurement for fitness exercises using video,” PCT/US2018/051342, filed Sept. 2018.

[2] Qiang Zhu, Mingliang Chen, Min Wu, Chau-Wai Wong, “Methods and apparatus for tracking weak signal traces,” Patent Application, filed Sept. 2019.

[3] M. Chen, Q. Zhu, Q. Wang, and M. Wu: “Method for Vital Sign Estimation Using Photoplethysmography Signals,” provisional patent filing August 2020.

[4] Q. Zhu, M. Chen, Z. Lazri, C-W. Wong, and M. Wu: “System and Method for Heart Rate Measurement Using Facial Video,” provisional patent filing August 2020, formal filing 2021.

[5] X. Tian, M. Wu, J. R. Mathew, and C-W. Wong: “Contactless Image-Based Blood Oxygen Estimation,” provisional patent filing June 2021.

Funding Acknowledgement:

We gratefully acknowledge the funding support from multiple programs on various aspects this exciting line of research: NSF i-Corp, NSF RAPID from ECCS, Maryland MII, and UM Venture MDDF COVID Challenge.