Introduction

The goal of this project is to create a mobile application that accurately measures the heart rate of a person using the video captured by the front camera of a mobile phone. For this project, we referred to the paper “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone” presented in 2012 Annual International Conference of the IEEE.

Platform and Tools Specifications

· Development Environment – Android (Supports KitKat, JellyBean, Icecream Sandwich)

· Testing Environment – Android and MATLAB R2014b, R2014a

· Integrated Development Environment – Android Studio, Eclipse

· External Libraries – OpenCV4Android 2.4.9

· Mobile Device – Samsung Galaxy S4

· Camera (Front) - 2 MP, 1080p@30fps

Overview

We extract heart rate and beat lengths from video by measuring subtle change in skin color due to the influx of blood at each beat. A camera captures these subtle variations and we extract the heart beat signal from the average pixel intensity value variation and then calculate the heart rate.

In order to achieve this, we have implemented the following steps:

1. Viola Jones Face Detection and Tracking from OpenCV

2. Calculating the average pixel value of Red,Green and Blue Channels of each Frame

3. Cubic Spline Interpolation

4. Temporal Filtering

5. Independent Component Analysis

6. Peak Detection

Method

Viola Jones Face Detection and Tracking from OpenCV

We are using the “lbpcascade_frontalface.xml” Classifier for the Face Detection. We have downloaded a sample Face Detection and Tracking code from the OpenCV sample code and implemented this in our project.

Cubic Spline Interpolation

An average person’s heart rate is between 40 and 150 beats per minute that translates to [0.67, 2.5] Hz in the frequency domain. Our frame rate varies in between 10 and 11 frames per second. The authors perform interpolation on their signal to obtain a 250 Hz signal.

Due to the hardware constraints of a mobile device, we could perform a Cubic Spline Interpolation by a factor of 2. Therefore, our samples improved from 10-11 frames per second to 20-22 frames per second.

Temporal Filtering

We are implementing a Band-Pass Filter because the low and high-frequency components of the signal negatively impacts the heart rate response. For this, we are implementing a 5^th Order Butterworth Band-Pass Filter with a pass band of [0.75, 5] Hz. In order to achieve better results, we are using the filter coefficients that are generated using MATLAB. To improve the computational efficiency of the mobile device, these filter coefficients ‘a’ and ‘b’ have been generated in the Sampling Frequency range of 20 Hz to 22 Hz and with a pass band of [0.75, 5] Hz using MATLAB.

In the above figure, as there are no frequency components above and below the threshold, the response is the same.

Independent Component Analysis

The underlying source signal of interest is the variation of “Average Skin Color” value caused by the cardiovascular pulse. Whereas the average skin color value is a mixture of signals emanating from various other sources e.g. Varying Lighting Conditions. We must decompose this mixed motion into sub-signals to isolate the pulse.

Since we have 3 signals (the Red, Green and Blue Channels) to work with, we need to calculate the most periodic signal that relates to the Heart Rate. In order to achieve this functionality, the paper proposes to implement Independent Component Analysis on the sources. Due to hardware constraints on a mobile device, we implement Fast-ICA (Features from Accelerated Segmentation Test) which is a form of Independent Component Analysis. Also, we limit the number of computations to ‘2.’

Thus, we pull out the most periodic signal after performing Independent Component Analysis

Peak Detection

After pulling out the most periodic signal we performed a frequency domain analysis to calculate the period of the signal. The results were poor for elevated heart rates, and thus we couldn’t use a DFT to calculate the heart rate. Instead, we used Peak Detection technique with a “calibrated” Window Size and Stringency Level to detect the “valid” peaks of the signal. We take the average time difference between the peaks and calculate the Heart Rate.

Challenges Faced

We faced tons of challenges while implementing this project. Some of them are listed below:

• The complexity of this project was enormous, and the results were uncertain. We had to first simulate the entire project on MATLAB to even consider this project as feasible.
• There was No source code available for this project.
• Even after satisfactory test results in MATLAB, there was no guarantee that this project would work on an Android device due to its hardware constraints (processor, memory, etc.).
• Next, we faced a challenge in integrating OpenCV with Android as there was no support for the latest version of Android 5.0 (Lollipop).
• While performing Face Detection, we managed to get a frame rate of only 10fps. This affected the performance of the system.
• Cubic Spline Interpolation was a big challenge to implement on Android.
• Implementing Butterworth filter for our system with varying frame rate was quite taxing. We reached a stage where the size our code for this exceeded limits the tools could handle. We backtracked on precision.
• Independent Component Analysis (ICA) was the most challenging task as we could limit the number of iterations to a maximum of 2. We banked on this data as our hands were tied because of the hardware constraints of a mobile device.
• A lot of experiments were performed on Peak Detection by tweaking the window size and stringency level to get accurate results.
• Finally, the test environments were challenging for this project. We had to keep the relative motion between the mobile phone and our head to a minimum. By varying the lighting conditions, the response would change drastically. Our morals were low during many stages of this project, but we never gave up. We continued researching on different areas and reworked the code as many times as needed.

Despite facing these challenges, we achieved the goal of our project.

Conclusion

We dreamed of creating a Real-time heart rate tracking application and we made it into a reality! This application works like a charm but has certain limitations. Based on our experiments, we found that our application does not work well when a person breathes heavily. Also, we noticed certain bugs in the OpenCV face detection, where it detects false positives thereby affects our performance negatively. For best results, we need to reduce the camera shake while capturing the video.

Project Demonstration

We have prepared a short video demonstration (~5 min) of our project and uploaded it on YouTube.

Future Work

· Monitoring the heart rate and displaying it continuously

· Performing multiple heart rate calculation when more than one person is detected

· Can be implemented for Rear-facing camera as well

· Integrating the Accelerometer module to the project to minimize the camera shake error