Introduction

The goal of this project is to create a mobile application that accurately measures the heart rate of a person using the video captured by the front camera of a mobile phone. For this project, we referred to the paper “Detecting Pulse from Head Motions in Video” presented in CVPR 2013 [1].

Platform and Tools Specifications

• Development Environment – Android (Supports KitKat, JellyBean, Icecream Sandwich)
• Testing Environment – Android and MATLAB R2014b, R2014a
• Integrated Development Environment – Android Studio, Eclipse
• External Libraries – OpenCV4Android 2.4.9
• Mobile Device – Samsung Galaxy S4
• Camera (Front) - 2 MP, 1080p@30fps

Overview

We extract heart rate and beat lengths from video by measuring subtle head motion caused by the Newtonian reaction to the influx of blood at each beat. A rough estimate of the displacement of head movement is 5 mm.

In order to achieve this, we have implemented the following steps:

1. Viola Jones Face Detection and Tracking from OpenCV.
2. Cropping out the nose region and calculating the feature points around it.
3. Lucas-Kanade Tracking for the feature points.
4. Cubic Spline Interpolation.
5. Temporal Filtering.
6. Independent Component Analysis.
7. Peak Detection.

Method

Viola Jones Face Detection and Tracking from OpenCV

We are using the “lbpcascade_frontalface.xml” Classifier for the Face Detection. We have downloaded a sample Face Detection and Tracking code from the OpenCV sample code and implemented this in our project [2].

Cropping out the nose region and calculating the feature points around it

Instead of calculating the feature points from the entire face, we found that the feature points around the “Nose” region gives us the best results.

In order to do this, we are cropping out the “Nose” region and selecting the 10 best feature points. We are using the method “goodFeaturesToTrack” from the “Imgproc” Class of OpenCV to get the results. Also, we are using the “cornerMinEigenVal” instead of “cornerHarris.” [3]

Lucas-Kanade Tracking for the feature points

Once we have the feature points, we calculate an optical flow for a sparse feature set using the iterative Lucas-Kanade [4] method with pyramids. By implementing this method, we get accurate trajectories of all the 10 feature points. We track the feature points for 50 samples of data that translates to about 8 seconds of video with a frame rate of 6 fps.

We will deal with the low fps in the “Challenges faced” section. For now, we have to take a video for about 8 seconds before we proceed with the other stages of this project.

Cubic Spline Interpolation

An average person’s heart rate is between 40 and 150 beats per minute that translates to [0.67, 2.5] Hz in the frequency domain. Our frame rate varies in between 6 and 7 frames per second. The authors perform interpolation on their signal to obtain a 250 Hz signal.

Due to the hardware constraints of a mobile device, we could perform a Cubic Spline Interpolation by a factor of 2. Therefore, our samples improved from 6-7 frames per second to 12-14 frames per second [5], [6].

Temporal Filtering

We are implementing a Band-Pass Filter because the low and high-frequency components of the signal negatively impacts the heart rate response. For this, we are implementing a 5th Order Butterworth BandPass Filter with a pass band of [0.75, 5] Hz. In order to achieve better results, we are using the filter coefficients that are generated using MATLAB. To improve the computational efficiency of the mobile device, these filter coefficients ‘a’ and ‘b’ have been generated in the Sampling Frequency range of 12 Hz to 14 Hz and with a pass band of [0.75, 5] Hz using MATLAB [7], [8].

In the above figure, as there are no frequency components above and below the threshold, the response is the same.

Independent Component Analysis

The underlying source signal of interest is the movement of the head caused by the cardiovascular pulse. The feature point trajectories are a mixture of this movement as well as other motions caused by sources like respiration, vestibular activity, and changes in facial expression. We must decompose this mixed motion into sub-signals to isolate the pulse.

Since we have 10 best feature points to work with, we need to calculate the most periodic signal that relates to the Heart Rate. In order to achieve this functionality, the paper proposes to implement Principal Component Analysis on the sources. Due to hardware constraints on a mobile device, we implement Fast- ICA (Features from Accelerated Segmentation Test) instead which is a form of Independent Component Analysis. Also, we limit the number of computations to ‘2.’ [9], [10]

Thus, we pull out the most periodic signal after performing Independent Component Analysis.

Peak Detection

After pulling out the most periodic signal we performed a frequency domain analysis to calculate the period of the signal. The results were poor for elevated heart rates, and thus we couldn’t use a DFT to calculate the heart rate. Instead, we used Peak Detection technique with a “calibrated” Window Size and Stringency Level to detect the “valid” peaks of the signal. We take the average time difference between the peaks and calculate the Heart Rate. [11] [12]

Challenges Faced

We faced tons of challenges while implementing this project. Some of them are listed below:

1. The complexity of this project was enormous, and the results were uncertain. We had to first simulate the entire project on MATLAB to even consider this project as feasible.
2. There was No source code available for this project.
3. Even after satisfactory test results in MATLAB, there was no guarantee that this project would work on an Android device due to its hardware constraints (processor, memory, etc.).
4. Next, we faced a challenge in integrating OpenCV with Android as there was no support for the latest version of Android 5.0 (Lollipop).
5. While performing Face Detection, we managed to get a frame rate of 10 fps but when we implemented Real-time Face Tracking our frame rate fell to 6-7 fps. This affected the performance of the system as we were trying to track a shift of mere 5 mm change.
6. We tested many regions of the face to get best feature points and eventually came up with the “nose” region.
7. Cubic Spline Interpolation was a big challenge to implement on Android.
8. Implementing Butterworth filter for our system with varying frame rate was quite taxing. We reached a stage where the size our code for this exceeded limits the tools could handle. We backtracked on precision.
9. Independent Component Analysis (ICA) was the most challenging task as we could limit the number of iterations to a maximum of 2. We banked on this data as our hands were tied because of the hardware constraints of a mobile device.
10. A lot of experiments were performed on Peak Detection by tweaking the window size and stringency level to get accurate results.
11. Finally, the test environments were challenging for this project. We had to keep the relative motion between the mobile phone and our head to a minimum. By varying the lighting conditions, the response would change drastically. Our morals were low during many stages of this project, but we never gave up. We continued researching on different areas and reworked the code as many times as needed.

Despite facing these challenges, we achieved the goal of our project.

Conclusion

We dreamed of creating a Real-time heart rate tracking application and we made it into a reality! This application works like a charm but has certain limitations. Based on our experiments, we found that our application does not work well when a person breathes heavily. Also, we noticed certain bugs in the OpenCV face detection, where it detects false positives thereby affects our performance negatively. For best results, we need to reduce the camera shake while capturing the video.

Project Demonstration

We have prepared a short video demonstration (~5 min) of our project and uploaded it on YouTube.

Future Work

• Monitoring the heart rate and displaying it continuously.
• Performing multiple heart rate calculation when more than one person is detected.
• Can be implemented for Rear-facing camera as well.
• Integrating the Accelerometer module to the project to minimize the camera shake error.

References

[1] G. Balakrishnan, F. Durand and J. Guttag, "Detecting Pulse from Head Motions in Video," CVPR , 2013.

[2] O. H. Jensen, "Viola–Jones object detection framework," 2001. [Online]. Available: http://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework.

[3] S. a. C. Tomasi, "OpenCV Documentation," 1994. [Online]. Available: Feature Detection.

[4] B. a. K. T. Lucas, "Motion Analysis and Object Tracking," 1981. [Online]. Available: http://docs.opencv.org/modules/video/doc/motion_analysis_and_object_tracking.html.

[5] E. W. Weisstein, "MathWorld--A Wolfram Web Resource," January 2014. [Online]. Available: http://mathworld.wolfram.com/CubicSpline.html.

[6] W. contributors, "Spline interpolation," 2015. [Online]. Available: http://en.wikipedia.org/w/index.php?title=Spline_interpolation&oldid=648319849.

[7] W. contributors, "Butterworth filter," 2015. [Online]. Available: http://en.wikipedia.org/w/index.php?title=Butterworth_filter&oldid=645607662.

[8] "Butterworth filter design," [Online]. Available: http://in.mathworks.com/help/signal/ref/butter.html.

[9] M.-Z. D. J. M. a. R. W. P. Poh, "Non-contact, automated cardiac pulse measurements using video imaging and blind source separation," Optics Express, 2010.

[10] B. Moore, "PCA and ICA Package," [Online]. Available: http://www.mathworks.com/matlabcentral/fileexchange/38300-pca-and-ica-package.

[11] Mathworks, "Peak Analysis," Mathworks, [Online]. Available: http://in.mathworks.com/help/signal/examples/peak-analysis.html.

[12] M. Muller, F. Hirth and A. Koch, "Comparison of different peak detection algorithms with regards to spectrometric fiber Bragg grating interrogation systems," IEEE, 2009.