Research
 

PhD Thesis Title: Driver Fatigue Detection Using Facial Features

Abstract: Lack of concentration of a driver due to fatigue is a major cause of road accidents. This project aimed to develop a video-based method to automatically detect driver fatigue and warn the driver, in order to prevent accidents. I analysed facial features using computer vision, image/signal processing, artificial intelligence and optimisation techniques to determine whether the driver is fatigued. Tests showed that fatigue can be detected with reasonable accuracy. In addition, several contributions were made in face recognition, as recognising the driver can contribute to the fatigue detection process.

Research areas: Computer vision, image/signal processing, pattern recognition, and the application of artificial intelligence and optimisation




Driver Fatigue Detection Using Computer Vision

Automatic identification of driver fatigue has recently become an important research area, and computer vision is considered a more suitable approach. I worked on developing a video-based automated driver fatigue detection system, and investigating into efficiently using ocular measures (e.g., PERCLOS or percentage eye closure, blink rate), and head movements (e.g., head nodding, slouching, postural adjustments) to detect the fatigue level of the driver.

In vision-based methods where ocular measures are used, the main steps of detecting driver fatigue are:

  • face localisation (locating the face automatically) in the first frame
  • locating and tracking the face and the eyes in the subsequent frames
  • estimating the cues (state of the eyes: open/closed), and thereby determining the state of the driver.

Sample video clips of eye tracking that uses a tracking technique based on small dark regions:

Sample video clip of eye tracking that uses my proposed eye model based on optical flow:

 


Face Recognition

Face recognition is a challenging problem in pattern recognition research. Many face recognition methods based on various approaches such as appearance-based approach, deformable approach, etc., have been proposed in the past few decades. In the appearance-based approach, statistical methods such as Principal Component Analysis, Linear Discriminant Analysis, Independent Component Analysis, Bayesian methods, Local Feature Analysis, etc., are used to extract features from the intensities of the image directly. In deformable methods, such as deformable templates, Elastic Bunch Graph Matching (EBGM), Active Shape Model (ASM), Active Appearance Model, etc., a model varies its shape or geometrical structure according to a set of parameters, to fit to an object or part of object in the image.

Inspired by EBGM and ASM, I proposed a new face recognition algorithm called Landmark Model Matching (LMM) optimised by Particle Swarm Optimisation. It is a fully automatic algorithm and can be used for face databases where only one image per person is available. A face is represented by a landmark model consisting of nodes labelled with jets and grey-level profiles. A landmark distribution model is created from a few training images. The model similarity between the landmark distribution model and the deformable landmark model that has to be fitted to the face in the image is maximised by particle swarm optimisation, to find the optimal model to represent the face. Improved results were obtained by this method compared with EBGM.

 

Fig. 1:  The phases of the fully automatic LMM

 

 Features at the nodes are extracted by using a set of Gabor wavelet kernels.

Fig. 2:  The 3D view of the real component of a Gabor wavelet

 

(a) 5 frequencies

(b) 8 orientations

Fig. 3: The 2D view of the 5 frequencies and the 8 orientations of the real components of the wavelets

 

Results on FERET Database:

                                        Table 1: Recognition rates for fb and dup1 data sets
 

Algorithm

Recognition rate (%)

 

fb

dup1

 

Rank 1

Within rank 50

Rank 1

Within rank 50

Fully automatic algorithms

   USC’s EBGM

94.3

98.3

58.3

81.6

   My EBGM

86.3

98.3

43.4

67.2


   LMM

92.8

98.8

52.7

78.7

Partially automatic algorithms

   USC’s EBGM

95.0

99.2

59.1

82.5

   CSU’s EBGM

89.8

99.0

46.3

79.5

   LMM

96.9

99.6

62.0

85.8

 
Extensive details of my face recognition work are available in my PhD thesis: PartofPhDThesis
 
I can provide Matlab programs of my face recognition work (my own implementation of the original EBGM algorithm as well as my proposed LMM algorithm) to anyone interested.
 
 






Publications

     Journal papers:

  • Rajinda Senaratne, Budi Jap, Sara Lal, Arthur Hsu, Saman Halgamuge, Peter Fischer, "Comparing Two Video-Based Techniques for Driver Fatigue Detection: Classification versus optical flow approach", Machine Vision and Applications, SpringerLink, vol. 22, no. 4, pp. 597-618, 2011. (available online in SpringerLink)

     This paper has been cited in the following articles:

  1. H. B. Mitchell, “Spatial Alignment”, Book chapter, Data Fusion: Concepts and Ideas, Springer, pp. 83-107, 2012. (available online here)
  2. B. I. Reiner, E. Kripunski, “Innovation Strategies for Combating Occupational Stress and Fatigue in Medical Imaging”, Journal of Digital Imaging, Springer,  pp. 1-4, 2011. (available online here)
  • Rajinda Senaratne, Saman Halgamuge, Arthur Hsu, "Face Recognition by Extending Elastic Bunch Graph Matching with Particle Swarm Optimization", Journal of Multimedia, Academy Publisher, vol. 4, no. 4, pp. 204-214, 2009. (available online here)

     This paper has been cited in the following articles:

    1. H. M. Hasan, W. A. AL.Jouhar, M. A. Alwan, "3-D Face Recognition Using Improved 3D Mixed Transform", International Journal of Biometrics and Bioinformatics, CSC Journals, vol. 6, no. 1, pp. 11-23, 2012. (available online here)
    2. H. M. Hasan, W. A. AL.Jouhar, M. A. Alwan, "Face Recognition using Improved FFT Based Radon by PSO and PCA Techniques", International Journal of Image Processing, CSC Journals, vol. 6, no. 1, pp. 26-37, 2012. (available online here)
    3. J. H. Kim, "Fully automatic facial recognition algorithm by using Gabor feature based face graph", pp. 31-39, vol. 11, no. 2, 2012. (available online here)
    4. P. Buyssens, "Fusion levels of visible and infrared modalities for face recognition", Proceedings of the Fourth IEEE International Conference on Biometrics: Theory Applications and Systems, pp. 1-6, 2010. (available online here)
  • Rajinda S. Senaratne, Saman K. Halgamuge, "Optimal Weighting of Landmarks for Face Recognition", Journal of Multimedia, Academy Publisher, vol. 1, no. 3, pp. 31-41, 2006. (available online here)

     This paper has been cited in the following articles:

    1. O. Erogul, M. E. Sipahi, Y. Tunca, S. Vurucu, "Recognition of Down syndromes using image analysis", Proceedings of the 14th National Biomedical Engineering Meeting, pp.1-4, 2009. (available online here)


     Conference papers:

  • N. C. Maddage, R. Senaratne, L-S. A. Low, M. Lech, N. Allen, "Video-Based Detection of the Clinical Depression in Adolescents", Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, pp. 3723-3726, Minneapolis, September 2009. (available online in IEEE Xplore)
  • I. Saeed, A. Wang, R. Senaratne, S. Halgamuge, "Using the Active Appearance Model to Detect Driver Fatigue", Proceedings of the Third International Conference on Information and Automation for Sustainability, IEEE, pp. 124-128, Melbourne, December 2007. (available online in IEEE Xplore)

     This paper has been cited in the following articles:

    1. C. Xiong, M. Xie, L. Wang, "Driver Fatigue Detection Based on AdaBoost", Book chapter, Advances in Future Computer and Control Systems, Advances in Intelligent and Soft Computing, Springer, vol. 159, pp. 13-17, 2012. (available online here)
    2. Y. Punsawad, S. Aempedchr, Y. Wongsawat, "Weighted-frequency index for EEG-based mental fatigue alarm system", International journal of applied biomedical engineering, pp. 36-41, vol. 4, no. 1, 2011. (available online here)
    3. H. H. Tsai, Y. S. Lai, Y. C. Zhang, "Using SVM to design facial expression recognition for shape and texture features", Proceedings of the International Conference on Machine Learning and Cybernetics, pp. 2697-2704, 2010. (available online here)
    4. H. Z. Dong, M. Xie, "Real-time driver fatigue detection based on simplified landmarks of AAM", Proceedings of the International Conference on Apperceiving Computing and Intelligence Analysis, pp. 363-366, 2010. (available online here)
  • Rajinda Senaratne, David Hardy, Bill Vanderaa, Saman Halgamuge, "Driver Fatigue Detection by Fusing Multiple Cues", Proceedings of the 4th International Symposium on Neural Networks, Nanjing, Lecture Notes in Computer Science, Springer-Verlag Berlin Heidelberg, vol. 4492, part II, pp. 801-809, June 2007. (available online in SpringerLink)

     This paper has been cited in the following articles:

    1. V. Ivanov, "Fuzzy Methods in Ground Vehicle Engineering: State-of-the-Art and Advanced Applications", Proceedings of the 8th International Conference on Structural Dynamics, EURODYN, 2011. (available online here)
    2. J. Jo, S. J. Lee, H. G. Jung, K. R. Park, J. Kim, "Vision-based method for detecting driver drowsiness and distraction in driver monitoring system", Optical Engineering, vol. 50, no. 12, pp. 127202-1 - 127202-24, 2011. (available online here)
    3. M. E. Funke, "Neuroergonomic and Stress Dynamics Associated with Spatial Uncertainty During Vigilance Task Performance", PhD Thesis, Dept of Psychology of the McMicken College of Arts and Sciences, University of Cincinnati, 2011. (available online here)
    4. H. García, A. Salazar, D. Alvarez, Á. Orozco, "Driving Fatigue Detection Using Active Shape Models", Book chapter, Advances in Visual Computing, Lecture Notes in Computer Science, Springer, vol. 6455, pp. 171-180, 2010. (available online here)
    5. H. Wang, L. B. Zhou, Y. Ying, "A novel approach for real time eye state detection in fatigue awareness system", Proceedings of the IEEE Conference on Robotics Automation and Mechatronics, pp. 528-532, 2010. (available online here)
    6. P. Jimenez, J. Nuevo, L. M. Bergasa, M. A. Sotelo, "Face tracking and pose estimation with automatic three-dimensional model construction", IET Computer Vision, vol. 3, issue 2, pp. 93–102, 2009. (available online here
    7. E. Vural, "Video Based Detection of Driver Fatigue", PhD Thesis, Graduate School of Engineering and Natural Sciences, Sabanci University, 2009. (available online here)
    8. I. J. G. Enríquez, J. M. R. Cortés, J. M. Carballido, R. E. Caldera, M. N. I. Bonilla, "Sistema bimodal de seguridad para la conducción basado en la detección de somnolencia por visión y acelerometría", XI Congreso Mexicano de Robótica. Celaya, Guanajuato, México, 2009. (available online here)
    9. Y. Dong, Z. Hu, K. Uchimura, N. Murayama, "Driver Inattention Monitoring System for Intelligent Vehicles: A Review", IEEE Intelligent Vehicles Symposium, pp. 875-880, Shaanxi, 2009. (available online here)
    10. I. J. G. Enríquez, M. N. I. Bonilla, J. M. R. Cortes, "Segmentación de rostro por color de la piel aplicado a detección de somnolencia en el conductor", 2009. (available online here)
    11. P. Jimenez, J. Nuevo, L. M. Bergasa, "Face Pose Estimation and Tracking Using Automatic 3D Model Construction", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Alaska, 2008. (available online here)
    12. L. M. Bergasa, J. M. Buenaposada, J. Nuevo, P. Jimenez, L. Baumela, "Analysing Driver's Attention Level Using Computer Vision", Proceedings of the 11th International IEEE Conference on Intelligent Transportation Systems, pp. 1149-1154, Beijing, October 2008. (available online here)
  • R. Senaratne, S. Halgamuge, "Optimised Landmark Model Matching for Face Recognition", Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, IEEE Computer Society, Southampton, pp. 120-125, April 2006. (available online in IEEE xplore)

     This paper has been cited in the following articles:

    1. H. Dibeklioglu, A.A. Salah, T. Gevers, "A Statistical Method for 2-D Facial Landmarking", IEEE Transactions on Image Processing, vol. 21, no. 2, pp. 844-858, 2012. (available online here)
    2. H. Dibeklioglu, A.A. Salah, T. Gevers, "Automatic landmarking and alignment for facial expression analysis", IEEE 18th Signal Processing and Communications Applications Conference, pp. 208-211, 2010. (available online here)
    3. O. Erogul, M. E. Sipahi, Y. Tunca, S. Vurucu, "Recognition of Down syndromes using image analysis", Proceedings of the 14th National Biomedical Engineering Meeting, pp.1-4, 2009. (available online here)
    4. B. Gökberk, A. A. Salah, L. Akarun, R. Etheve, D. Riccio, J. L. Dugelay, "3D Face Recognition", Book chapter, Guide to Biometric Reference Systems and Performance Evaluation, Springer London, pp. 263-295, 2009. (available online here)
    5. B. Gokberk, A.A. Salah, N. Alyuz, L. Akarun, "3D Face Recognition: Technology and Applications", Book chapter, Handbook of Remote Biometrics, Advances in Pattern Recognition, Springer London, pp. 217-246, 2009. (available online here)
    6. R. Elizes, "Investigating the Value of Using Sequences of Natural Computing Heuristics to Address Traveling Salesman Problem", PhD Thesis, Pace University, 2008. (available online here)
    7. B. Kroon, A. Hanjalic, S. M. P. Mass, "Eye localization for face matching: is it always useful and under what conditions?", Proceedings of the 2008 International Conference on Content-based Image and Video Retrieval, ACM Press, Ontario, pp. 272-279, 2008. (available online here)
    8. I. Kenny, "A brief history and critique of the developments in of particle swarm optimisation", Technical report, Dept of Computing, The Open University, UK, 2008. (available online here)
    9. G. M. Wagner, "Face Authentication with Pose Adjustment Using Support Vector Machines with a Hausdorff-based Image Kernel", PhD Dissertation, Department of Computer Science, Texas Tech University, 2007. (available online here)
    10. R. Poli, "An Analysis of Publications on Particle Swarm Optimisation Applications", Technical Report CSM-469, Department of Computer Science, University of Essex, UK, 2007. (available online here)
    11. B. Kroon, A. Hanjalic, S. Boughorbel, "Comparison of Face Matching Techniques Under Pose Variation", Proceedings of the 6th ACM International Conference on Image and Video Retrieval, ACM Press, Amsterdam, pp. 272-279, 2007. (available online here)
    12. A. A. Salah, L. Akarun, "3D Facial Feature Localization for Registration", International Workshop on Multimedia Content Representation, Classification and Security, Turkey, Lecture Notes in Computer Science, Springer-Verlag Berlin Heidelberg, vol. 4105, pp. 338-345, 2006. (available online here)
  • R. Senaratne, S. Halgamuge, "Face Localisation for Driver Fatigue Recognition", Proceedings of International Conference on Information and Automation 2005, IEEE, pp. 13-18, Colombo, December 2005. (available online here)
  • R. Senaratne, S. Halgamuge, "Using Particle Swarm Optimisation for Elastic Bunch Graph Matching to Recognise Faces", Proceedings of IEEE TENCON, IEEE, Melbourne, November 2005. (available online in IEEE Xplore)

               This paper has been cited in the following articles:

    1. Y. Owechko, S. Medasani, "Graph-based cognitive swarms for object group recognition in a 3N or greater-dimensional solution space", United States Patent 7672911, 2010. (available online here)
    2. Y.Zheng, Y. Meng, "Object detection and tracking using Bayes-constrained particle swarm optimisation", Book chapter, Computer Vision Research Progress, pp. 245-260, 2007. (available online here)
    3. R. Poli, "An Analysis of Publications on Particle Swarm Optimisation Applications", Technical Report CSM-469, Department of Computer Science, University of Essex, UK, 2007. (available online here)