Face Detection with Deep Learning using Multi Tasked Cascased CNN
Face Detection with Deep Learning using Multi Tasked Cascased CNN
Embedded Files
Detecting faces in a photograph is easily solved by humans, although has historically been challenging for computers given the dynamic nature of faces. For example, faces must be detected regardless of orientation or angle they are facing, light levels, clothing, accessories, hair color, facial hair, makeup, age, and so on.
Detecting faces in a photograph is easily solved by humans, although has historically been challenging for computers given the dynamic nature of faces. For example, faces must be detected regardless of orientation or angle they are facing, light levels, clothing, accessories, hair color, facial hair, makeup, age, and so on.
In my previous article, I’ve talked about Face Detecting using Open CV with Haar Cascades. If you didn’t have the chance to go through it yet, you can read it by clicking the following link below.
By using that Model I’ve explained, Running the example on a static photograph, we can see that many of the faces were detected correctly, but the result is not perfect and perhaps better results can be achieved with further tuning, and perhaps post-processing of the bounding boxes.
Face Detection With Deep Learning
Face Detection With Deep Learning
There are myriad of methods demonstrated for face detection and out of all methods, the “Multi-Task Cascaded Convolutional Neural Network” or MTCNN for short, described by Kaipeng Zhang, et al. in the 2016 paper titled “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks.”
The network uses a cascade structure with three networks; first the image is rescaled to a range of different sizes (called an image pyramid), then the first model (Proposal Network or P-Net) proposes candidate facial regions, the second model (Refine Network or R-Net) filters the bounding boxes, and the third model (Output Network or O-Net) proposes facial landmarks
According to their own words:
“ In this paper, we propose a new framework to integrate these two tasks using unified cascaded CNNs by multi-task learning. The proposed CNNs consist of three stages. In the first stage, it produces candidate windows quickly through a shallow CNN. Then, it refines the windows to reject a large number of non-faces windows through a more complex CNN. Finally, it uses a more powerful CNN to refine the result and output facial landmarks positions. “
The image below taken from the paper provides a helpful summary of the three stages from top-to-bottom and the output of each stage left-to-right.
The model is called a multi-task network because each of the three models in the cascade (P-Net, R-Net and O-Net) are trained on three tasks,
e.g. make three types of predictions; they are: face classification, bounding box regression, and facial landmark localization.
Three models are not directly connected and act as a lap of a sprint race. When one round finishes, the next one starts and so on until the 3rd lap finished.
This allows additional processing to be performed between stages, for example, non-maximum suppression (NMS) is used to filter the candidate bounding boxes proposed by the first-stage P-Net prior to providing them to the second stage R-Net model.
MTCNN architecture is reasonably complex to implement. Thankfully, there are open source implementations of the architecture that can be trained on new datasets, as well as pre-trained models that can be used directly for face detection. Of note is the official releasewith the code and models used in the paper, with the implementation provided in the Caffe deep learning framework.
Perhaps the best-of-breed third-party Python-based MTCNN project is called “MTCNN” by Iván de Paz Centeno, or ipazc, made available under a permissive MIT open source license.
The MTCNN project, which we will refer to as ipazc/MTCNN to differentiate it from the name of the network, provides an implementation of the MTCNN architecture using TensorFlow and OpenCV. There are two main benefits to this project;
- It provides a top-performing pre-trained model.
- It can be installed as a library ready for use in your own code.
Part 1: Install MTCNN
Part 1: Install MTCNN
pip install mtcnnWait till all the components get installed.
By default, the library will use the pre-trained model, although you can specify your own model using the ‘weights_file‘ argument and specify a path or URL, for example:
model = MTCNN(weights_file='filename.npy')The minimum box size for detecting a face can be specified via the ‘min_face_size‘ argument, which defaults to 20 pixels. The constructor also provides a ‘scale_factor‘ argument to specify the scale factor for the input image, which defaults to 0.709.
Once the model is configured and loaded, it can be used directly to detect faces in photographs by calling the detect_faces() function.
This returns a list of dict object, each providing a number of keys for the details of each face detected, including:
- ‘box‘: Providing the x, y of the bottom left of the bounding box, as well as the width and height of the box.
- ‘confidence‘: The probability confidence of the prediction.
- ‘keypoints‘: Providing a dict with dots for the ‘left_eye‘, ‘right_eye‘, ‘nose‘, ‘mouth_left‘, and ‘mouth_right‘.
Part 2: Python Script
Part 2: Python Script
Step01: Detecting Faces
Step01: Detecting Faces
* Let’s run a face detection on this following College Students Image
And this group photo:
Baby steps; Let’s try the first image since it has only four people.
#face detection with mtcnn on a photographfrom matplotlib import pyplotfrom mtcnn.mtcnn import MTCNN#load image from filefilename = 'fl.jpg'pixels = pyplot.imread(filename)#create the detector, using default weightsdetector = MTCNN()#detect faces in the imagefaces = detector.detect_faces(pixels)for face in faces: print(face)By running this script; Our college student’s image loads, and then loads the model, perform face detection and prints a list of each face detected.
Each box lists the x and y coordinates for the bottom-left-hand-corner of the bounding box, as well as the width and the height. And as I’ve mentioned above Keypoints are generated such ‘left_eye‘, ‘right_eye‘, ‘nose‘, ‘mouth_left‘, and ‘mouth_right‘. The results suggest that four (4)bounding boxes were detected. Which means there are four faces.
Step02: Draw Squares on detected Faces
Step02: Draw Squares on detected Faces
#get coordinatesx, y, width, height = result['box']#create the shaperect = Rectangle((x, y), width, height, fill=False, color='red')Step03: Draw circles on Keypoints
#draw the dotsfor key, value in result['keypoints'].items(): #create and draw dot dot = Circle(value, radius=2, color='red') ax.add_patch(dot)Below is a function named draw_image_with_boxes() that shows the image and then draws a red square for each bounding box detected.
def draw_image_with_boxes(filename, result_list): #load the image data = pyplot.imread(filename) #plot the image pyplot.imshow(data) #get the context for drawing boxes ax = pyplot.gca() #plot each box for result in result_list: #get coordinates x, y, width, height = result['box'] #create the shape rect = Rectangle((x, y), width, height, fill=False, color='red') #draw the box ax.add_patch(rect) #show the plot pyplot.show()The complete example making use of this function is listed below.
Complete code as a Gist
Running the example plots the photograph then draws a bounding box and facial key points for each of the detected faces.
Step03: Run and Detect the box Faces
Step03: Run and Detect the box Faces
We can see that all the faces (4) were detected correctly and all the keypoints are fairly accurate.
I’m impressed :)
Now, Let’s try the complex image with dozen of people in it.
Step04: Extract Each face separately
Step04: Extract Each face separately
We may want to extract the detected faces and pass them as input to another system.
This can be achieved by extracting the pixel data directly out of the photograph; for example:
# get coordinatesx1, y1, width, height = result['box']x2, y2 = x1 + width, y1 + height# extract faceface = data[y1:y2, x1:x2]We can demonstrate this by extracting each face and plotting them as separate subplots. You could just as easily save them to file. The draw_faces() below extracts and plots each detected face in a photograph.
# draw each face separatelydef draw_faces(filename, result_list): # load the image data = pyplot.imread(filename) # plot each face as a subplot for i in range(len(result_list)): # get coordinates x1, y1, width, height = result_list[i]['box'] x2, y2 = x1 + width, y1 + height # define subplot pyplot.subplot(1, len(result_list), i+1) pyplot.axis('off') # plot face pyplot.imshow(data[y1:y2, x1:x2]) # show the plot pyplot.show()The complete example demonstrating this function for our second group photo is listed below.
Running the example creates a plot that shows each separate face detected in the photograph of the second group photo.
This was good, but facial detection is a big useless if we could only pass in images. I wanted to modify it so I could use it with my webcam. In doing so, I can also test the paper’s claim that it would be able to locate faces in real time.
Once again, I imported OpenCV and MTCNN, then created a detector:
import cv2from mtcnn.mtcnn import MTCNNdetector = MTCNN()To use my webcam, I created a VideoCapture object. Since I only have 1 camera, I passed in 0.
cap.read() returns a boolean (True/False) that states whether or not a frame is read in correctly. If an error occurs and a frame is not read it, it will return False and the while loop will be broken.
Then, similar to example.py, I called detect_faces on each frame. Since sometimes a face may not be in the frame (and result will be empty), I added “if result != []” for the program to continue running even when there are no faces in the frame.
In addition, there may be more than one face in the frame. In that case, result will return back multiple sets of coordinates, one for each face. Hence, I ran a for loop in result to iterate through every individual face. For each face, I drew out the bounding box frame and dotted the 5 facial landmarks.
Finally, I displayed each individual frame. Each frame will wait for 1 millisecond for the key ‘q’, then move on to the next frame. To close the window, all I have to do is press ‘q’. Once I do that, I release the video capture and close the window.
cap = cv2.VideoCapture(0)while True: #Capture frame-by-frame __, frame = cap.read() #Use MTCNN to detect faces result = detector.detect_faces(frame) if result != []: for person in result: bounding_box = person['box'] keypoints = person['keypoints'] cv2.rectangle(frame, (bounding_box[0], bounding_box[1]), (bounding_box[0]+bounding_box[2], bounding_box[1] + bounding_box[3]), (0,155,255), 2) cv2.circle(frame,(keypoints['left_eye']), 2, (0,155,255), 2) cv2.circle(frame,(keypoints['right_eye']), 2, (0,155,255), 2) cv2.circle(frame,(keypoints['nose']), 2, (0,155,255), 2) cv2.circle(frame,(keypoints['mouth_left']), 2, (0,155,255), 2) cv2.circle(frame,(keypoints['mouth_right']), 2, (0,155,255), 2) #display resulting frame cv2.imshow('frame',frame) if cv2.waitKey(1) &0xFF == ord('q'): break#When everything's done, release capturecap.release()cv2.destroyAllWindows()Running this new file, I see that the MTCNN network can indeed run in real time. boxing my face and marking out my features. If I move my face out of the frame, the webcam continues running as well.
Click here to read about the structure of the MTCNN model!
Click here to read about how MTCNN does face detection!
Download the MTCNN paper and resources here:
- Github download:https://github.com/vaibhavhariaramani/mtcnn
- Research article: http://arxiv.org/abs/1604.02878
You can access the full source code from here. It has 4 separate Python Scripts and I’m sure you can understand what each means by reading the name of the script file.
All this knowledge is Thanks to https://www.linkedin.com/in/jasonbrownlee/ (Jason Brownlee) and Connect me on LinkedIn https://www.linkedin.com/in/vaibhav-hariramani-087488186/
Thanks!
Google Sites
Report abuse