Face Recognition

Note from the Developers

The face recognition project is aimed at students who have an interest in neural networks but have not had a chance to implement them for a specific use case. Students will implement a face recognition IoT module and gain experience working with Python, Keras, and OpenCV to further solidify their machine learning background. Although the project focuses on an implementation of a popular CNN (Convolutional Neural Network), the intuition behind many other networks stems from the same ideas. We hope that you will find the information following of use in your future career.

Sincerely,

FR Team

System Overview

There are three major phases to this project. The first phase is intended to provide enough background experience in working with Python and various modules commonly used for machine learning tasks. The second then take this knowledge and builds upon it by incorporating the usage of cloud computing for the purpose of setting up the face recognition system. Finally, the last phase takes all of the ideas from before and tasks students with building a face recognition from the ground-up that allows for remote queries.

More specifically, in the first phase, you will focus on learning the core concepts of machine learning and why neural networks work. You will then learn about convolutional neural networks and why they are functionally different from a vanilla neural network. After this, you will gain experience in Python and a couple libraries by writing a couple scripts that incorporate PiCamera usage. This will form the baseline knowledge expected for the other projects. While the Python introduction is not comprehensive, you are strongly encouraged to practice writing scripts on your own time (or the later module might become a bit difficult).

The second and third phase are heavily related to each other so they will be explained together. In the second phase, you will be collecting images of yourself based off of code written earlier in order to incorporate into a dataset for usage in the CNN. You will then split this dataset using a script you design in order to generate three different subsets of the dataset. Finally, you will create a simple CNN using the Keras module in hopes that doing so allows you to understand the programming paradigm associated with the Keras module. The third phase extends on this by tasking you with implementing a larger, much more popular network, VGG16, for use with your created subsets. You will "train" this network and then incorporate it into a system that allows for remote face querying. While the remote portion is largely provided, you are encouraged to look over the code as the Flask module used is simple and intuitive to understand.

If you are planning to use Azure instead of AWS for whatever reason, be sure to use the Azure environment setup instead of the following default AWS setup. There are quite a few differences, but overall the setup is equivalent to a more verbose AWS instance setup. Be sure to take care when choosing an OS for the virtual environment in Azure as they have more control over the system preferences than an AMI.

Local Setup

In this part, we seek to setup both the RPi and potentially the student's personal computer in order to create the environment necessary for the project to succeed. In an effort to make sure that this page does not become cluttered with installation information, take a look here for what is required for the project on the Raspberry Pi and a method of installation that works on Linux and Windows (although local setup on a student's PC is not strictly required).

Note that installation on a Raspberry Pi (RPi) can take some time as the software needed can be a bit large and data transfer rates on a micro SD card are quite limited. This means that you should feel free to progress in the project while setting up the RPi. This will ensure that the least amount of time is wasted while waiting for software to be downloaded and copied.

As a final step, make sure that you duplicate the repository using your own Github account either by manually cloning and adjusting the remote or by using the following Github page in order to import the Face Recognition repository. Simply paste the original repository link into the URL and name the project whatever you like. Making a private repository would be preferred in this scenario as this project is meant to be a starting point only.

Introduction to Python

Please read all of the required material from the Intro to Python sub-page here and then come back to complete the following assignment.

Hello, Terminal!

As proof that you have finished this section, use Python to implement a script that prints out the current time for the next ten seconds every second and then terminates with a "Goodbye, World". Additionally, this program should only run when called through the terminal. That is, if you were to open up the Python interpreter and import your script, you should receive no output.

For this exercise, I suggest looking at the datetime module. You are free to print the time any way you want (including Unix time if desired). As for the extra constraint of running only in the terminal, I suggest looking at this Stack Overflow page on the particular construct to use. Feel free to ask any questions regarding either the API or the construct.

File Relocation

One important part of this project requires the shuffling of files into random directories. The purpose of this is to ensure that our network has a training, validation, and testing set that all cover sufficient variability to properly describe performance on a potential unseen dataset. For this part, write a Python script that takes the original images found in the /images directory and randomly shuffles them into a training, validation, and testing dataset with corresponding percentages. Note that the input dataset is already organized by classes. Since all the new datasets will still be organized in the same way, the new script should simply be copying old files to the three new datasets generated under the /Data/Train, /Data/Validation, and /Data/Test directories. For convenience, name this script 'dataSplitter.py' and place it wherever you like (notice that the base is in the /Project1 directory). The output file structure required can be visualized in the image that follows.

Image Capturing and Modification via OpenCV

In this section, we explore the OpenCV module and use it to manipulate an image and interface with the camera to localize faces. For those curious, we will be using a Haar cascade composed of pre-trained weak classifiers generated from Haar features. Note that although the purpose of this project is to recognize faces, we do not ask the network to localize them first. Due to this inability to localize faces, we use the Haar cascade as a sort of region proposal component for the network.

OpenCV Basics

OpenCV is an open-source computer vision Python module. It contains some very basic and not so basic algorithms for image manipulation, object localization/detection, and optimization. It also happens to also include support for camera usage; however, since the RPi camera functions slightly differently from normal cameras, it is not supported by the OpenCV libraries natively. In order to get around this, we will be using another module that works very similarly to the native library.

We seek to first get accustomed to the API and the nature of the returns by first working on some basic image manipulation tasks. For the first assignment, use the template of Geisel Library in the project repository as a starting point. First, display the image of Geisel and reshape it so that it is 50% of it's original size. Next, draw a bounding box over some part of the building (it does not need to be precise). Finally, save the output image as "geiselBox.jpg". You do not need to put too much effort into placing the box over the entirety of Geisel or whatever part you would like boxed. The purpose of this section is simply to introduce you to some basic functions and how positioning works in OpenCV. Make sure your output looks something similar to what follows:

The starter code can be found in /project1/process_geisel.py. The instructions contained within the file are the same as those listed here. It is placed there mostly for your convenience. (That will be the case for most of these future starter files.)

Real-Time Face Detection

With a very minimal introduction to the OpenCV API and knowledge of how to process a RPi camera stream, the next task is to now localize faces within an image. To do this, we will be following this guide to understand how the Haar Cascade classifier is initialized and used in order to detect faces. Feel free to read up on the theory of how the algorithm works, but it is not required that you understand how the weights were generated. It will be used as a starting point for the discussion about neural networks in a future section but covered in depth. For now, focus on the implementation of the face detection algorithm.

For this next assignment, use the camera stream to detect any faces and, if possible, draw a rectangle over the entire proposed face region. In general, the process for determining whether a face exists would be as follows:

1. Set up the classifier and load the weights for the face classifier

2. While the camera feed generates images:

   1. Run the cascade classifier on the image to generate proposed face regions.

   2. For every proposed face region:

      1. Draw a rectangle in the part of the image that corresponds to the face.

   3. Display the image.

Please modify your process_realtime.py file in order to follow this algorithm (call the new file face_detector.py). At any point if a face is upright in a frame, you should ideally see a rectangle around the face. If you still haven't been checked off for a previous part of the assignment, then feel free to make a copy and modify the copy instead.

Face Collection

With the above completed, it is easy to extend the system to now save pictures of your face. Recall that the purpose of this project is to create a remotely query-able recognition system for people who have been seen before. That is, we are guaranteeing that the person seen has been seen before and is thus present in the training set for the network (the problem of identifying new people as they come in is a completely different problem in machine learning!).

To be able to identify ourselves then, requires that we have a massive dataset of our own faces to train the network with. For this section, make a copy of the face_detector.py file and modify the code in order to save the cropped grayscale images of your face in different poses. This will naturally require the use of a minor delay in order to ensure the images are different enough to capture a sufficient degree of intraclass variation for you and your partner's face. [On the topic of sufficient, about 100-200 images is usually enough to capture this variability with a moderate snapshot delay.] Furthermore, since the faces in the original dataset range from 200x200 and larger squares, you are also expected to have images at least this large for your entire dataset.

Finally, one of the most important things to recognize is that the network itself was trained on square images. Since the warping of training images may actually harm the generalization procedure, it is required that students modify the aspect ratio of the saved image to be 1:1 (that is, your image must be a square). You may use any method you like in order to enforce this requirement, but remember that the face should cover a significant portion of the image in order to be better interpreted by the neural network due to the nature of the provided dataset. An example of a properly formatted image given an input frame (in this case, the Lenna image) can be seen below [notice that the image is extracted with equivalent height and width for the region of interest]:

Dataset Generation

In the following section, we will cover the generation of the dataset for use in the face recognition project. In order to ease the process of data collection (and prevent some nasty bugs from showing up as a result of a ill-formed dataset), the repository provides a starter dataset to which students can append their personal images. Note that there will be some intuition given for dataset generation, and students will be expected to look at the possible datasets and be able to explain why the dataset given works for this project.

Relation to Machine Learning Models

Note that the machine learning methods covered in this project (Haar cascade classifier / neural networks) are supervised machine learning techniques. Supervised learning techniques are models that use both the input and measured output in order to train a model. To be more precise, these models are discriminative learning techniques since they try to model the probability of the input directly from the input. In probabilistic terms, this amounts to estimating the conditional distribution of the outputs with respect to the input, P(Y|X;Θ), through some unspecified learned feature function, Φ(X,Y;Θ). It is for this reason that the dataset chosen is extremely important. Since this function's parameters, Θ, are estimated directly with respect to a given set of given inputs and outputs, the inputs and outputs must be representative enough to properly form a satisfactory estimate of the conditional distribution through Φ(∝ Θ). Sadly, in most practical applications, the outputs are often acquired through measurements that incorporate noise. Furthermore, we often have no idea of the form of Φ, so creating a particular function type along with an unknown parameter set is even more difficult. To narrow things down, we will be considering how datasets affect the multi-class classification problem directly. All this means is that we enforce that the cardinality of the set of possible classes Y be greater than one, or |Y|>1, and that Y be a set of discrete labels each representing a mutually exclusive class (that is, there can be no overlap between classes).

So, with the above, we have motivated the discussion of the choice of datasets for our supervised classification task. But what are some things that are intrinsic to the dataset that are required in order to successfully train a multi-class classification model? There should be three things considered for most datasets assuming the data collected cannot be completely described by the measurements (or images, in this case):

• Intra-Class Variation

• Inter-Class Variation

• Dataset Biases

In order to get a better understanding of the larger picture, each of these terms will be discussed along with their implications. In order to have a sufficient amount of intra-class variability, the classes themselves must have a sufficient amount of variation to describe the classes individually. For example, in our problem of classifying faces, intra-class variation would be represented through the varying light exposure, face position, face pose, negative space details, and so on. Essentially, anything that makes one picture of your face different from another picture of your face is an attribute that would contribute to intra-class variation. Without a sufficient level of intra-class variability, something known as the covariate shift (the testing dataset distribution is not the same as the testing or real-world distribution) could occur when evaluating, and the testing performance of the model will be low. Consider an example of this in Fig. 4 for a single class in a 2D space, where the given ellipse represents the true domain space:

In order to have a sufficient amount of inter-class variability, the classes should have a sufficient amount of variation to describe each class with respect to one another. In other words, the features that would help a typical person distinguish one face from another would contribute to this inter-class variation. For our particular task, this should ideally only be dependent on facial structure. With an insufficient amount of inter-class variability, it is easy to see that classes will become indistinguishable from each other and the model with perform very poorly. Note that in practice this often overlaps with some attributes that would normally contribute to inter-class variability since our datasets are not infinitely large. Consider an example of this in the below Fig. 5 for a two-class classification problem using the class above with varying levels of inter-class variation.

Use the above information in order to build the model in Keras. Consult the Keras API documentation for any questions that you may have regarding functional arguments. If you have more general questions regarding the implementation, feel free to ask a TA for help.

Once you are done with your Keras implementation of LeNet, show a TA the successful compilation of the model and a model summary with dimensions matching output feature layer representations in the image above.

VGG16 - Theory

While LeNet is technically a deep neural network, it is nowhere near as deep as most networks that followed. For example, the ResNet paper presented a network with 1000+ layers to show the benefit of adding residual connections to a neural network. While it would be interesting to implement the ResNet1202 model, we will choose an alternative family of models that has become just as popular as smaller ResNet models for images: the VGG family of networks.

Much of the explanation for VGG16 can be found in the paper this paper by K. Simonyan and A. Zisserman. To save some time, some motivating factors of the paper will now follow. Before VGG, many award-winning models were experimenting with larger kernel sizes placed earlier in the network in an attempt to increase the initial receptive field of the network. The hope was that by including larger convolutions earlier, a better generalization for earlier layers would give later layers a better overall representation. The creators of the VGG models, however, decided to do the exact opposite. Instead of training a small network with larger kernel sizes, they chose to train a deep network with successive small convolutions in order to attain similarly sized output receptive fields at the output of each "block" of convolutions. With non-linear activation functions placed following each convolution, their intuition behind this choice was that a similarly sized 7x7 kernel could be estimated by a series of 3 3x3 convolutions and a 5x5 kernel could similarly be decomposed into a series of 2 3x3 convolutions. The results of doing so on the training of the network were immediately obvious. Where a 7x7 kernel would require 49 parameters to describe, a series of 3 3x3 kernels would only require 27 parameters to describe. Following this logic, the resultant equivalent of a larger kernel network could be represented using this convolution "blocks" with less than half of the required parameters. This would prove to be significant for networks that already required the training of millions of parameters! Furthermore, by forcing the network kernels to be decomposed into smaller kernels, the model architecture naturally functioned as a kernel weight regularization technique.

One final thing I want to add before finishing up the logic behind the model is present in model C in the paper. Note that the model actually uses 1x1 kernels for convolution at the end of a series of two 3x3 convolutions. For those wondering how exactly a 1x1 convolution would do anything, consider looking at the output dimensions and number of channels of the convolution. First of all, it should be clear that the only difference in dimensions comes from the number of channels between the input and output of the convolution layer. The number of channels can be set by the number of kernels desired, and, since the convolution uses only a 1x1 kernel, the output width and height remain the same regardless of the padding method used. If that is the case, then what exactly is the use case for the 1x1 kernel? Basically, the 1x1 kernel serves as a channel pooling layer and increases the non-linearity of the output since an activation function would naturally follow. The VGG paper only uses them for the latter as the number of input and output channels remains the same, but many newer networks take advantage of this depth-wise pooling to manage the number of channels within a block (think of the Inception networks).

We will be focusing in particular on the Keras implementation of VGG16 (Model D in the paper). For convenience, the table of models covered in the paper along with the model to be implemented can be seen below this section. Hopefully, by looking at this table of models, students should be able to explain why the network was blocked off in such a way and explain how this network is better or worse than other networks through a comparison of the difficulty in training and representational power.

VGG16 - Evaluation

With a trained VGG16 model at hand, we are now ready to evaluate the "real-world" performance of the model. In essence, this means attempting to classify never seen before images and assessing the overall accuracy of the model. Using the model saved from the previous sub-module and the training dataset, evaluate the performance of the model and show the results to a TA/Tutor. Comment on whether these results are expected or unexpected and why.

You may find the starter code for this section under /Project_3/test_vgg.py.

Flask and Web Server Implementation

This final section will cover the integration of a Flask web server into the entire classification process in order to enable off-site processing and user identification. Most of the Flask implementation will be given to the students; however, some cursory understanding of how the API works will be required in order to create a functioning client and server pair. As part of the original repository, a root page (index.html) is given that allows students to test the server python file separately from the client in order to allow for a more streamlined debugging process. In order to run the server remotely with support for webcam usage, make sure to run the server script in a SSL context by calling the file with the --ssl flag as follows:

python3 server.py --ssl

Server Implementation

The server is based off of a simple Flask python implementation. The actual server is started using a single line while the accessible routes are determined by the functions that are decorated by the app.route decorator. If you recall from the Python introduction section, this function is actually a decorator factory. It returns a decorator that is custom made to allow the function access to data received on a particular route as specified by the argument of app.route. Included in the given server starter code, /Project_3/, is actually an additional root page (index.html), which is automatically served when accessing the root page of the publicly viewable IPv4 address. That is, in order to access the website, it should be done in the following fashion:

http[s]://(Public IPv4 Address):8080/

Anything following the final "/" can be considered a unique link which can be served by the server.

The root page itself can be used as a testing site for the server file. When run using the SSL context as shown in the section above, the site can request webcam access on your laptop/desktop and submit images. Note that since the image sent by the root page will not be in the proper format, it will not provide a valid output from the CNN unless the preprocessing steps are moved from the client file to the server file (this is not required). For this section, complete the server.py file within the /Project_3/ directory with the requested changes and ensure that the root page receives a proper response when sending an image from the webcam.

Client Implementation

The client functions very similarly to the process followed in the initial data collection scheme, but with a couple modifications intended for making data sending easy. The following is the algorithm that should be followed by the client:

1. Set up the classifier and load the weights for the face cascade.

2. For every frame received by the PiCamera:

   1. Process all possible faces in the PiCamera.

   2. If a face is detected, then:

      1. Show the face detected and prompt the user for confirmation of the request.

      2. If user agrees, then for the most likely face:

         1. Send the cropped grayscale face to the server endpoint for classification.

         2. Request the server's prediction for the user's identity

         3. Display the image along with the name and probability value.

This should run indefinitely as the number of frames captured by the PiCamera would generate a constant stream. Furthermore, ensure that the image sent to the server is a square as any non-square images will be force resized by the time the server receives the image. Such warping is likely to cause an incorrect identification as the features the network was trained on will not be preserved. The starter code can be found in the client.py file within the /Project_3/ directory.

Module Testing [Required] & Extension [Optional]

With the server and client files both properly set up, it is now time to connect the system and have each file interface with each other. Make sure to run the client on the Raspberry Pi and the web client on a server with enough compute power to hold the VGG16 model and any temporary variables. Show the TA that your model properly outputs a proper guess given a camera capture of with your face in it.

The only issue with this algorithm is that the model only returns the output for a single person. Adjusting the model to work on multiple users is as simple as adjusting a single loop in the program. If you finished early then try modifying the client algorithm to function to work with multiple face inputs! [When working with multiple faces, it is advisable to remove confidence values and simply place the suggested users above the bounding boxes.]

Alternatively, recall earlier that the server.py file would benefit by having the preprocessing moved to it for the convenience of being able to also take images in the root server page and have proper identification results. Feel free to modify the client.py and server.py files so that the entire image is sent instead of only the face file. Note that there should still be some detected faces for the server to not be overloaded with requests!

Resources

• CSE 231n - Great source for pretty much the entire project sequence. Covers all of the theory necessary to build models and understand the process in modules 1 & 2 of the course notes page. Note that much of the introduction is spent on motivating the usage of neural networks over other classifiers (such as SVMs) which you may find useful if you have some basic prior ML knowledge.

• Deep Learning by Ian Goodfellow - Perhaps the most useful math-based introduction to neural networks by one of the men who came up with the concept. It also has a small section on state-of-the-art (at least during that time!) networks that are still architectures used in today's networks. If you don't mind the math, this is usually the best place to start understanding neural nets in depth.

• Flask API Tutorial - A decent introduction to using the Flask API to host web servers locally. It covers some basic examples along with the thought process behind implementing certain functions.