Projects

Pedestrian detection is an essential task in any video surveillance system. Given an image we would like to detect regions where pedestrians exist. This becomes a challenging task due to variations in human poses, appearance and clothing. Therefore, we need a robust feature set such that it can distinguish pedestrians independent of their background.

This project highlights the implementation of Pedestrian detection in two methods:

1. The first method extracts the features of an image using Histogram of Oriented Gradient (HOG) descriptor and then classifies them using linear Support Vector Machine (SVM).

2. To make the detection more invariant to multiple size bounding boxes of different scales, a Region Proposal based Convolutional Neural Network (R-CNN) was developed, this consisted of two steps, viz., selective search for generating region proposals and a ResNet based CNN architecture for feature extraction and classification.

Currently, existing machine learning algorithms are typically implemented on expensive, high powered Graphical Processing Units (GPU) which have high computational capabilities. Such implementations are generally not feasible for cost effective applications which have strict power and computational constraints. Digital Signal Processors (DSP) have high computation speed due to efficient implementation of mathematical operations and are energy efficient. The main objective of the project is to have an image of a hand written text and then digitize it by recognizing the characters written on it. Key contributions towards the project were :

• Training a 3 layer deep CNN digit classifier on MNIST data-set with 98% accuracy using pytorch framework.
• Extracting learned parameters i.e weights and biases from every convolution and fully connected layer, mean and variance from batch norm layers for inference on the DSP.
• Implementing the forward pass of the convolution neural network on the TMS320C6748 in C using Texas Instruments Code Composer Studio.
• Interfacing a video camera to the DSP to capture hand written digit.
• Implemented block average based image thresholding and re-scaling operation to match the testing image to samples in the train set.

Visual speech recognition (also known as lipreading) is a field that is becoming increasingly important. It is has emerged out as a natural complement to speech based recognition systems that can facilitate transcriptions even in noisy environments. Perfect Lip reading is still a challenge due to variations in lip articulations that exist while producing a particular utterance, also many phonetically similar words have similar lip movements, encouraging us to develop a system that harnesses clues from not only the target word but also from contextual words that the target word generally co-occur with. Key contributions towards the project were :

• Developing a novel 24 layer deep 3-D convolution neural network with hierarchical residual connections.
• Carefully, pooling across time and using LSTMs to extract temporal features.
• Visualizing intermediate convolutional layers to understand the various lip articulations at which model attends itself to.
• Attained a Word Error Rate (WER) rate of 20% on the test set of the BBC-Oxford 'Lip Reading in the Wild' data-set ( BBC LRW), samples of the data-set can be obtained from here.

Detection of text in images finds important application in content based search. It is also a key step in Optical Character Recognition. In this project, we describe a method to detect, localize and extract horizontally aligned text in images, that usually appear in the form of on-screen text and subtitles in TV advertisements, news channels and movies. Key contributions towards the project were :

• Generation of an edge image based on the fact that character contour pixels show high contrast compared to their local neighbors.
• Considering strongest edge component in horizontal, vertical and diagonal direction.
• Analyzing the horizontal projection profile of the edge image to locate potential text areas.
• Making use of Power law transform to avoid merging of adjacent characters with the background.
• Eliminating Non-text elements using connected component analysis.
• Visual results indicate good performance on images taken from screen texts, subtitles in TV advertisements, news channels and movies.

Ear bio-metrics are gaining importance in surveillance applications at a very rapid pace. Ear Images are now considered unique for person identification. Ear Images captured by surveillance cameras are typically poor in resolution. Wavelets are known to capture incremental information while going from one resolution to a higher resolution. In this project we aim to super resolve low quality ear images by using a deep convolution neural network to predict wavelet coefficients of high resolution image. Key contributions towards the project were :

• Interpreted Haar DWT & iDWT as Convolution & Deconvolution operations for faster on the fly pre-processing on GPUs
• Trained a deep CNN on the USTB: Hello Ear dataset consisting of 6,10,000 ear images to predict patch based wavelet coefficients of SR image from LR image
• This model was trained on Two Nvidia GTX 1080 (8GB) GPUs using Pytorch framework