John Evans

Handwritten text RECOGNIzer

Image taken from ISTOCKIMAGE.COM

Introduction

The world has been digitalized to a great extent and yet there are few sensitive areas where we are forced to use traditional methods. Bank cheques are one of them. Most of the financial transactions are done by a paper cheque all over the world. When these handwritten bank cheques are in play there is also a scope for manual errors. These manual errors could be costly and could save a lot of money if this problem is addressed.

Project Objective

In my project, I would try to build an application which can scan the cheques, images, analyze the handwritten text. This way we can avoid the scope for manual errors to a significant extent. I am also extending my project to detect more handwritten texts to further use the algorithm to different application where we need a handwritten recognizer to identify the words.

Find my project in my GitHubRepository

About the Data

For the data, I had to rely on the handwritten text data and found the IAM Handwriting Database.

Briefly the dataset characteristics are as follows:

657 writers contributed samples of their handwriting
115'320 isolated and labeled words

A preliminary analysis of the dataset gave the following details:

The words have been extracted from pages of scanned text and are saved as png files
Each file has a word label given which would give us the coordinates of the box that it is chopped, tagset data (Parts of Speech tagset)
The word itself

Image taken from Visual Studio Magazine

Literature Survey

Automatic Cheque processing is an interesting upgrade in the banking industry. Nevertheless handwritten text on the cheque is a pit stop for any further automation. What if we detect the handwritten text and make the process easy? Not only we avoid the manual errors in this space but we also can make the work easier for the bankers. There are projects that have discussed the ways of identifying the MICR Text, Routing numbers and any printed formats but it gets messier when the machine tries to analyze from the handwritten text of the customer[1]. Using the deep learning techniques we can surely make this work. We need to train our model against the handwritten text and try to identify the words written on the cheque. Several techniques are used such as OCR(Optical Character Recognition) where it can detect the printed format from a scanned image[2]. Handwritten text could be detected by CNN (Convolution Neural Networks)[3]. In my project I will try to better the process using the modern day neural networks like RNNs and LSTM Networks and build a web application with an easy GUI to follow.

Phase 1 Presentation Link

Images Before Processing

Images After Processing

PROBLEMS TACKLED

Problem 1: Images being different sizes. Preprocessed the images to 128x128 size images. Below are the sample images after preprocessing.

Problem 2:

Tensors cannot accept the labels being in text
Tried changing the labels into ascii sequence (As word increases the length of the sequence increases by 3 times, so we get long type)
Since Tensors cannot handle long type, dropped that idea
Taken each character as a class and the label would be an array of integer values, where these values are character codes
Example: {1 : 'A', 2: 'B', 3: 'C',....etc.}
There are 80 classes in total

Phase 2 Presentation Link

SAMPLE PREPROCESSED IMAGES AND THEIR LABELS

Implementation Methodology

Clean and preprocess the words data
Link the word pictures with the labeling
Divide the data into train, test and validation data
Build a CNN model and train the pre-processed images
Picked the best model, test the model and predictions on validation data
Web Application is developed where the image is taken as an input from the user and the results are displayed on the webapp

Phase 3 Presentation Link

PROJECT LIFE CYCLE

APPLICATION OVERALL ARCHITECTURE

Webpage is a flask application where the input image is drawn in the canvas text box

We save the image, preprocess and send it as an input to the model which in turn returns an array of values which is decoded to the resultant text

Homepage

Image Drawn

Detected Text

FUTURE WORK

Beautify the application Interface
Extend the application to upload an image and detect the handwritten text
Make copies of transformed images and train model again

References

Zhang, X., Wang, M., Wang, L., Huo, Q., & Li, H. (2015). Building Handwriting Recognizers by Leveraging Skeletons of Both Offline and Online Samples. 2015 13th International Conference on Document Analysis and Recognition (ICDAR). doi:10.1109/icdar.2015.7333793
Krishnan, P., Dutta, K., & Jawahar, C. (2016). Deep Feature Embedding for Accurate Recognition and Retrieval of Handwritten Text. 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). doi:10.1109/icfhr.2016.0062
Jayadevan, R., Kolhe, S. R., Patil, P. M., & Pal, U. (2011). Automatic processing of handwritten bank cheque images: A survey. International Journal on Document Analysis and Recognition (IJDAR), 15(4), 267-296. doi:10.1007/s10032-011-0170-8

Page updated

Report abuse