Research Interests:

Camera Captured Document Boundary Detection
Dewarping of Camera Captured Documents
Camera Captured Document surface modeling
Handwriting Quality Assessment
Automated Answer Script Evaluation
Speech to Equation Conversion
Error Analysis of OCR and ASR outputs
Text Accuracy Enhancements using Language models for various Indian Languages
Automatic Reference identification and linkage creation in documents
Heritage Document (palm leaf, stone tablets etc.) Processing
Text Summarizations
Old manuscript conversion to electronic formats in multiple languages

Internship & Student Projects Available

Boundary Detection for Smart Phone Camera Captured Document Images using Deep Learning Techniques
Text accuracy enhancement with Deep Learning based language models (English and Indian Languages)
Hand Writing quality assessment from camera captured handwritten images
Mathematical modeling of camera captured page surfaces for de-warping
Automated evaluation of descriptive answers using deep learning models
Reference Entity identification extraction from documents
Mathematical modeling of textual error patterns
Punch hole removal from palm leaf documents

Automated Answer Script Evaluation

Student Answer scripts in many forms. Technology exists to high level of maturity for assessment of objective type answers (one word, MCQ's etc.). However, for descriptive answers, research is still at early stage.

Our research work deals with many facets for descriptive answer evaluation and assessment. The type of answer evaluation we are, currently , looking at are:

Natural Language: short descriptive answers; typically, 2 - 3 sentences
Formal Language: Programming language answers (C, Python etc.)
Constructive formal language: High School Algebra Answers

As part of this project, we are trying to understand the vector space representation of the descriptive answers for natural language as well as formal language text answers. For natural language text answers, we are focusing on Indian Languages and scripts. For formal language text, we are focusing on Mathematics (Algebra) and Programing languages (C, Java, Python etc.). In addition to scoring, we study the mechanism for feedback generation.

Various cutting edge works in these areas have led to multiple publications. These works have led to more unanswered research questions (RQ) than answered RQ's.

We are still pondering over questions such as:

What constitutes an assessment? what components are involved?
What should be the baseline for assessment and outcome for it?
How does the assessment framework fit with the DIKW framework?
How to ensure standardization into the assessment scenario?

While these questions are related to the field of education and pedagogy, a deeper understanding them helps with developing proper solutions for automation.

Legal Informatics

Our research deals with helping the people in marginalized societies gain better access to legal services. We use different language models to:

Predict outcomes for a case
Generate Summaries for a case in English and Indian languages
Perform sentiment analysis of court case judgements

Handwriting Quality Assessment

The project deals with automatic assessment of handwriting using structural features such as size, distance between characters, and shape attributes. The attributes are extracted from the binarized version of input images. Upon classification, the algorithm suggests a handwriting to be either good or bad. We have been able to attain 98+% accuracy on scanned handwritten line and paragraph images.

Above is an example of input image with good handwriting.

This work is done by an enthusiastic & awesome group of B Tech students (Ms. Chandana, Mr. Rithish, Mr. Sriharsh & Mr. Subash). I acknowledge their contribution to this project and thank them for all the hard work they have put into this.

Out of this work, our paper "Handwriting Quality Assessment using Structural Features and Support Vector Machines" has been selected for presentation at 7th I2CT, Pune, in Apr-2022.

Top

Speech to Equation Converter

The project deals converting a spoken equation into its graphical format. The speech is collected from a user's device (Ex: Android phone) recognized through Google's speech recognition software, corrections applied, content parsed and finally the equation is generated as a displayable content. This work is done by Mr. Shreyas Vinaya, a B Tech Student from BITS Pilani Goa Campus. Our paper, "Speech to Equation Conversion using a PoE Tagger", has been selected for presentation at 7th I2CT, Pune, Apr'2022.

Home

Text Accuracy Enhancement with Language Models

Various kinds of errors are introduced into the recognized text obtained through Optical Character Recognition (OCR) and Automated Speech Recognition (ASR) technologies. The patterns of errors vary from the errors observed in typing as well as between OCR and ASR. Humans, however, have a great capability to read and interpret these erroneous text documents and make sense out of them. This comes from extensive training of the human mind in the language and grammar. It's, therefore, possible to employ various models that capture the subtle aspects of linguistic representations and obtain accurate text from the input erroneous text.

Research works have been performed, across the world on various languages, to study the efficacy of such techniques. Interesting results are obtained and reported. Most popular techniques employed are various architectures of RNN, attention models, transformer architectures etc.

My research group performs studies in this problem area on OCR and ASR recognized texts in English & Indian Languages.

Camera Captured Document Boundary Detection

Detection of boundaries of camera captured documents is the first step in dewarping the document. Most of the available technologies either define the document boundary with 4 corner points or 8 points (4 corners and 4 mid-points on the four edges). The underlying assumption behind these constraints is a page boundary is defined either by 4 to straight lines. Such assumption gets violated when the surface of the paper is not flat while being captured by a camera.

When the paper takes a complex non-planar surface, the boundaries of the page may be formed by complex curvatures. Such curved boundaries may be defined as a combination of very large to countably infinite straight lines. Our work deals with modeling the page boundaries with flexible set of control points. The solution involves determining the count and position of these control points so that the boundary may be well modelled leading to higher accuracy of reconstructive correction of the page.