Processes

This flowchart depicts the project architecture of the connections between all of our subsystems. Image © Team 6 original work.

MATLAB: Method for Image Processing and Filtering

1. First, we read in the image using imread and im2double. Then, we convert the image to grayscale using im2gray.

2. Next, we translate a flipped copy of the image to the frequency domain using fft2 (a 2-dimensional Discrete Fourier Transform) and reorganize the signal with fftshift. We used this new representation to pass the image through a high pass filter that effectively only keeps parts of the image with very quick changes in intensity, since those parts translate to the higher frequencies. The quick changes correspond to the edges of detected objects in the image. The image is then taken back out of the frequency domain using ifftshift and ifft2 .

3. We use Sobel Operators to convolve the filtered image with two kernels that perform horizontal and vertical edge detection. We then add the resulting matrices together from the horizontal and vertical convolutions. This step has the effect of leaving only the very edges, since the high pass filter will still leave a small gradient up to the edges of an object.

4. We convert the image to a binary representation with imbinarize to have access to MATLAB's binary image processing bw functions. First, we use imfill to enhance the definitions of all background pixels near detected edges from Step 3. Next, we use bwareopen to remove all connected edges that spans less than 30 pixels and effectively remove all tiny scratches. Lastly, we use bwpropfilt to remove all connected regions that span less than 200 pixels in area to further remove any scratches and imperfections on the writing surface.

5. This image is then passed through two more filters, mainly to get rid of any remaining large scratches. These filters sweep over the horizontal and vertical directions to remove any detected thin objects. The thin scratches and edges of the board will be removed, but the letters are relatively thick, so they remain unfiltered after this step.

MATLAB: Methods for Character Isolation

1. This step focuses on creating individually indexable boundaries around each distinct character. We used the function bwboundaries to outline enclosed areas in our binary image. Furthermore, we obtain the following outputs from this function:

• B: traces the individual boundaries.

• L: labels each boundary with their position to allow for indexing .

• n: the number of objects found.

• A: an adjacency matrix denoting the edge connections between boundary nodes.

2. In this step we make sure we index the characters in the correct order because bwboundaries does not index the characters from right to left and top to bottom -- in the order of English writing. To correct the indexing, first we iterate from left to right through all vertical lines of text from the binarized image representation until we find a single line with significant whitespace (denoting characters). Then we search every row from top to bottom around that vertical line to find the line that intersects with the most unique letters and take this as our first line of text. Lastly, we traverse the line and use various MATLAB commands to extract each letter individually from its index and feed to the classifier.

3. We use the function poly2mask to mask the detected characters over the original grayscale sample image. This has the effect of only extracting the detected letters from the image by setting all other regions to gray, and therefore fixes the problem with shading in encircled letter regions produced in the Image Processing and Filtering subsection.

It is worth acknowledging that MATLAB has a built in edge detection function called edge. Unfortunately, edge repeatedly missed letters during sample image testing, so we did not get acceptable performance with this function.

All Images below © Team 6 original work.

K- Nearest Neighbor, Multinomial Naïve Bayes, Support Vector Machine, Decision Tree:

We implemented these four classification methods using the Python's sklearn library to compare the classifiers' performances and opt to use the most efficient classifier or machine learning algorithm. Our Jupiter Notebook files for these methods are almost identical, apart from setting nb_classifier to the given method. In total, this is the breakdown of our process:

1. Importing the required libraries

2. Training the model (creating X_train and y_train) on the EMNIST training samples sub dataset

3. Correcting the X_train shape from EMNIST's 3D array into a 2D array in order to properly use the sklearn classifiers

4. Setting the nb_classifier to the given method

5. Training the classifier using sklearn's .fit() command with our X_train and Y_train data arrays

6. Testing the model (creating X_test and y_test) on the EMNIST testing samples sub dataset

7. Correcting the X_test shape from EMNIST's 3D array into a 2D array in order to properly use the sklearn classifiers

8. Predicting the y_values from the X_test data array using .predict()

9. Creating a confusion matrix with a heatmap to show the legend depicting the frequency of predicted values

10. Using .accuracy_score() to identify the percent accuracy prediction.

Convolutional Neural Network:

We implemented a convolutional neural network following the LeNet structure. Our process in Python can be broken down into creating the network architecture, training the network, and making predictions with the network.

1. Create a torchvision module class

2. Define each layer of the network and structure the forward propagation (connect layers)

3. Load various EMNIST datasets depending on samples being tested

4. Train model and validate along the way for five epochs by backward propagation of loss

5. Feed test data to network and compare predictions in a confusion matrix

MATLAB: Methods for LaTeX Document Formatting

We implemented the LaTeX Document Formatting subsection in a MATLAB script. The function takes in the user-defined lecture title and date metadata, then combines that information with the lecture content in the body of the document. The following steps describes our program:

projectPDF.m

1. Locate to the local directory storing the classified text from the Character Classification subsystem.

2. Initialize a LaTeX file in MATLAB called output.tex and open the file using fopen.

3. Use fprintf to write various formatting settings to the LaTex File including the user-defined title and date. Note: the LaTeX syntax requires two backslashes ('\\') in front of all outputted information.

4. Output each line of text from the Character Classifier file output on a separate newline in the LaTeX document. Multiple paragraph breaks are recorded and implemented using baselineskip.

5. Read the entire file and replace all underscores within quotation marks with '\_' to avoid the subscripts.

This program produces a .tex file that the user can easily convert to a PDF using free converters online such as Overleaf.