ASL Translator Project
The overall goal is create a program that uses a generic web camera to process American Sign Language (ASL), then translates the processed information into American English. I plan to use Deep-Learning Algorithms to process hand gestures, this will include both dynamic and static signs.
Overview
Goals
[Main Goal] Translate the ASL alphabet into text
Allow for an easily editable dictionary for adding signs to the program
Translate words in ASL into text
[Reach Goal] Translate American English into ASL, ASL displayed with images on Monitor
[Reach Goal] Allow for text to speech for American English output and speech recognition for American English input
Possible Specifications
Python w/ OpenCV, MediaPipe, Tensorflow, and Numpy
Windows 10
Github
Website
Limitations
Will process information using deep learning algorithms
Milestones and Timeline
Gather more information on ASL, Python Webcam libraries, and methods of processing images /video with code. Do additional research on Deep Machine Learning if time permits.
Create use cases, sequence diagrams, and state machine diagrams for the program. Start to record progress on website (www.agnorokdomain.com) for documentation of the implementation phases. Create basic UI design for simple application.
Set up the basis of the program, make it process hands and ignore any unrelated stimuli. Make sure the program clearly understands palm and finger positions. Add to the base program, process hand motion (, may implement Machine Learning here if time permits). Make sure the program can differentiate a variety of hand motions.
Create a dictionary system for identifying processed ASL Signs, then retrieve the equivalent string. Make the system for adding signs simple and orderly.
Based on the design, create a basic application for running the program. The output should be displayed via the app or should be typed by the program where the text cursor is located. There will be a toggle for disabling and enabling the application from taking input.
Add reach goal of translating American English into ASL and displaying ASL with images outputted to the Monitor. Add reach goal of allowing text to speech for American English output and speech recognition for American English input.
Test all successfully implemented functions together. Cycle with Debugging for the final product.
*each Implementation phases will have their own Testing and Debugging done
Research Phase
Includes any resources relevant to the project, will add to the raw links whenever I find more useful information relating to the project. Also Use Cases are
Research Links
Real Time Sign Language Detection with Tensorflow Object Detection and Python | Deep Learning SSD
Google Translate, but for Sign Language Tutorial | OpenCV Python | Computer Vision (2021)
Tensorflow Object Detection in 5 Hours with Python | Full Course with 3 Projects
Real-time Hand Gesture Recognition using TensorFlow & OpenCV (Techvidvan Website)
Real-time Hand Gesture Recognition using TensorFlow & OpenCV (GitHub Project)
Cornell University Paper: Real Time Hand Gesture Recognition
General Movement recognition frameworks (Face, Body, Hands, Etc)
Use Cases
Webcam detected and Default Configuration file found - Desired Series of Events
Pre-condition: Webcam is connected and valid Default Configuration file present
Post-condition: The Main Graphical User Interface is started and displayed
Story
User starts application
System checks for a webcam
Webcam is detected
System checks for the Default Configuration file
Default Configuration file is found
System attempts to load the settings given by the Default Configuration file
Settings loaded successfully
The Graphical User Interface is started and displayed
End of Story
Next use case(s) - Main Application
Webcam detected and Default Configuration isn’t file found
Pre-condition: Webcam is connected and no Default Configuration file present
Post-condition: Error message is displayed “Unable to find the Default Configuration file”
Story
User starts application
System checks for a webcam
Webcam is detected
System checks for the Default Configuration file
Default Configuration file isn’t found
Error Message is displayed, user may restart or close the application
End of Story
Next use case(s) - Startup and Initialization
Webcam detected and corrupted Default Configuration file found
Pre-condition: Webcam is connected and corrupted Default Configuration file present
Post-condition: Error message is displayed “Unable to read the Default Configuration file”
Story
User starts application
System checks for a webcam
Webcam is detected
System checks for the Default Configuration file
Default Configuration file is found
System attempts loads the settings given by the Default Configuration file
System can’t parse the file as expected
Error Message is displayed, user may restart or close the application
End of Story
Next use case(s) - Startup and Initialization
Note: the status of the Def Config file is irrelevant, application won’t start without a valid webcam / may change how this is handled later
Webcam not detected
Pre-condition: Webcam isn’t connected or is malfunctioning
Post-condition: Error message is displayed “No webcam detected, abort?”
Story
User starts application
System checks for a webcam
Webcam isn’t detected
Error Message is displayed, user may restart or close the application
End of Story
Next use case(s) - Startup and Initialization
Note: These cases are subject to change / this case runs in parallel with GUI
Webcam is detected and the System processes an American Sign Language (ASL) Sign - Desired Series of Events
Pre-condition: Webcam is detected and Main app is loaded
Post-condition: System detects a recorded ASL word
Story
System checks Live Configuration, is “Disable Hand Tracking” On?
It isn’t
System checks for a webcam
Webcam is detected
System attempts to compare current sign with signs in the ASL with Deep Learning (based on past input)
Matching sign found in Dictionary
System handles results, updates Main Application GUI
System repeats process
End of Story
Next use case(s) - Main Application, Image Processing
Processed sign not in System ASL Dictionary - Common series of events
Pre-condition: Webcam is detected and Main app is loaded
Post-condition: System doesn’t detect a recorded ASL word
Story
System checks Live Configuration, is “Disable Hand Tracking” On?
It isn’t
System checks for a webcam
Webcam is detected
System attempts to compare current sign with signs in the ASL with Deep Learning (based on past input)
No matching sign found in Dictionary
System repeats process
End of Story
Next use case(s) - Main Application, Image Processing
Live Configuration“Disable Hand Tracking” is enabled
Pre-condition: Webcam is detected and Main app is loaded
Post-condition: Main Application continues
Story
System checks Live Configuration, is “Disable Hand Tracking” On?
It is
System repeats process
End of Story
Next use case(s) - Main Application
Webcam is not detected
Pre-condition: Webcam isn’t detected and Main app is loaded
Post-condition: Error message is displayed “No webcam detected, abort?”
Story
System checks Live Configuration, is “Disable Hand Tracking” On?
It isn’t
System checks for a webcam
Webcam is not detected
Error Message is displayed, user may restart or close the application
End of Story
Next use case(s) - Main Application
Design Phase
Includes diagrams further expanding the application's structure and fleshing out some of the ambiguous ideas proposed
Implementation Phase
Includes implementation status and progress for when I worked on the project. (More will be shown towards the end of the timeline)
Collecting data to further improve recognition
Currently implementing the following
Letters A-Z
Numbers 0-9
"Space" sign
Current GUI
Coming Soon!
Final Testing and Debugging Phase
Just as described, my process for testing and debugging both the project as a whole and individual pieces
