Milestone 4
4.1 Optimization
Utilization of Machine Learning Models:
When designing our minimum viable product, we considered the application of TensorFlow.js, a library for machine learning in JavaScript to be directly used in a browser or within Node.js. Since our prediction and translation models were already considered to be trained using the Python version of TensorFlow, this was a great way to utilize these models natively without any intermediate. However, when attempting to port over the model into our Electron and React framework, the model could not be stored locally and the output data from the JavaScript implementation of MediaPipe could not be manipulated to send to the prediction model.
This problem led to an unlikely optimization in streamlining our codebase, as we then decided to have a separate frontend and backend linked together with a Python-based web socket (using the FastAPI library). This would allow asynchronous communication between the JavaScript frontend and the Python backend. Most importantly, this change allows the machine learning models to be utilized in the native language the models were developed in and be able to access important helper libraries like NumPy and the full capabilities of Keras.
4.2 Delivery
Our Project Website: https://www.sign.ml/
Our Final Project Github: https://github.com/BriannaPGarland/ASL-Transcription
The team has successfully managed and adhered to our project plan through weekly in-person meetings, virtual communication, and rigorous development/training on an individual basis. We had to adapt our plan when we discovered a technical problem, that our Python/Tensorflow.js-based backend was not compatible for integration with our React-based frontend, but we adapted our design such that the backend and frontend are linked through a Python-based web socket and data transfer can occur. While such issues affected our plan, our understanding of how to adapt to and work around these technical problems improved. The remaining task breakdown consists of completing the training of our ML algorithms to expand the library of words SIGN can translate and fully integrating the backend, both of which will be accomplished within the next week, and we are on schedule to do so. Once this is accomplished, we will have achieved an on time completion of our senior design project.