Design
Methodology
This app utilizes three main aspects in order to run the app. The camera pointing at the piano, the Noteform app itself, and the Servers.
Camera
The first aspect of the NoteForm system is the camera. The user of the app plays the piano, with the phone in the background recording. The camera captures the user's hand movements and sends snapshots to the app.
NoteForm App
The next aspect of the app setup is the app itself. The app serves as a center for processing and displaying information. The camera sends the snapshot to the app, and the app replays the snapshot to the Firebase server for processing. After processing, the app displays the information such as the number of hands detected, the number of correct fingers, and the number of correct hands. The app keeps the data from each session and this data can be accessed later by users. This app is built on the Dart language using the Flutter framework.
Firebase Database
The another aspect of the app setup is the Firebase server. Firebase is a service offered by Google that allows for hosting and the processing of information. This server is used for authenticating logins and storing data. The emails and passwords of the users of the app are stored here. Once data is processed, the data is also stored here so users are able to reflect back on their past practicing sessions.
Pose Estimation Server
A Flask server is implemented to process the images sent by the Flutter app. To analyze the frames capture by the camera, pose estimation through MediaPipe is used. This is applied to the fingers on the snapshots of the fingers to calculate the correctness of each finger through calculating angles. The component relies on machine learning, and has been trained with sample images that are correct. After processing, the server sends back information to the app for display. This server is hosted on Render.