Assignment 4

Real Time training:

For the real time training model, I decided to make a model that detects whether I raised my hand or not with the Pose Classification with Posenet and ml5.neutralnetwork model. First I collected the data with this p5 model editor.p5js.org/Alicelong/sketches/NgPnVsMYq. I thought the keypress-input function didn't work really well for my case and I also wanted the label to be editable. User can type in the label they want. So I created an input box and two buttons. One for submit the input as the label and start collecting data. The other is for saving the data. Once the user types the label and hits submit, they can start doing the pose. After they finishing collect all the data, they can hit save and the JSON file would be downloaded. I collected two groups of data: one is labeled with "raising" with my arm raising; one is labeled with "not raising" with my arm laying down.

Then I imported the JSON file to the model training p5 editor.p5js.org/Alicelong/sketches/Zbg3_AHVj and start training the model. Luckily, the training was successfully done without any accidents.

When I imported the model to the actual deploy p5 editor.p5js.org/Alicelong/sketches/N8RVtJu1x, I found that the model couldn't really identify my gesture. Whenever I put my hand up or down, it classified it as "raise". To figure out what happened, I console.log the confidence score and found out that all the confidence were really high, around 0.97. Then I realized that maybe there are some issues with my data collection. I adjusted my standing position, distance to the screen, angel,etc and eventually I found that I had to face slightly toward 45 degrees to the screen to make the recognition work effectively. Perhaps that was caused by me rotating my body when collecting data and the model recorded my rotation.

To make it more fun, I am thinking about using the confidence score to create a regression output display. I can map the confidence score to volume so that the user can create beat by raising and lowering the arm. But I think the regression model in the Github, the slider can make this easier.