Embedded Files
Week of 9/14
- Goals: Preliminary research on Richard's tool ideas and related existing tools
- How I Did: No new ideas, but picked up some terminology and ideas that are important to film composition in general. Also found some cool existing tools we could possibly analyze for ideas.
- Results:
- Mezzo: Generates Neo-Romantic piano soundtracks for video games in real time
- Article giving high-level explanation of film spotting
- Film Composer John Paesano discussing common tools used for film composition
- Article about visual analysis (particularly colors)
- Stochastic melody generator
- Next Steps:
- Background reading/studying on CNNs
- UI ideas and/or sketches for QFVA tool
VIP Notes for 9-14.pdfWeek of 9/21
- Goals: Study CNNs and come up with ideas for a Visual Film Analysis tool
- How I Did: Learned a lot about CNNs from a really good video series I found. Even though I've worked with them before I feel like I understand them much better now. Also made a simple sketch of a UI for the main view of the VFA tool and took notes on some ideas for more detailed views.
- Results:
- I feel more comfortable with CNNs, but there were some things about machine learning in general I was still having trouble understanding, so I plan to continue with the video series I started.
- My UI sketch and ideas are shown in my notes to the right.
- Next Steps:
- Continue studying machine learning
- Do more background research on what exactly we would be analyzing using the VFA tool and effective ways to present the resulting data to composers in the tool
- Research ways to analyze the director aesthetics Richard mentioned
VIP Notes for 9-20.pdfWeek of 9/28
- Goals: Continue studying through machine learning video series and researching background info that can be used in development of VFA tool
- How I Did: Didn't do any research on VFA tool, but did make it through several machine learning videos. Also decided on a project for the next few weeks in our Friday meeting.
- Results:
- I have a better understanding of loss, overfitting/underfitting, batch size, and other concepts important to machine learning and neural networks.
- Next Steps:
- Continue machine learning fundamentals video series (1/3 of the way done!)
- Set up an Anaconda environment for Python 3.6 to use for VIP development
- Begin reading up on and working on reimplementing a preexisting CNN model for classifying faces/objects
VIP Notes for 9-28.pdfWeek of 10/5
- Goals: Get a popular CNN model for object recognition and classification called the YOLO model working on my computer.
- How I Did: Have it working for one image, but need to make it work with recorded videos and/or live video from the computer's webcam.
- Results:
- I have an idea of how the YOLO model works.
- I also watched more videos and took more notes about different types of machine learning.
- Next Steps:
- Finish machine learning video series (been watching it for a while, so I'd like to be done within two weeks).
- Make YOLO work on videos and spend more time understanding how it works.
VIP Notes for 10-5.pdfWeek of 10/12
- Goals: Get YOLO working with videos.
- How I Did: Have it working with webcam footage, but not yet with video files.
- Results:
- I made progress on my script for passing images and videos into YOLO. Even if we eventually end up using a different object recognition, my code should be useful in the final visual analysis application.
- I finished the machine learning video series I had been watching.
- Next Steps:
- Keep working on my YOLO script.
- Look into other object recognition methods.
- Consider combining my code with Jacob and Damon's code soon.
VIP Notes for 10-12.pdfWeek of 10/19
- Goals: Try to get YOLO working in real time on videos (i.e. play a video file and have YOLO show its object predictions in real time).
- How I Did: Didn't make much progress. I found a new library called ImageAI that I wanted to use, but it wasn't cooperating with some other libraries I was trying to use to display the video.
- Results:
- Decided to go with what I have for now and change focus for next week.
- Next Steps:
- Look for or develop an efficient tool for labeling image data for use in training object detection models.
- After doing this, start planning a way to include Shimon as an object category in a preexisting model.
- Experiment with running YOLO on images processed in different ways (will have to come up with processing ideas).
Week of 10/26
- Goals: Find an image labelling tool and make a Shimon dataset.
- How I Did: Found a nice labelling tool called LabelBox. I don't think it would be a good general purpose labelling tool for future VIP use, but it is the one I have chosen to use for my work this semester. I plan to look for one for future VIP use too though. I also did some different preprocessing on the Shimon data so that we can enhance the Shimon dataset and see how YOLO performs on images processed different ways in one stroke.
- Results:
- Now have a dataset of around 300 images of Shimon (~40 images processed in different ways; grayscale, brightened, darkened, horizontally flipped, vertically flipped, flipped on both axes, and Gaussian blurred).
- Next Steps:
- Decide how to train YOLO on my Shimon data (stick with just Shimon data for now or include other data classes too, should I try transfer learning on YOLO, etc.).
- Label the Shimon dataset using LabelBox.
- Start planning what I want to show at the midterm presentation.
Week of 11/2
- Goals: Label the Shimon dataset and get it ready to be used with YOLO.
- How I Did: Finished labelling the dataset and wrote a script to convert it to the format YOLO needs. LabelBox exports the data labels in JSON format, but they have to be converted to text files to work with darknet (the neural network framework I have been using YOLO with).
- Results:
- Need to do a couple more brief preparations and then I can train YOLO on the Shimon dataset.
- Next Steps:
- Train YOLO on just the Shimon dataset with no transfer learning or anything else fancy. I didn't realize last week that darknet came with command line options that make it relatively easy to train on different data.
- Get together the results for the midterm presentations.
- My goal past midterm presentations (for the final presentation) is to have YOLO trained on the Shimon data combined with some other significant dataset (COCO, etc.). This way the ability to detect Shimon will actually be useful in contexts outside of only looking for Shimon.
VIP Notes for 11-2.pdfWeek of 11/9
- Goals: Start training YOLO on the Shimon dataset.
- How I Did: Not great. I had trouble getting the GPUs in Couch to work with Darknet. I tried training on my CPU, but it would have taken roughly a week of training to get meaningful results.
- Results:
- Planning to work with Richard within two weeks or so to figure out the issue with the GPUs and get training.
- Next Steps:
- Start training as soon as possible.
- Since the end of the semester is getting close, start organizing my work.
Week of 11/16- Midterm Presentations
- I spent my VIP time this week organizing my work and getting ready for the presentations. My end of semester goal is to make an organized and relatively simple pipeline that future VIP students can use to add an object class of their choice to YOLO, so I began organizing that on Github here: https://github.com/kmorris72/detecting_shimon.
Google Sites
Report abuse