Jacob

Week of September 17

• Goals: Research CNNs and basic UI mockup
• How I did
  • Got a basic idea for a QVFM UI, described below
  • Gained more knowledge about CNNs
  • Also read UVA paper on MovieScope and Anna Weisling post on "Logan" trailer analysis.
• Results:
  • UI idea
    • Basic desktop application windowed display, with a toolbar across the upper part of the screen
    • A toolbar element called "Analyze" which will prompt the user for a video file and then run the analysis on it
    • Other various config tools in toolbar
    • If a video is analyzed, the main part of the screen shows a "timeline"
      • Can use actual timestamps in seconds or frames, depending on how the visual analysis is done
      • Composer can view separate timelines for each charater identified by the analysis tool, select the character by tabs underneath the timeline
        • Displays when the character is on screen and the emotional context of their appearance
      • Also a complete summary including all identified characters can be shown
      • As much visual representation as possible, very little actual verbiage to read/parse for the composer
        • Quick visual summary
    • Can save/display previous analysis results (export the timeline? export the output of visual analysis tool, then generate visualization on load?)
  • CNNs
    • Better higher level understanding, need to get my hands dirty with CNNs
• Next Steps
  • Continue to develop CNN knowledge, try coding some examples
  • Identify useful information to display to a composer given a film to analyze(emotional high points, key scenes, etc)
  • Sketch/mock my UI idea

Week of September 24

• Goals: Continue CNN familiarization, draft a UI mockup
• How I Did: Watched YouTube lectures on CNNs and read about how they worked in further detail. Drew a handsketch of my idea for a UI for the Quick Visual Analysis and brainstormed relevant qualities of a movie to a composer
• Results
  • Relevant Information To A Composer
    • Emotional points
      • Emotional Highs (happy/joyful parts)
      • Emotional Lows (sad/defeated parts)
    • Character appearances
      • When a character appears
      • How often they appear
    • Key events
      • Important Character deaths
      • Battles
    • Changes in overall mood/tone of a movie
    • Potentially map interactions between characters
    • Scene changes
  • UI Sketch (Richard has photos)
• Next Steps
  • Work on CNN model that Richard sends me

Week of October 1

• Goals - Get this face detection/ emotion and gender classification model up and running, and seeing how it can be applied to the final tool.
• How I Did: I cloned the git repo and ran both the real-time classification demo and the singe-image classification demo on my own test images.
• Results
  • Test images - See below. These were just my own images, there were included test images in the repo as well
    • In my tests, the model didn't perform super great at detecting multiple faces - probably to be expected given all the potential noise in random test photos and differences from the training set.
  • This team was interested in building models which eliminate the final fully-connected layer, because they were interested in building a model for real-time systems (in particular, robots). Real-time systems need to be fast, so eliminating the final fully-connected layer, and thus the majority of parameters in a deep-learning neural network, will help with this.
    • Speedup of 1.5 when compared to the architecture of this model.
    • I think for our tool (at least how the visual analysis models are currently structured), we wouldn't need it necessarily to be real-time, and thus we don't need to eliminate the final fully-connected layer like this team. However, if we potentially move to real-time video analysis as opposed to running the CNNs over individual, single frames, then this is an interesting idea to improve performance
  • Real-time demo seemed to work okay, it was able to track how my emotions were changing in real time reasonably accurately
• Next Steps
  • Expand this analysis model to any arbitrary video

Week of October 8

• Goals - write a program to perform emotion/gender analysis on an arbitrary video
• How I Did
  • Initially, I tried to write a Tkinter app which will display the video in a window and have a button which will capture the current frame being displayed and perform the deep-learning emotion/gender analysis on that frame.
    • I ran into some issues getting Keras to work with the Tkinter app.
  • Ended up with just a Python script for the time being. My next goal would be to convert the functionality of the script to a GUI/Tkinter application
• Results
  • My script does the following: when you provide it with a path to a video, it will take the video, convert it to individual frames, and write those frames to a directory. Then, it will run the model on each frame in the new frame directory. Finally, it will save the analysis done on the individual frame as an image, and write it to a new 'frame_analysis' directory.
• Next Steps
  • Find potential better deep learning models to use
  • Merge my script with the other scripts in the group

Week of October 15

• Goals - Find an improved face detection model
• How I Did
  • Found this model and wrote another function in my script which uses this model.
  • I also learned that this library has support for facial recognition, so I implemented basic facial recognition, detecting and remembering characters as they appear on screen
• Results
  • The facial detection is much more consistent than the previous week.
  • Facial recognitions works okay, could definitely be more consistent.
• Next Steps
  • Improve facial recognition
  • Display occurances and frequency of identified characters

Week of October 22

• Goals - Improve facial recognition and display occurances and frequency of identified characters
• How I Did
  • I modified my script to dump the occurances by frame number of each character into a csv file
• Results
  • The facial recognition is about the same as before. I may switch the face detection model to a deep net rather than HOG because it is slightly more accurate. The tradeoff, however, will be in run-time of the script.
  • The CSV file is good for raw data. I'd like to do some visualization of the data.
• Next Steps
  • Reintroduce emotional analysis of faces
  • Produce visualizations of the emotion data and character appearance data.

Week of October 29

• Goals - Reintroduce emotional analysis / produce visualizations of my current data
• How I Did
  • Attempted to re-introduce the emotional recognition from the first model I used.
  • Explored potential ways to visualize my character appearance data (matplotlib, Tableau, etc)
• Results
  • Had difficulty meshing the old emotional analysis model with the current facial recognition model
  • May have to reconsider the way I have my data formatted to easily visualize it.
• Next Steps
  • Prepare for 3/4 presentations
  • Continue to try to reintroduce emotional analysis, find a new model if needed

Week of November 5

• Goals - Prepare for 3/4 presentation, continue to explore emotional analysis models
• How I Did
  • Looked around on GitHub for alternative models
• Results
  • Couldn't really find a useful new model
• Next Steps
  • Continue to try to re-implement emotional analysis with my current script

Week of November 12

• Goals - Try to re-implement emotional analysis with my script
• How I Did
  • Not well. I wasn't able to get the old emotional analysis model to work with my new facial detection model, and I haven't been able to find a decent standalone model for detecting emotion in faces.
• Results
  • Couldn't find a good standalone emotional mode. I'm going to table this and move in a new direction and focus on improving the readability of my data.
• Next Steps
  • Find a new metric to measure characters instead of just raw frames that they appear in

Week of November 19

• Goals - Find a new, more concise way to weight character appearances rather than just a CSV of the frames that they appear in
• How I Did
  • I decided that number of seconds of on-screen time would be a step up from what I have currently, and I implemented logic that generates a CSV file showing the number of seconds(roughly) a character is on-screen.
• Results
  • Now have a more concise way to show character involvement in a film
• Next Steps
  • Generate a visualization for this data

Week of November 26

• Goals - Add visualization for character onscreen time data.
  • How I Did
    • Successfully implemented logic to generate a visualization of the onscreen time in seconds from a CSV
  • Results
    • Now, when run, the program generates individual frames with characters identified, two CSV files, and a visualization showing which characters appeared onscreen most often
  • Next Steps
    • Prepare for final demo

Week of December 3