Saturday Night at the Movies

Amy Ngo - Separate .list files in Python to CSV file, Clean up data in R, Format data in R.

Desiree Murray - Clean up data in R, Format data in R.

Links

Youtube

https://youtu.be/mX-VnxD-Bqc

GitHub

https://github.com/amyngo3/cs424_project3

Site

http://shiny.evl.uic.edu:3838/p3g5/

Google Doc

https://docs.google.com/document/d/1gJMWqOxxzxTdzs8VoewGEd_Dj_MQXXXmjDW_VnAcrmc/edit?usp=sharing

How To Use

We decided to have the quick info boxes at the top for users to know the average films per year, month, and running time. Then the user can select a year and decade drop down box that changes the data on multiple graphs.

The plots on the dashboard allow the user to glean information about various aspects of the database. They are able to see the top ten keywords that are associate with films as well as the distribution of genres, certificates, and running times. Additionally, there are plots that display the number of films released per month and year across the entire database.

Finally, there is an about page that can be selected from the dashboard sidebar menu which displays information about the creation of the dashboard.

Data

We converted the list files to text files and removed lines before and after the list of films. Then we used the Python language to find the correct criteria needed into a CSV file. In the process of filtering the data, Python’s regular expression is convenient to sort out data not required and data we need. Some data in a few files could not be sorted, thus we used R to sort the data.

The data came from the internet database (IMDB) which are available at ftp://ftp.fu-berlin.de/pub/misc/movies/database/frozendata/

Another site we used is https://regex101.com/. This site is very useful to write regular expressions and test strings that fit within the expression. The explanation box provides details of the typed regular expression, match information will match the test string in groups, and a quick reference guide for the user.

The files used were:

• release-dates.list
• running-times.list
• certificates.list
• genres.list
• keywords.list
• movies.list

Interesting Things

• Despite the amount of keywords to remove, a lot of the data in the keywords.list file also had many lines to remove. Thus, we created multiple CSV files in order to get faster results from the Python code.
• Running-times.list file had many complications just to simply get time over 60. Whether there is a country, colon, semi-colon, etc. that have to be considered to extract the data.
• From the plots we could see that the top 3 most common film genres were Drama, Comedy, and Documentary, and the top keywords were"independent-film", "based-on-a-novel", and "character-name-in-title"
• Based on our data there seem to be two large spikes in the release of films, in the early 1910's and in the 2010's

Sites References

• StackOverflow
• RDocumentation

Libraries Used

• Ggplot2
• Lubridate
• Stringr
• Shiny
• Shinydashboard
• Dplyr
• Hrbrthemes