Data

Source of our Data

IMDB from December 2017: ftp://ftp.fu-berlin.de/pub/misc/movies/database/frozendata/

Transformations done to the Data

1. Removed
   • TV episodes (TV) or lines that start with a " can be removed
   • made for video entries (V) can be removed
   • video game entries entries (VG) can be removed
   • internet entries entries (internet) can be removed along with re-releases (re-release) and blu-ray releases (Blu-ray premiere)
2. Limited the data
   • any entry that doesn't have a year (????) can be removed
   • any entry with a rating of USA:X or USA:NC-17 can be removed
   • remove any entries in the following genres: Short, Adult, Reality-TV, Talk-Show, Game-Show, News, and any from any genres with fewer than 100 entries - note that this one is different as films can have multiple genres, so if a film has ANY of these genres all of its lines need to be removed, which should leave about 22 genres
   • there are 250,000+ unique keywords, so remove all keywords with less than 20 uses, this should remove about 95+% of them, and remove any lines that use those exact keywords. Note that you need to be careful how you remove lines with those less-used keywords to make sure you are doing exact matches with the entire keywords and not just substrings of the keywords.
   • for the certificates we are only going to use the USA: certificates, you can remove those from other countries. You can also remove the USA-TV certificates (TV-G, TV-14, TV-Y, TV-PG, TV-MA) and lines for any films that were re-rated for TV (TV rating) and anything with a video game certificates (E, E10+, T, C, M, Not Rated)
   • for running times we are only going to deal with films 60+ minutes long