US PRESIDENTS' SPEECH READABILITY

This project analyzes the readability of “State of the Union Addresses and Messages” from 1993-2017 to track the readability pattern over time and among the four most recent presidents (William J. Clinton, George W. Bush, Barack Obama, and Donald J. Trump).

The readability of “State of the Union Addresses and Messages” varies among different presidents but is consistent (standard deviation less than 0.5) during the presidency of a given president.

The speech scripts come from American Presidency Documents Archive: http://www.presidency.ucsb.edu/sou.php

“State of the Union Addresses and Messages” from 1993 to 2017 will be copied and pasted from the website as plain text. Each document will be named in the format of “PresidentNameYear.txt” as separate .txt files.

Downloaded Text Files: https://drive.google.com/open?id=0B2rUC8oOBYH4bU5zQ3BwelAwZ1U

1. Read all raw text files from the directory specified by user in the command line. Command should be entered in the following format: python3 AoyingHuang_Proj2.py directory_of_rawtext_folder. Error message will appear if elements lacking in the command line.
2. For each raw text file, first count the number of sentences using the function called sentencecount. This function find the number of sentence end-start patterns with Regular Expressions (Regex) and the total # of sentences = # of patterns + 1. Specifically, the pattern includes the following 5 cases:
   • .... + space + any letter (case insensitive)/digits or "
   • ? + space + any letter (case insensitive)/digits or "
   • ! + space + any letter (case insensitive)/digits or "
   • . + space + any letter (case insensitive)/digits or "
   • " + space + any letter (case insensitive)/digits or "
3. Cleaning text: lowercase all words, remove punctuations, digits, and anything in [] as they are not part of the actual speech.
4. ALGORITHM 1: AUTOMATED READABILITY INDEX: The automated readability index requires the number of characters, the number of words, and the number of sentences in a text for computation. This index corresponds to a grade level.
   • Turn the text file into a list of words with textfile.split().
   • The length of this list is the total number of words and the number of elements in this list is the total number of characters in the text file.
   • Calculate ARI with formula: ARI = 4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43
5. ALGORITHM 2: FLESCH-KINCAID READABILITY INDEX: This index computes the number of syllables, the number of words, and the number of sentences in a text for computation. This index estimate the grade level required for understanding a given text.
   • Turn the text file into a list of words with textfile.split(). The length of this list is the total number of words in the text.
   • Count the number of syllables using the following pattern with Regex:
     • when a vowel (excluding e) is present. This vowel may be followed by zero or more adjacent vowels (including e).
     • when the vowel e is present, followed by one or more adjacent vowels (including e).
     • when the vowel e is followed by exactly one non-vowel letter.
   • Calculate FKGL with formula: FKGL = 0.39(total words / total sentences) + 11.8(total syllables / total words) - 15.59
6. Format the results in the order of "President's Name + Year of the Speech + ARI + FKGL". Write the output result in a new .txt file.

Python Program: https://drive.google.com/open?id=0BwLqgtoS-HrCWDhCYXhreEFtSEE

For instructions, please refer to the comments at the beginning of this Python program.