I started a spreadsheet and started listing words that had large noticeable temporal spikes in their frequency. If the reason for a word's frequency wasn't obvious, I used the Contexts tool to see if there was a clear usage pattern - the Context tool can be applied to a specific segment of the corpus:
Often there wasn't, but occasionally a word that seemed to increase in frequency randomly was part of a phrase that made the historical context clearer.
For instance, the word "looked" has a huge spike in 2004, quintupling in relative frequency from .00007 to .00055 - this seems hard to explain
Looking at the context of the word "looked" for 2004, I noticed that the word "intelligence" showed up near it in about a third of its usages.
I searched Trends again, and added a bit of metacharacter search magic: "looked intelligence~7", which looks for all occurrences of both words within 7 words of each other, we see that this pattern explains our jump. President Bush used the phrase "we looked at the intelligence" or some variation of it many times that year while explaining his justification for the invasion of Iraq.
IMPROVING MY RESEARCH QUESTION:
By identifying these outlier words, and drilling down on them to understand the context they were appearing in, I started to form a system a categorization. I came up with four basic reasons why a specific word or short phrase would spike in frequency.
For some words, a change in usage reflected simply a change in either the nature or presentation of the material in the corpus. For instance, "proclamation":
"Proclamation" might seem to be a favorite word of Ronald Reagan based on this graph, but what actually happens is that "Presidential Proclamations" are gathered together in an appendix starting in 1989, rather than being included as individual sections. There are some more subtle occurrences of this kind of change in document style as well - for instance, "resides":
The sudden dropoff here is explained by a change in the format of official announcements of new appointments to federal office; their biographical details and place of residence stop being included in the announcements. The word "married" has a nearly identical pattern for this reason.
Many changes in word frequency can be explained by the news cycle, or by policy initiatives being pushed by the administration at the time, such as the earlier example of "looked". Other examples were easy to find in the data:
The words "crime" and "terrorism " here show noticeable spikes in usage that are correlated to the 1994 Crime bill championed by the Clinton administration, and 9/11.
The events that caused the increased usage of "cleanup" in the corpus are the Exxon Valdez oil spill and the Deepwater Horizon disaster.
Why did things get longer in 2005? No obvious answer springs out, but the Context tool shows that G.W.Bush's responses to a variety of policy issues involved this word:
Some phrases tend to become more common every four years - while I didn't drill extensively into the context for all of these phrases, it's easy to draw a correlation between election years, particularly when these phrases are most common in second-term election years (1984,1992,1996,2004, 2012)
Families, for instance, seem to become much more significant when the President is being challenged in an election.
Governors or members of Congress get mentioned a lot more often by the President when they're trying to replace him in an election, as in 1992, 2004, and 2012.
Finally, there are some words and phrases that seem to increase in frequency for no other reason than that the president says them a lot. Although this of course can overlap a great deal with policy matters and agendas. For instance, one of the most striking graphs is this:
Although previous Presidents surely had enemies, George W. Bush used the word with a vastly greater frequency than any other President, which is consistent with the colorful and often assertive nature of his personal speech patterns.
Lest we accuse G.W. of simplicity, though, we should also note that he also greatly expanded the popularity of "love" in the presidential lexicon.
"Guys" is interesting in that it seems like a nondescript word, but there is clearly a similar pattern of usage between G. H. W. Bush and Obama, two speakers who otherwise do not seem to share any obvious stylistic traits in public speaking.
Another example where two presidents who seem to have dissimilar speaking styles - G. W. Bush and Obama - share a frequency trait that none of the other have. Perhaps this represents an overall trend towards less formal language?
A subset of the "Personal Quirks" category are words that seem to be more popular with either Democrats or Republicans. For instance:
The Bushes and Clinton show a noticeable contrast in their respective use of the words "message" and "thought". Reagan and Obama, however, do not.
My other method of probing this dataset, comparing terms that seemed conceptually connected, was less productive in terms of interesting results, largely because of a lack of any quantitative process; I was just thinking up stuff that seemed like it might be interesting. Sometimes the lack of distinct patterns is itself interesting, though.
With the exception of the period during the fall of the Berlin Wall, the nations of Western Europe are very similar in their frequency, suggesting that operate as a bloc in both policy and perception.
This is in rather striking contrast to the nations of the middle east, where one nation in particular commands pretty much all the attention. Even Afghanistan is mentioned far less as late as 2014, although it would be very interesting to see how this trend continues in the present.
This comparison of hot-button policy issues shows a similar overall pattern as the European nations; some topics have cycles of recurring attention, but the relative importance of these items remains fairly similar over time.
Using a somewhat more non-qualitative approach, I was able to dispel a popular misconception:
The scale here is the actual count of usages, rather then the relative frequency used in the other graphs. The phrase "I feel your pain", famously linked to Bill Clinton, is only recorded on 5 occasions, and the same phrase is used by G. W. Bush 4 times. I'm not sure what this means, but it seemed significant enough to record.
Tolstoy is consistently more popular than Dostoevsky.
I did try using Voyant's Topic Analysis tool on the corpus, but the results didn't seem particularly meaningful. To successfully extract topic information or other machine-learning driven data from the corpus, I suspect it would have to be segmented at much greater depth, ideally with each individual "chapter" within the volumes being a segment. This would require a fairly high level of resources both in terms of computational power and time.
Comparing this text to another source of dated news-type material, such as AP press reports, would also be interesting but difficult.