Keyword Analysis

Association measures can be divided in:

1. effect size measures (e.g. PMI, Difference of Propositions, DICE Coefficient, etc.)

2. strength of evidence measures (e.g., Chi-Squared test, Log Likelihood Ratio, z-test, Fisher's exact test, etc.)

While effect size measures quantify the magnitude of a difference, strength of evidence measures quantify how statistically solid the evidence is that there is a true difference, by introducing the concept of "expected frequency".

Additional information about keyword measures here: 

• https://alvinntnu.github.io/NTNU_ENC2036_LECTURES/keyword-analysis.html 

• N. Levshina, "How to do Linguistics with R." (https://doi.org/10.1075/z.195)

Which measure fits best with my research question?

It depends. Consider that:

• Effect size measures are less reliable when absolute frequencies are very small.

• Effect size measures tend to favor low frequency words (such as content words, e.g., Plato “Plato”, Athenae “Athens”).

• Strength of evidence tend to favor high frequency words (such as function words, e.g. et “and”, est “is”).

• It is a good practice to combine one effect size measure with one strength of evidence measure.

How?

Have a look at the step-by-step tutorial to use different measures to extract keywords in Python (........).

How to interpret the results?

Pointwise Mutual Information (PMI) score

The output of PMI from the tutorial is a list of words in the target context associated with a score. Values range from -∞ to +∞. Zero is the neutral score. The higher the PMI score, the greater the attraction between the word and the target context. The lower the PMI score, the higher the repulsion between the word and the target context.

In the example, the words with the highest PMI scores are place names (Lixus, Horcus, Trinacria, etc.). This can be interpreted considering that geographical books contain tens of place names that are much less frequent (or not attested at all) in the rest of Pliny's work. In this sense, place names are keywords of the geographical books when compared with the rest of the NH.

Remember that PMI, as an effect size measure, favors low-frequency words, that is words with the highest PMI score may occur only once in the target context.

If necessary, you can set a minimum threshold of frequency of the word in the target corpus. For example, setting the threshold at 3 occurrences, the words with highest PMI in geographical books of the NH are: Sigeum “(cape) Sigeum”, Milesii “the Milesians”, emporium “market”, polus “pole”.

Difference of proportions (ΔP)

The output of ΔP is a list of words in the target context associated with a score. ΔP values range from -1 to 1, with zero being the neutral score. The higher the ΔP value, the higher the attraction between the word and the target context.

In the example, the words with the highest ΔP values are ab “from”, oppidum “city”, mare “sea”, amnis “stream”, mons “mountain” (see plot below). The measure effectively identifies frequent words related to geography which are keyword of the target context compared with the reference context.

The results are consistent, even after the top results. Other words with high ΔP scores are: flumen “river”, qui “who”, ad “to”, pes “foot”, sinus “gulf”, insula “island”, promontorium “promontory”, gens “people”, appello “to call”, terra “land”, pars “part”. 

Conversely, the words with the lowest ΔP scores are et “and”, ex “from”, in “in”, genus “kind”, folium “leaf”, uinum “wine”, sum “to be”, cum “with”, si “if”. These words are present in the target context, but are less or not attracted to it compared with the reference context. In fact, content words related to goods (uinum “wine”, mel “honey”, sucus “juice”, semen “seed”, acetum “vinegar”, oleum “oil”, herba “plant”), although attested in geographical books, are much more relevant in other books.

Chi-squared test (χ2)

Chi-squared quantifies the discrepancy between the observed and expected frequencies of words. It ranges from 0 to ∞. The higher the value, the stronger the evidence that the distribution of the word is not independent from the target context (with the Null Hypothesis being that the distribution is independent).

The second output of χ2 test in the tutorial is a list of words in the target context associated with a score called p-value for the chi-squared test. If the p-value is lower than 0.05, there is enough evidence to reject the Null Hypothesis.

In the example, words with the lowest p-value are oppidum “city”, promontorium “promontory”, paeninsula  “peninsula”, septentrio “north” which are keywords of the geographic books. 

Consider that we also find words such as mel “honey”, uinum “wine”, dolor “pain” which are not specifically related to geographic books. In this case, a low p-value indicate that their frequency significantly differ from the expected frequency (under-represented). [= the Chi-squared test is a two-sided test]

Log-likelihood Ratio (G)

Similar to the chi-squared test, the higher the G value, the stronger the evidence that the frequency of the word is not independent from the target context.

In the example, words with the lowest p-value for G score are oppidum “city”, colonia “colony”, herba “plant”, circuitus “perimeter”. The same considerations expressed for the chi-squared test are valid for the log-likelihood test.

Fisher's exact test

Similar to the chi-squared test, the higher the p value, the stronger the evidence that the frequency of the word is not independent from the target context.

Frequency-Inverse Document Frequency (TF-IDF)

TF-IDF is a statistical metric that measures how sparse a word is across a collection of documents. Unlike the methods described above, TF-IDF takes into account the number of documents in the collection.

The TF-IDF of a word in a document is calculated by multiplying the term frequency TF (i.e. the frequency of the word in the document) by the inverse document frequency IDF (i.e. the ratio between the total number of documents in the collection and the number of documents in which the word occurs).

TF-IDF values range from 0 (the word occurs at least once in every document in the collection) to 1 (the word is more frequent in the document and less sparse across the collection).

Words in the description of Greece with the highest TF-IDF score are the names of Greek places (Attica, Thessalia, Geraestus, Hebrus, Epiri, Aetolia, Macedonia) which are the most frequent words and less sparse across the geographic books.

Common nouns such as oppidum, promontorium, flumen, etc. have a TF-IDF score of zero due to the fact that they appear (at least once) in all geographical books. Interestingly, the adjective liber “free” and the common noun colonia “colony” have positive TF-IDF score, indicating that they are more specific to some geographical areas (including Greece) and are not mentioned in others.