Lesson 3

Introduction

Text analysis is the process of deducting meaning, finding facts, relatonships and assertions from text and other written communications such as the ones we find in social media (posts, comments, etc.) or in reviews from users of products, or user feedback. Text analysis consist of some techniques that help us understand and analyse large amounts of unestructured data. Some Text analysis techniques help in the process of word frequencies, categorization, clustering, pattern recognition, and visualization. It is used in marketing, advertising, but also in academic research from sociological, psychological, linguistic, rhetorical, and many other diverse approaches. However, Text analysis should be learned not just for specialized people, buy by anybody, since it is a new way of reading not only books in a corpus, but to be able to read the reality that surround us in many ways of communication.

Learning text analysis allows us to approach corpus of texts that we could not read naturally. With the help of technology, we can have a greater knowledge and a broader perspective of the content of a text, or as Matthew Jockers says in his book Text Analysis with R for Students of Literature (2014), many times it is just a matter of testing and verifying some hypotheses we have about a corpus. In my experience, more than verifying, it has been discovering many other significant aspects of the narrative that are not seen with the naked eye.

RStudio is a very helpful software that allow us performe text analysis since the pre-processing, the actual analysis and the plotting of graphs. Also, there are some packages in this software that may help us begin since the webscraping process to extract data from websites. RStudio is used by many people from a very diverse background, because it is for statistical analysis. In this lesson, our focus is on learning basic word frequency analysis of extracted text from an Association whose purpose is to help and build community among returned migrants to Mexico. So, the center of this lesson is based on some strategies we could improvise to analyse text when it is bilingual and bicultural. For this reason, this lesson is divided in the following sections:

1) Introduction, where we are going to see the basic structure of a command.

2) Pre-processing. In this section we will learn some commands to prepare the text for analysis.

3) Analysis of the corpus, where we are going to make some decision making, more cleaning and a table of word frequencies.

4) Frequency Graphs. In this part of the lesson, we will learn some commands to plot two bar charts.

5) Correlation between words. Finally, we will learn am command to see the correlation between the most frequent words and the rest of the corpus.

That said, in this lesson we will not go over learning to code, but we will learn basic R command structure and some commands used for pre-processing, analysis and ploting graphs, so that we can read what we are doing in every step of the code.

• Installation instructions for RStudio

Lesson

1) Introduction

library(tm)

library(NLP)

library(SnowballC)

library(wordcloud)

library(RColorBrewer)

library(ggplot2)

library(dplyr)

library(readr)

library(cluster)

#In the console you will see what we fixed in lesson 2:

#Sintax

#In R the most basic command follow this order:

(Assigner)

#Object <- Function ("Argument 1", "Argument 2", "Argument 3")

#Now we are going to create an object. For this lesson we are going to use the "odaclean.txt file which is the one we extracted since lesson 1. Then, we will use the read_lines function from the "readr" library.

oda_raw <- read_lines("odaclean.txt")

# Now, lets type the name of the object and run.

oda_raw

#In the console you will see the complete text because this .txt file contains only 103 posts.

#The oda_raw object is a character "chr" type with 103 elements.

#The "string" function str() in R Language is used for compactly displaying the internal structure of the object.

str(oda_raw)

#In the console you will see:

#Creating paragraphs

#Since posts is social media, and particularly in our text, we are going to create a vector called: diez (10) with ten repetitions (rep) of the numbers from 1 to the number of lines in the document divided by 10 (length(oda_raw)/10.

#By doing this, we will have a vector of groups of 10 lines (posts) each until the end of the 103 posts.

#diez <- rep(1:ceiling(length(oda_raw)/10), each = 10)

#This vector may not have the same number of lines per elements since each post has a different lenght.

#diez <- diez[1:length(oda_raw)]

#Now, we will create a new object (oda_text), so we will assign these groups of 10 to oda_text. We will have a column with the lines of text and another one with a number that identify each group.

#Also, we will make a data.frame so that columns can be identified with a name that will be helpful in the following steps.

#We use aggregate to concatenate the lines (posts) (FUN = paste, with collapse = " " to preserve the white space between words.

#What we are going to have now is a one column with the paragraphs and the number of the group.

#We will also transform oda_text into a matrix, since this will help us in the next steps.

oda_text <-

cbind(

rep(1:ceiling(length(oda_raw)/10), each = 10) %>%

.[1:length(oda_raw)],

oda_raw

) %>%

data.frame %>%

aggregate(

oda_raw ~ V1,

data = .,

FUN = paste,

collapse=" ") %>%

select(oda_raw) %>%

as.matrix

dim(oda_text)

#In the console you will see:

#Meaning that we have 11 groups.

#Type:

oda_text

#In the console you will see the divided paragraphs.

2) Pre-processing

#Now, we are going to continue with the pre-processing by removing special characters using regex. Regex will help us identify special characters such as: line breaks and tabs. We will use the regex [[:cntrl:]]

oda_text <- gsub("[[:cntrl:]]", " ", oda_text)

oda_text

#We need to get rid of some of the words such as: "http", "https", "bitly"... are there any more words to remove?

oda_text <- removeWords(oda_text, words = c("http", "https", "bitly"))

#Next, let's transform all the text to lower case letters.

oda_text <- tolower(oda_text)

oda_text

#Let's use removeWords with the stopwords("spanish") to remove the word from this langage that carry very little useful information.Example of these words are: prepositions and fillers. Then, we will do the same for the English Stop Words. Remember that these posts are written in both languages.

oda_text <- removeWords(oda_text, words = stopwords("spanish"))

oda_text <- removeWords(oda_text, words = stopwords("english"))

oda_text

#Then, we will remove punctuation.

oda_text <- removePunctuation(oda_text)

oda_text

#EIn this case, we will remove numbers, since we are not interested in dates or any other numerical information.

oda_text <- removeNumbers(oda_text)

oda_text

#Next, we will remove the extra white spaces between words. Some of them were produced after the changes we made.

oda_text <- stripWhitespace(oda_text)

oda_text

#In the console you will see a cleaner text:

3) Analysis of the Corpus

#Now that the text is prepared for the analysis, we will create a corpus. This corpus is made up of all the paragraphs. This corpus will be assigned to a new object called: oda_corpus. We will use the VectorSource and Corpus functions for this purpose.

oda_corpus <- VCorpus(VectorSource(oda_text))

oda_corpus

#In the console you will see:

#WordCloud

#Next, we will map our corpus. We will use the tm_map and PlainTextDocument functions for this purpose. We create a new object to assign these functions.

oda_ptd <- tm_map(oda_corpus, PlainTextDocument)

#Now, we will be able to create a word cloud using the library of this same name. This word cloud will have the most frequent words of the corpus.

wordcloud(oda_ptd, max.words = 80, random.order = F, colors = brewer.pal(name = "Dark2", n = 8))

#In the plot window you will see:

# Do you see any more more words that need to be removed?

# If your answer is yes, we can proceed to a second round for cleaning using the removeWords function.

# This function requires the vector of characters that we used earlier (oda_text).

oda_text <- removeWords(oda_text, words = c("zwgmj"))

#Once we have cleaned the oda_text for the second time. We have to use this new version of the oda_text to generate a new corpus and map it again.

oda_corpus <- oda_text %>% VectorSource() %>% VCorpus()

oda_ptd <- oda_corpus %>% tm_map(PlainTextDocument)

#Next, we will create a new word cloud, that may be different.

wordcloud(oda_ptd, max.words = 30, random.order = F, colors=brewer.pal(name = "Dark2", n = 8))

#In the plot window you will see:

#Term Document Matrix

#Now, we will map the corpus as a matrix, so that in the future steps we can identify correlations between words.

#The Document Term Matrix lists all occurrences of words in the corpus and by document. In the matrix we can see the documents in rows and the words in columns. This matrix will assign a number according to the number of a word occurs in a particular document. If it does not appear in the document, then the entry is "0", if it is only one word in the document, then "1" and if the word is twice in the same document, then, the number will be "2".

# We will use the function TermDocumentMatrix in the corpus and assign the result to a new object called: oda_tdm.

oda_tdm <- TermDocumentMatrix(oda_corpus)

oda_tdm

#In the console you will see:

#According to the resulting information we have 1408 terms in the 11 documents (paragraphs). This means that there are 1408 unique words in the corpus.

Non-/ Sparce entries: 2517/12971

This means that in 2517 entries there are words (1408 of them appear only once in the corpus / while the sparce entries are 12971 which means that in that number of entries we may find zeros. In this case, 84% of the matrix entries are zeros.

#Word frequency

#Now, it is time to prepare the corpus to see quantities of the frequency of words. We need to transform the oda_tdm object in a matrix object. This object will have the number of unique words as lines and the number of documents as columns.

oda_mat <- as.matrix(oda_tdm)

dim(oda_mat)

#In the console you will see:

#Now, we will get the sum of lines (rowSums) ordered from highest to lowest sort(decreasing = TRUE) to learn the frequency of each word. After that, we will transform the results to a new object of data.frame class, so we can have two columns, word and frequency in a table.

oda_mat <- oda_mat %>% rowSums() %>% sort(decreasing = TRUE)

oda_mat <- data.frame(palabra = names(oda_mat), frec = oda_mat)

#With the matrix object we can also create a word cloud.

wordcloud(

words = oda_mat$palabra,

freq = oda_mat$frec,

max.words = 70,

random.order = F,

colors=brewer.pal(name = "Dark2", n = 8))

#In the plot window you will see:

#Next, let's type:

oda_mat[1:20, ]

#In the console you will see the table of frequencies:

4) Frequency Graphs

#Frequency bar charts

We will create a bar chart with the number of frequency of the words. For that purpose, we will use ggplot2. Ggplot has its oun functions that we will ot see here, but it is important to mention that we are using the oda_mat for this purpose.

oda_mat[1:20, ] %>%

ggplot(aes(palabra, frec)) +

geom_bar(stat = "identity", color = "black", fill = "#87CEFA") +

geom_text(aes(hjust = 1.3, label = frec)) +

coord_flip() +

labs(title = "Twenty most frequent words", x = "Words", y = "Number of frequencies")

#In the plot window you will see:

#Now, instead of using the number of frequencies, we are going to use percentage of use. For this purpose we will use the dplyr library.

oda_mat %>%

mutate(perc = (frec/sum(frec))*100) %>%

.[1:20, ] %>%

ggplot(aes(palabra, perc)) +

geom_bar(stat = "identity", color = "black", fill = "#F5B041") +

geom_text(aes(hjust = 1.3, label = round(perc, 2))) +

coord_flip() +

labs(title = "Ten most frequent words", x = "Words", y = "Percentage")

#In the plot window you will see:

5) Correlation between words

We are going to use a vector, so we are going to be able to introduce more than one word in the command.

The function we will use is findAssocs. First we will use only one word. In this case, let's find the correlation of "join"
with other words from the text.

The correlation (corlimit) near zero means that the words do not correlate. On the contrary, if the values is close to 1, means that the terms correlate.

Let's use the .7 and see what happens:

findAssocs(oda_tdm, terms = c("join"), corlimit = .7)

#In the console you will see:

#As you can see there are 9 words that higly correlate with join. You may change the corlimit to see other correlations. Also you can find more associations of the most frequent words and use a different corlimit and see what happens. Also, you will decide the corlimit according to your own research interests.

Congratulations!!! You have completed a basic text analysis and you have created word clouds and graphics to plot the data from extracted text. YAY! You are ready to work by yourself and extract data of your interest, clean it and analyze it. Best of luck!