Keyword Extraction

Keyword extraction is the automated process of extracting the words and phrases that are most relevant to an input text. Keywords are routinely used for many purposes such as

• Identifying topics of interest

• Retrieving documents during a web search

• Summarizing documents for indexing

• Helps in Topic Modeling tasks

Steps involved,

• Pre-Processing

• Candidate generation

• Candidate Scoring

• Final Ranking

Approaches used for Keyword Extraction

There are multiple techniques are available for extracting the keyword from the large text data.

• Spacy

• Naive Counting

• Gensim

• TF-IDF

• TextRank algorithm

• Rake

• Yake

TF-IDF (Term Frequency and Inverse Term Frequency)

Term Frequency – How frequently a term occurs during a text. it's measured because the number of times a term t appears within the text / Total number of words within the document

Inverse Document Frequency – How important a word is during a document. it's measured as log(total number of sentences / Number of sentences with term t)

TF-IDF – Words’ importance is measure by this score. it's measured as TF * IDF

Here are some steps Involved while implementing,

1. Import Packages

2. Declare Variables

3. Remove stopwords

4. Find total words in the document

5. Find the total number of sentences

6. Calculate TF for each word

7. Function to check if the word is present in a sentence list

8. Calculate IDF for each word

9. Calculate TF * IDF

10. Create a function to get N important words in the document

11. Get the top 5 words of significance