NLP Tools

Jieba

"Jieba" (Chinese for "to stutter") is a Chinese text segmentation tool.

1. Features

(1) Support three types of segmentation mode: Accurate Mode, Full Mode, Search Engine Mode.

(2) Supports Traditional Chinese

(3) Supports customized dictionaries

(4) MIT License

2. Algorithm

(1) Based on a prefix dictionary structure to achieve efficient word graph scanning. Build a directed acyclic graph (DAG) for all possible word combinations.

(2) Use dynamic programming to find the most probable combination based on the word frequency.

(3) For unknown words, a HMM-based model is used with the Viterbi algorithm.

3. Functions

1. Cut

(1) Full Mode: gets all the possible words from the sentence. Fast but not accurate.

Input:

Output:

(2) Accurate Mode (Default): attempts to cut the sentence into the most accurate segmentations, which is suitable for text analysis.

Input:

Output:

(3) Search Engine Mode: based on the Accurate Mode, attempts to cut long words into several short words, which can raise the recall rate. Suitable for search engines.

Input:

Output:

2. Keyword Extraction

(1) TF-IDF algorithm

TF-IDF is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. TF refers to term frequency, which always counted by the number of times that term t occurs in document d. IDF refers to inverse document frequency, which is a measure of how much information the word provides, that is, whether the term is common or rare across all documents. It is the logarithmically scaled inverse fraction of the documents that contain the word, obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.

(2) Textrank algorithm

TextRank is an unsupervised method for computing the extractive summary of a text based on PageRank algorithm. The expected end result for this application is a set of words or phrases that are representative for a given natural language text. The units to be ranked are therefore sequences of one or more lexical units extracted from text, and these represent the vertices that are added to the text graph. Any relation that can be defined between two lexical units is a potentially useful connection (edge) that can be added between two such vertices. We are using a co-occurrence relation, controlled by the distance between word occurrences: two vertices are connected if their corresponding lexical units co-occur within a window of maximum words, where can be set anywhere from 2 to 10 words. Co-occurrence links express relations between syntactic elements, and similar to the semantic links found useful for the task of word sense disambiguation (Mihalcea et al., 2004), they represent cohesion indicators for a given text.

(3) Text segmentation category

Stopwordlist

In computing, stop words are words which are filtered out before or after processing of natural language data (text). Though "stop words" usually refers to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools, and indeed not all tools even use such a list. Some tools specifically avoid removing these stop words to support phrase search. Here we select the stop words based on <A Dictionary of Current Chinese> published by Institute of Languages, Chinese Academy of Social Sciences.