Embedded Files
Stopword list
Stopword list
This list is for text segmentation, including symbols, pronouns and some modals which help to divide the whole sentence into different parts.
This list is for text segmentation, including symbols, pronouns and some modals which help to divide the whole sentence into different parts.
Badword list
Badword list
This word list includes the words describing sexual content, violence, addiction of tobacco and drugs and dirty words, people under 17 are not admitted.
This word list includes the words describing sexual content, violence, addiction of tobacco and drugs and dirty words, people under 17 are not admitted.
Bayes probability of words in general novels
Bayes probability of words in general novels
This list is the calculation of frequency of the words in general novels. The program figures out the 50th most important keyword in 500 general novels based on TF-IDF algorithm and calculate the Bayes probability.
This list is the calculation of frequency of the words in general novels. The program figures out the 50th most important keyword in 500 general novels based on TF-IDF algorithm and calculate the Bayes probability.
Test example
Test example
This is one of the best test output of this model. The training set generates two word probability lists from 649 G level novels and 335 A level novels, and the test set is composed with 50 G level novels and 50 A level novels. For each time, 20 novels are randomly chosen from the test set. The test process has been run for 10 times, and the accurrancy is close to 90%.
This is one of the best test output of this model. The training set generates two word probability lists from 649 G level novels and 335 A level novels, and the test set is composed with 50 G level novels and 50 A level novels. For each time, 20 novels are randomly chosen from the test set. The test process has been run for 10 times, and the accurrancy is close to 90%.
Future work
Future work
Although the test run out a pretty results, some weak points are still exit：
Although the test run out a pretty results, some weak points are still exit：
1. As the novels in training set are limited, the Bayes probability of words isn't accurate enough, the more novels to train, the more accurate probability we can get.
1. As the novels in training set are limited, the Bayes probability of words isn't accurate enough, the more novels to train, the more accurate probability we can get.
2. The classification process is just based on the 50th keywords of a novel, other words that are not too relevant should also be considered when doing the classification.
2. The classification process is just based on the 50th keywords of a novel, other words that are not too relevant should also be considered when doing the classification.
3. The wordlists in this model just including nouns and verbs. Chinese writing is creative, new words are being created everyday, not only nouns and verbs but also adjectives and other more. The wordlists should be extend to record more kinds of words in later time.
3. The wordlists in this model just including nouns and verbs. Chinese writing is creative, new words are being created everyday, not only nouns and verbs but also adjectives and other more. The wordlists should be extend to record more kinds of words in later time.
4. The update of bad words is too rigid now, the probability should not be the only judgement when doing the filtering and updating.
4. The update of bad words is too rigid now, the probability should not be the only judgement when doing the filtering and updating.
In the future, more novels should be run in this model to keep updating the word probability. In most of the country, rating system of novels has at least three classes: G, PG-13 and A. However, the model right now just includes two rating level. Thus, the model also needs an update to distinguish more rating levels.
In the future, more novels should be run in this model to keep updating the word probability. In most of the country, rating system of novels has at least three classes: G, PG-13 and A. However, the model right now just includes two rating level. Thus, the model also needs an update to distinguish more rating levels.
Google Sites
Report abuse