Embedded Files
In this project, the learning model is based on naïve Bayes classifier. The following picture shows the processes of the training model using in this project. First, a database is set up as the training data set which includes two kinds of novels, one for general level and one for adult level. All the input novels are .txt files encoding in utf-8 and gbk. After text segmentation and removing stop words and repetition, the original input will be abbreviated to several sets of words. A concept learning model based on Find-S algorithm will be implemented to extract a bag of words only for adult level novels called “bad words” and their average frequency. The result then becomes an important component for the following rating system. For each new novel, by doing a naïve Bayes classification, the model will give out an appropriate rating class and some unknown words toward the current database, if those words appear in the later novels which are marked with adult level, they will be add to the “bad words” database and update the rating principle.
In this project, the learning model is based on naïve Bayes classifier. The following picture shows the processes of the training model using in this project. First, a database is set up as the training data set which includes two kinds of novels, one for general level and one for adult level. All the input novels are .txt files encoding in utf-8 and gbk. After text segmentation and removing stop words and repetition, the original input will be abbreviated to several sets of words. A concept learning model based on Find-S algorithm will be implemented to extract a bag of words only for adult level novels called “bad words” and their average frequency. The result then becomes an important component for the following rating system. For each new novel, by doing a naïve Bayes classification, the model will give out an appropriate rating class and some unknown words toward the current database, if those words appear in the later novels which are marked with adult level, they will be add to the “bad words” database and update the rating principle.
The rating system here is mainly assembled by two parts: the “bad words” and the allowed frequency. To make the problem uncomplicated, all novels will be divided to only two categories: General and Adults. Any novel consists of words describing sexual content, violence and addiction of tobacco and drugs which shows in “bad words” database will be considered as Adult level. Also, an allowed fervency for impolite conversation like profanity and cursing will be restricted to incur in General level.
The rating system here is mainly assembled by two parts: the “bad words” and the allowed frequency. To make the problem uncomplicated, all novels will be divided to only two categories: General and Adults. Any novel consists of words describing sexual content, violence and addiction of tobacco and drugs which shows in “bad words” database will be considered as Adult level. Also, an allowed fervency for impolite conversation like profanity and cursing will be restricted to incur in General level.
Classifier
Classifier
Notice: If P(ki|G) = 0 or P(ki|A) = 0, means that the word probability lists are not including such a word, the model set this value to the same magnitude of the minimum in word probability. For instance in the test example, this value is 0.001.
Notice: If P(ki|G) = 0 or P(ki|A) = 0, means that the word probability lists are not including such a word, the model set this value to the same magnitude of the minimum in word probability. For instance in the test example, this value is 0.001.
Google Sites
Report abuse