Embedded Files
Keywords: Novel rating system, Python 3, Naive Bayes classification, Nature language processing, TF-IDF, Word2vec, Jieba
Keywords: Novel rating system, Python 3, Naive Bayes classification, Nature language processing, TF-IDF, Word2vec, Jieba
Introduction
Introduction
The wide use of electronic equipment brings a convenience for reading, following by this trend, e-books seem to replace traditional paper books to be our first choice for reading especially for reading long novels. However, there is a consensus around the world that some special topic like sex, gambling and drugs should not be permitted on minors’ reading materials. Thus, a rating system for novels becomes an urgent demand for parents in China, as many of them insist on content classification for minor and adult. This project aims at building a machine learning model to analyze the content of novels written in Simplified Chinese and give them appropriate ratings. The model will be implemented in Python 3.
The wide use of electronic equipment brings a convenience for reading, following by this trend, e-books seem to replace traditional paper books to be our first choice for reading especially for reading long novels. However, there is a consensus around the world that some special topic like sex, gambling and drugs should not be permitted on minors’ reading materials. Thus, a rating system for novels becomes an urgent demand for parents in China, as many of them insist on content classification for minor and adult. This project aims at building a machine learning model to analyze the content of novels written in Simplified Chinese and give them appropriate ratings. The model will be implemented in Python 3.
Problem Analysis
Problem Analysis
Traditionally, e-book publishers in China like Jinjiang BBS and Baidu online library have their classification department, in which a group of people take charge in reading all the publish content in turns and decide if the books are permitted for all or just for adults. This work needs a lot of time. A general manual classification process shows as below: a reviewer firstly reads the abstract of the book to know the theme, and then randomly chooses some chapters in different parts of the book to check if it has some inappropriate content, if not pass it to a second reviewer, otherwise marks it. The computer can do the same work in short time by using machine learning algorithms to build a e-classifier model to simulate this process. The key problem for this study is to decide which words should be mark and let the model learn them. Different from other nature language processing tasks, this study will also focus on solving a common difficulty in Chinese novel classification: the model should have the ability to recognize new phases that describe prohibited contents as writers are always creative.
Traditionally, e-book publishers in China like Jinjiang BBS and Baidu online library have their classification department, in which a group of people take charge in reading all the publish content in turns and decide if the books are permitted for all or just for adults. This work needs a lot of time. A general manual classification process shows as below: a reviewer firstly reads the abstract of the book to know the theme, and then randomly chooses some chapters in different parts of the book to check if it has some inappropriate content, if not pass it to a second reviewer, otherwise marks it. The computer can do the same work in short time by using machine learning algorithms to build a e-classifier model to simulate this process. The key problem for this study is to decide which words should be mark and let the model learn them. Different from other nature language processing tasks, this study will also focus on solving a common difficulty in Chinese novel classification: the model should have the ability to recognize new phases that describe prohibited contents as writers are always creative.
More Information
More Information
Introduction of algorithms using in this model. Click here
Introduction of algorithms using in this model. Click here
Basic tools and concepts for this classifier. Click here
Basic tools and concepts for this classifier. Click here
Details and flow chart for this model. Click here
Details and flow chart for this model. Click here
How to Use
How to Use
Click here to download the classifier.
Click here to download the classifier.
The classifier includes two parts : Wordlibrary and Model . Here is the introduction of them and the steps to use it.
The classifier includes two parts : Wordlibrary and Model . Here is the introduction of them and the steps to use it.
1. Wordlibrary includes 2 tools：
1. Wordlibrary includes 2 tools：
（1）common stop word list
（1）common stop word list
（2）Some bad words that should not show in any A level novels.
（2）Some bad words that should not show in any A level novels.
Attention： People under 17 are not admitted to see this list!!!!
Attention： People under 17 are not admitted to see this list!!!!
2. Model is the code of the classifier
2. Model is the code of the classifier
(1) Get_Words_Probability.py : This is to find out Bayes probability of words show in novels.
(1) Get_Words_Probability.py : This is to find out Bayes probability of words show in novels.
(2) Classification.py : This is a Naive Bayes classification program using for rating the novels.
(2) Classification.py : This is a Naive Bayes classification program using for rating the novels.
(3) Classifier.py : This is the main program of our novel classifier including rating program and the update program for word probability list.
(3) Classifier.py : This is the main program of our novel classifier including rating program and the update program for word probability list.
3. Steps to run this novel classifer：
3. Steps to run this novel classifer：
(1) Run the Get_Words_Probability.py to set up word probability list of different level novels. In this classfier, all novels are divided into two levels: "G" for genaral and "A" for adults only.The more novels to learn, the more accurate probability to get.There are two word probability list samples which can be used when running the model.
(1) Run the Get_Words_Probability.py to set up word probability list of different level novels. In this classfier, all novels are divided into two levels: "G" for genaral and "A" for adults only.The more novels to learn, the more accurate probability to get.There are two word probability list samples which can be used when running the model.
(2) Create a file folder to save all the novels that waiting to get the rating.
(2) Create a file folder to save all the novels that waiting to get the rating.
(3) Run the Classifier.py based on the path of folder created in step (2). When finish rating process, the word probability list will also update.
(3) Run the Classifier.py based on the path of folder created in step (2). When finish rating process, the word probability list will also update.
Google Sites
Report abuse