Embedded Files
Useful statistical methods tend to produce results that are hard to draw insight from.
Useful statistical methods tend to produce results that are hard to draw insight from.
For example, Principal Component Analysis is a way to reduce the complexity in a data set, yet it is difficult to derive meaning from the analysis because you have interpret coefficient values. Other methods, such as Random Forests and Support Vector Machines, have good performance and, while they have cool sounding names, are difficult to interpret.
For example, Principal Component Analysis is a way to reduce the complexity in a data set, yet it is difficult to derive meaning from the analysis because you have interpret coefficient values. Other methods, such as Random Forests and Support Vector Machines, have good performance and, while they have cool sounding names, are difficult to interpret.
In contrast, a human-readable model is something like linear-regression or a decision tree. Both of these types are easy for the human brain to read and interpret quickly. When it comes to conveying a narrative to a third person this is important. In fact, there is a growing argument that these simple linear models can outperform more complex machine learning / AI driven algorithms.
In contrast, a human-readable model is something like linear-regression or a decision tree. Both of these types are easy for the human brain to read and interpret quickly. When it comes to conveying a narrative to a third person this is important. In fact, there is a growing argument that these simple linear models can outperform more complex machine learning / AI driven algorithms.
The model we have developed is based on a decision tree. These are good at classifying data problems, which is at the heart of what we do when we analyze data. To remove bias we use a method called k-fold cross-validation.
The model we have developed is based on a decision tree. These are good at classifying data problems, which is at the heart of what we do when we analyze data. To remove bias we use a method called k-fold cross-validation.
Cross-validation is a technique that ensure we don't build a model which is perfect for one data set but wildly inaccurate for a different data set.
Cross-validation is a technique that ensure we don't build a model which is perfect for one data set but wildly inaccurate for a different data set.
K-fold means we use K subsets of the data. You can set K to be 0 or 10. 10 is the better option since it creates 10 models and then averages the results of all 10. Each of these models is tested on a unique 10% of the data and trained on the remaining 90%. Every row of data in the data set gets to be in 9 training sets and one test set.
K-fold means we use K subsets of the data. You can set K to be 0 or 10. 10 is the better option since it creates 10 models and then averages the results of all 10. Each of these models is tested on a unique 10% of the data and trained on the remaining 90%. Every row of data in the data set gets to be in 9 training sets and one test set.
Learn more in this wikipedia entry.
Report abuse