what ARE Decision trees? 

Decision trees are one of the most commonly used supervised machine learning algorithms used in the industry. It can be employed for both classification and regression tasks. The main reason for employing Decision trees for classification tasks and regression tasks is due to its simplicity, interpretability, and ability to handle both numerical and categorical data.  Unlike Naive Bayes and other classifiers, Decision Trees are not linear classifiers/separators. Another important point to note is that Decision Trees come under the category of non-parametric models, i.e. they can't be expressed as parametric equations, unlike linear regression. 

STRUCTURE OF DECISION TREE: 

In a decision tree structure, there are 3 different nodes.

1. Root Node

2. Parent Node/ Internal Node

3. Child Node/Terminal Node

The root node is the top node and is also called the base node.  It doesn't have a parent node/incoming node but has zero or multiple outgoing nodes. The node that has both incoming nodes and outgoing nodes is called the parent/internal node. The node that has only the incoming nodes and no outgoing nodes is called the leaf node/terminal node. 

APPLICATIONS OF DECISION TREE: 

Decision trees can be utilized for various purposes across multiple fields. Some of the common applications of Decision Trees include:

• Fraud detection

• Customer churn prediction

• Medical diagnosis

• Product recommendation

GINI, ENTROPY, Information Gain

GINI: 

The GINI index measures the impurity of a set of data points. It ranges from 0 to 1, where 0 indicates that all elements belong to a single class and 1 indicates maximum impurity (i.e., equal distribution among all classes).

Information Gain: 

Information gain measures the reduction in entropy or GINI index achieved by splitting a dataset based on a particular feature. It helps in selecting the best feature to split the data. The feature selection for node splitting is based on Information gain. If the information gain is high, then those features are selected for node splitting compared to the features that produce low information gain.

Illustration of decision trees 

Consider this simple dataset, which illustrates how decision trees are executed. Decision trees are operated in a similar way to how the human brains think. It looks for a pattern and divides the data into various partitions. In this example dataset, there are four columns

• Role

• City

• Education Level

• Salary > 100K

Creating a decision tree from a group of attributes can be done in various manners. However, determining the most effective decision tree is a highly demanding and challenging task due to the exponential growth of the search space, making it time-consuming and complex. 

To discover optimal trees, heuristic techniques such as Hunt's Algorithm are utilized, aiding in generating locally optimized solutions. 

A root node for the optimal tree is selected in such a way that maximum information gain is obtained. In this example lets assume the role is selected as root node then its divided into:

For instance, one of these parent nodes might inquire about the individual's education level, asking whether it's at the bachelor's level or not. Depending on the response, the tree further splits into leaf nodes, which serve as the endpoints of the tree. These leaf nodes provide the final classification, such as whether the salary is higher than 100K or not based on the criteria set by the parent nodes. This is an example of a decision tree with depth 3, ie the number of layers of the decision tree is three.

Pros:

• It is easy to understand and interpret decision trees.

• Decision trees can handle both categorical and continuous data and it is a nonparametric model which means there are no assumptions about the distribution of data.

Cons:

• Decision trees cannot be used in case of complex relationships between the features.

• Decision trees tend to overfit and cannot produce results effectively in case of class imbalance.