In the machine learning decision tree, there are three most commonly used algorithms: ID3, C4.5, CART. Here I will talk about my understanding of the ID3 algorithm.

1. Definition

The core of the ID3 algorithm is to apply information gain criteria to select features on each node of the decision tree, and build the decision tree recursively.
The ID in the ID3 name is the abbreviation of It-erative Dichotomiser.

2. Understanding

This tree-building algorithm uses information gain (not information gain rate!) to perform feature selection, and information gain is obtained by subtracting two entropies, which is the entropy after feature splitting minus before splitting Entropy of the node.
What is entropy? Entropy is a kind of uncertainty. At the beginning, the entropy of your data must be the largest. The purpose of your establishment is to reduce the uncertainty of your data step by step. The land is reduced to make the data in a node more pure (that is, we said that the classification is good).
When the current entropy is given, what we have to do is to make the split entropy (entropy(D,A)) as small as possible.

3. The process of ID3 algorithm

The data processed by the ID3 algorithm is discrete, and its basic algorithm process is as follows:

Create Root node
         If Examples are all positive,
                 Then return label=Positive single node number Root
         If Examples are all subs,
                 Then return the number of single nodes of label= Vice Root
         If Attribute is empty,
                 Then return the value of the most common Target_attribute in Root,label=Example.
         Otherwise
                 A←Atrribute the attribute with the best ability to classify Eaxmples (*)
                 Root's decision attributes←A
                 For every possible value of A Vi
                         Add a new branch under Root to test A=Vi
                         Let Examplevi be a subset of Examples satisfying the attribute value of A as Vi
                         If Example is empty
                                 Under the new branch, add a leaf node,
                                 Node label=the most common Target_attribute value in the example.
                         Otherwise
                                 Under the new branch, add a subtree of ID3 (including all satisfied examples, all labels, but the feature number -1).
                 End 
         Return to Root

Starting from the root node, calculate the information gain of all possible features for the node, select the feature with the largest information gain as the feature of the node, and establish the child node from the different values ​​of the node; then recursively calculate the child node Call the above methods to build a decision tree; until the information gain of all features is small or there are no features to choose from. Finally, a decision tree is obtained.

3. The advantages and disadvantages of ID3 algorithm

advantage

• 1. Assuming that the space contains all decision trees, the search space is complete.
• 2. Good robustness and not affected by noise.
• 3. The instances with missing attribute values ​​can be trained.

In general, the theory is clear, the method is simple, and the learning ability is strong

Disadvantage

The shortcomings of the ID3 algorithm are also obvious. As we said above, the ID3 algorithm will select features with many subcategories, because the result of this split will be purer and the entropy will be smaller. This is biased to our original intention. The pureness is not the pureness that is obtained by making it more detailed! If so, it's better to divide it into 100 categories. The purity of the data is very high.
So I summarized some shortcomings of the ID3 algorithm:

• 1. ID3 only considers sub-type features, but does not consider continuous features, such as length and density are continuous values, which cannot be used in ID3. This greatly limits the use of ID3.
• 2. The ID3 algorithm does not consider missing values.
• 3. The problem of overfitting is not considered.
• 4. The ID3 algorithm uses information gain as the evaluation criterion when selecting the branch attributes in the root node and each internal node. The disadvantage of information gain is that it tends to choose attributes with more values. In some cases, such attributes may not provide much valuable information.
• 5. The division process will stop due to insufficient statistical characteristics due to the small subset size.

4. Why tend to feature more feature options

In the ID3 algorithm, generally speaking, we make judgments based on information gain. The greater the information gain, the greater the "purity improvement" obtained by using the attribute a for division.

In information gain, there is also a concept of "information entropy".
Assume that the current sample set D $k$kThe proportion of class samples is $p_k(k=1,2…，|γ|)$pk​(k=1,2…，∣γ∣), Then the information entropy of D is defined as:

The formula for information gain is:

Because the number of samples contained in different branch nodes is different, the weight assigned to the branch node, that is, the more the number of samples, the greater the influence of the branch node.
So, ID3 algorithm will tend to feature more feature options