Section 2
Random Forests
6. Ensemble Methods: Machine Learning and Democracy
4:57 (Preview)
7. Random Forests: Decisions Don't Fall Far from the Decision Tree 📂
8. Random Forests out in the Wild 📂
9. Interpolation Through a Random Forest 📂
Section 3
Gradient Boosting
10. Give Yourself a Gradient Boost
07:01 (Preview)
11. Auto-Correction in a Forest of Stumps 📂
12. Gradient Boosting by Hand: Code Example 📂
13. XGBoost in the Wild 📂
14. Cross validate with the XGBoost API 📂
15. Conclusion, Certificate, and What Next?
4. Classification Problems with Decision Trees: A Code Walkthrough
Decision Trees
📂 Please register or log in to download resources

In this code walk-through, we explore the algorithm for decision tree learning using the scikit-learn implementation. Experimenting on a visual classification problem, we explore how to interpret the models results to explain the predictions.

📑 Learning Objectives
  • Deploy a decision Tree Classifier using scikit-learn.
  • See first hand the sensitivity of a decision tree to over-fitting.
  • Make good hyper parameter selections to deploy the algorithm.
  • Explore and explain the models results graphically.

Constructing a graphical classification problem

For this example, we are leaning into our favourite 2D demonstrator: the blobs. Let's begin by importing the data.

def blobs(num_samples, std_dev, seed):
    """Generate two 2D normal distributions in the NE and SW quadrants."""
    cov = np.asarray([[std_dev, 0], [0, std_dev]])
    mean_ne = np.asarray([2., 2.5])
    mean_se = np.asarray([2.5, 1.5])
    mean_sw = np.asarray([1.5, 1.5])
    ne = np.random.multivariate_normal(mean=mean_ne, cov=cov, size=num_samples)
    se = np.random.multivariate_normal(mean=mean_se, cov=cov, size=num_samples)
    sw = np.random.multivariate_normal(mean=mean_sw, cov=cov, size=num_samples)
    return ne, se, sw

As before, sampling from this data generator gives us the following dataset:

Classification Problem

Figure 1. Sample data.

Let's try and solve a more dificult problem--a classification problem--using the power of machine learning to optimise our decision tree splits. This shall lead us to an algorithm, and hence a model, that can be deployed on general datasets.

For this exampole, here's the data: We collect 300 samples, split equally amongst three generic classes: 'Gold', 'Blue' and 'Pink'.

Classification Problem

Our next step is to preprocess the data in the scikit-learn format:

from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split

NOISE = 0.12
ne, se, sw = blobs(NUM_SAMPLES, NOISE, 31)

# Organise the data
X = np.concatenate((ne, se, sw), axis=0)
y = np.asarray(len(ne)*[-1,] + len(se)*[1,] + len(sw)*[0,])

# Randomly order the data, for good measure
X_train, y_train = shuffle(X, y, random_state=31)

Then we are ready to train our decision tree.

from sklearn.tree import DecisionTreeClassifier


model = DecisionTreeClassifier(
    random_state=SEED, max_leaf_nodes=MAX_LEAF_NODES,
), y_train)

Of upmost importantly, we wish to interrogate and interpret the model. The tree structure lends itself to this analysis. To print out the graph of the tree we use the export_text utility.

from sklearn.tree import export_text

read_out = export_text(model)
|--- feature_1 <= 2.15
|   |--- feature_0 <= 2.06
|   |   |--- class: 0
|   |--- feature_0 >  2.06
|   |   |--- class: 1
|--- feature_1 >  2.15
|   |--- class: -1

To visualise this same graph we can use graphviz.

import graphviz
from sklearn.tree import export_graphviz

gv = export_graphviz(model, out_file=None)
graph = graphviz.Source(gv)
graph.render('images1/blobs-graph3', format='png')
Three-node graph

Figure 2. Output of `graphviz`.

Finally, to demostrace the decision boundaries, we plot the contours of the model.predict function.

x = np.linspace(AX_MIN, AX_MAX, 1000)
y = np.linspace(AX_MIN, AX_MAX, 1000)
xx, yy = np.meshgrid(x, y)

# Predict contours
f = model.predict(np.c_[xx.ravel(), yy.ravel()])
ff = f.reshape(xx.shape)
from matplotlib import cm

# x-axis
x0 = np.linspace(LINE_MIN, LINE_MAX, 1000)

# Plot data and ground truth
fig, ax = plt.subplots(1, 1, figsize=[6, 6])
plt.subplots_adjust(wspace=SPACE, hspace=SPACE)

# Axes
ax.set_xlim(AX_MIN, AX_MAX)
ax.set_ylim(AX_MIN, AX_MAX)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)

# Plot data
gold = ax.scatter(ne[:, 0], ne[:, 1], s=10, color='goldenrod')
blue = ax.scatter(se[:, 0], se[:, 1], s=10, color='navy')
pink = ax.scatter(sw[:, 0], sw[:, 1], s=10, color='lightpink')
# gt, = ax.plot(x0, -x0 + 4, color='lightcoral', linestyle='--', linewidth=2)

# Plot contour of predictive model
    xx, yy, ff, cmap=cm.get_cmap("magma_r"), alpha=0.5, linestyles=["-"],

    f'Decision tree of depth {model.get_depth()} '
    f'with {model.tree_.node_count} nodes')

    [gold, blue, pink],
    ['Gold', 'Blue', 'Pink'],
    loc='upper left',
Three-split data

Figure 3. Coutours of the decision tree predictor.

Next Lesson
5. Regression Problems with Decision Trees: A Code Walkthrough