How a Decision Tree Thinks

A linear model draws one straight line through the data and is done. But a lot of real decisions are not lines — they are chains of questions. Is the customer over thirty? Then do they earn more than fifty thousand? Then have they bought from us before? Each question narrows the possibilities, and the answer routes you to the next question, until you finally land on a verdict. That chain of yes/no questions is a decision tree, and it is the most human-readable model in all of machine learning. Where a neural network is a black box, a tree is a list of questions you can read out loud and argue with. This lesson does not build one yet — it teaches you how to read one, because reading is the skill the next two lessons depend on.

flowchart TD
  R["hours studied <= 5 ?"] -->|"yes"| A["predict fail"]
  R -->|"no"| B["past score <= 80 ?"]
  B -->|"yes"| C["predict fail"]
  B -->|"no"| D["predict pass"]
  style R fill:#3776ab,color:#fff
  style D fill:#166534,color:#fff

A tiny decision tree: each box asks one feature-vs-threshold question; the arrows route an example left (yes) or right (no) until it hits a leaf with a prediction.

The six words you need

A tree has a small, fixed vocabulary, and learning it is what makes every later diagram readable. The root is the first question, where every example begins. An internal node is any question below the root. A branch is the edge taken for one answer, and a leaf is a terminal node that holds the prediction — a class label like pass, or a number to return. Every internal node carries two things: a feature to test, and a threshold to compare it against. The comparison is always is this feature less than or equal to the threshold? — yes sends the example down the left branch, no down the right. Root, node, branch, leaf, feature, threshold: six words, and you can now read any tree ever drawn.

A tree as nested dictionaries, plus the function that routes an example through it. Press Run.

# feature 0 = hours studied, feature 1 = past score
tree = {
    'feature': 0,
    'threshold': 5,
    'left': {'leaf': 'fail'},
    'right': {
        'feature': 1,
        'threshold': 80,
        'left': {'leaf': 'fail'},
        'right': {'leaf': 'pass'}
    }
}

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[tree['feature']] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

print(classify(tree, [3, 90]))

How routing works

Run that code and classify(tree, [3, 90]) returns fail. Trace it. The example has feature 0 equal to 3, which is less than or equal to the root threshold 5, so the comparison says yes and we go left — straight into a leaf that predicts fail. Notice the example never even reaches the second question about the past score. That is the whole personality of a tree: it asks the minimum number of questions, and different examples follow different paths of different lengths. The classify function is just that walk, written as recursion: at each node, either you are at a leaf (return its prediction) or you pick a branch and walk into the subtree on that side.

flowchart LR
  S["whole dataset"] -->|"best question"| N["a node"]
  N -->|"left"| L["recurse here"]
  N -->|"right"| R["recurse here"]
  L --> LL["pure leaf"]
  R --> RR["pure leaf"]
  style N fill:#3776ab,color:#fff

Recursive partitioning: the whole dataset is split by the best question, then each half is split again, until every group is pure enough to become a leaf.

Different examples take different paths and different numbers of steps. Run it.

tree = {
    'feature': 0,
    'threshold': 5,
    'left': {'leaf': 'fail'},
    'right': {
        'feature': 1,
        'threshold': 80,
        'left': {'leaf': 'fail'},
        'right': {'leaf': 'pass'}
    }
}

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[tree['feature']] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

examples = [[3, 90], [7, 70], [7, 95], [2, 10]]
for row in examples:
    print(row, '->', classify(tree, row))

Why trees are not lines

The examples [7, 70] and [7, 95] share the same hours studied, yet the tree predicts fail for one and pass for the other. A single straight line could never do that, because a line weighs every feature once, everywhere. A tree weighs features locally: it can ask about the past score only inside the region where hours already exceed five. This is what makes trees so flexible, and it is also what makes them dangerous — a tree flexible enough to capture any pattern is flexible enough to memorise the noise, which is the trap you will build and break two lessons from now. For now, hold on to the key image: a tree carves the input space into nested boxes, one question at a time, and each box ends in a leaf.

What a leaf actually holds

A leaf is the end of the road, and it stores no further questions — only a prediction. For a classification tree that prediction is a class label, chosen as the majority class of the training examples that wound up in that leaf during learning. If ninety pass examples and ten fail examples all land in one leaf, that leaf predicts pass, full stop, no matter how close the vote was. This is worth pausing on: every example routed to the same leaf gets the same answer, even if their feature values differ. The tree has already done its discriminating work higher up, in the questions; the leaf just reports the consensus of whoever arrived. That is also why the size of a leaf matters — a leaf built from one training example is memorising, and a leaf built from a hundred is generalising.

Watch the decision path unfold: each step prints the question, the answer, and where it sends the example.

tree = {
    'feature': 0,
    'threshold': 5,
    'left': {'leaf': 'fail'},
    'right': {
        'feature': 1,
        'threshold': 80,
        'left': {'leaf': 'fail'},
        'right': {'leaf': 'pass'}
    }
}

def trace(tree, row, depth=0):
    pad = '  ' * depth
    if 'leaf' in tree:
        print(pad + 'leaf predicts ' + tree['leaf'])
        return tree['leaf']
    feat = tree['feature']
    thr = tree['threshold']
    val = row[feat]
    go_left = val <= thr
    print(pad + 'feature ' + str(feat) + ' = ' + str(val) + ', threshold ' + str(thr) + ' -> ' + ('left' if go_left else 'right'))
    nxt = tree['left'] if go_left else tree['right']
    return trace(nxt, row, depth + 1)

trace(tree, [7, 95])

flowchart TD
  START["new example to classify"] --> R["hours less than or equal to 5"]
  R -->|"no, go right"| B["past score less than or equal to 80"]
  B -->|"yes, go left"| C["predict fail"]
  style START fill:#b45309,color:#fff
  style C fill:#166534,color:#fff

Tracing one example: each question is answered, the matching branch is taken, and the walk continues until a leaf returns the prediction.

Exercise

What does this print? Follow the comparisons: feature 0 against threshold 5, then feature 1 against threshold 80.

tree = {
    'feature': 0,
    'threshold': 5,
    'left': {'leaf': 'fail'},
    'right': {
        'feature': 1,
        'threshold': 80,
        'left': {'leaf': 'fail'},
        'right': {'leaf': 'pass'}
    }
}

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[tree['feature']] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

print(classify(tree, [7, 70]))

Your prediction — what does this print?

Exercise

An example reaches an internal node during classification. What must that node contain for routing to continue?

Exercise

Write classify(tree, row) that walks the tree to a leaf. A leaf is a dict containing the key 'leaf'; any other node has 'feature', 'threshold', 'left', and 'right'. Go left when row[feature] <= threshold, otherwise right.

def classify(tree, row):
    # if this node is a leaf, return its prediction
    # otherwise compare row[feature] to threshold and recurse
    pass

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[tree['feature']] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

Exercise

This classify ignores the feature the node actually asks about and always tests feature 0 — so it routes examples by the wrong column whenever a node splits on a different feature. Read the feature out of the node instead of hard-coding the index.

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[0] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

def classify(tree, row):
    if 'leaf' in tree:
        return tree['leaf']
    if row[tree['feature']] <= tree['threshold']:
        return classify(tree['left'], row)
    return classify(tree['right'], row)

Exercise

Write count_leaves(tree) that returns how many leaves a tree has. A leaf is a dict containing 'leaf'; an internal node has 'left' and 'right' subtrees. Use recursion: a leaf counts as 1, and an internal node's count is the sum of its two subtrees.

def count_leaves(tree):
    # a leaf is one leaf; otherwise add the two subtrees
    pass

def count_leaves(tree):
    if 'leaf' in tree:
        return 1
    return count_leaves(tree['left']) + count_leaves(tree['right'])

Recap

A decision tree is a chain of yes/no questions; each internal node tests one feature against one threshold, and the answer routes the example left (<=) or right (>).
The six parts to name are root, internal node, branch, leaf, feature, threshold. A leaf holds the prediction; everything else routes.
Routing is recursion: at a leaf return its label, otherwise pick a branch by comparing row[feature] to the threshold and walk into that subtree.
Each node tests its own feature — read tree['feature'] every time, never a hard-coded index, or your classifier routes by the wrong column.
A tree carves the input into nested boxes, asking the minimum questions per example — flexible, but that same flexibility is what lets a deep tree memorise noise.

You can now read a tree. The next two lessons build one: first the split criterion that decides each question, then the recursive loop that grows the whole structure.

The six words you need

How routing works

Why trees are not lines

What a leaf actually holds

Recap

Checkpoint quiz

Go deeper — technical resources