Module 10 · Capstone Project Milestones ⏱ 17 min

Milestone 3: Statistical Analysis

By the end of this lesson you will be able to:
  • Compute the average word length with a generator expression
  • Build a word-length distribution with collections.Counter
  • Guard against division by zero on empty input

A frequency table tells you which words appear. Statistics tell you about the text itself: how difficult it reads, how long its words tend to be, whether the vocabulary is simple or dense. Two numbers carry most of that signal: the average word length, and the distribution of word lengths across the whole sample.

This milestone computes both, then folds them with a sentence count into a rough reading level. Together they turn a pile of tokens into a characterisation of the writing. The math is simple, but the discipline of computing it from clean tokens rather than raw characters, and of surviving empty input without crashing, is what makes the functions trustworthy enough to build on.

flowchart LR
  W["words"] --> M["length of each word"]
  M --> A["average length"]
  M --> D["length distribution"]
  style A fill:#3776ab,color:#fff
  style D fill:#3776ab,color:#fff
Each word's length feeds both the average and the length distribution.

Average word length

The average word length is the mean number of characters per word. Compute it in one expression: sum every word's length, then divide by how many words there are.

avg = sum(len(w) for w in words) / len(words)

The piece inside sum is a generator expression: len(w) for w in words yields the length of each word one at a time, and sum adds them up without ever building an intermediate list. That is the comprehension syntax you already know, minus the square brackets, dropped straight inside a function call.

The result is a float, even when the numbers divide evenly, because / always returns a float in Python 3. If you need a whole number for display, wrap it in round() rather than truncating with int().

Sum the lengths, divide by the count, and round only for display.
words = ['the', 'cat', 'sat']
total = sum(len(w) for w in words)
avg = total / len(words)
print('total letters:', total)
print('average length:', avg)
print('rounded:', round(avg))

The length distribution

An average hides shape. Five short words and five long ones can share an average with ten medium words, yet they read very differently. A distribution shows how many words there are of each length, and Counter builds it in one call.

from collections import Counter
lengths = Counter(len(w) for w in words)

Here the generator expression yields the length of each word rather than the word itself, so Counter tallies lengths: how many one-letter words, how many two-letter words, and so on. Reading lengths[3] tells you how many words are exactly three characters long.

Counter over word lengths gives a histogram of how long the words are.
from collections import Counter

words = ['a', 'be', 'cat', 'do', 'egg']
lengths = Counter(len(w) for w in words)
print('two-letter words:', lengths[2])
print('three-letter words:', lengths[3])
print('one-letter words:', lengths[1])
print('distinct lengths:', len(lengths))
flowchart TD
  I["words list"] --> Q{"empty?"}
  Q -->|"yes"| Z["return 0<br/>avoid ZeroDivisionError"]
  Q -->|"no"| C["compute average"]
  style Z fill:#b45309,color:#fff
  style C fill:#3776ab,color:#fff
An empty word list must be caught before the division, or the call crashes.

The trap: dividing by zero

Empty input breaks the naive average. When words is empty, len(words) is 0, and sum(...) / 0 raises ZeroDivisionError. The function looked correct on every normal sentence, then crashed the moment it met a blank field.

The fix is to guard the empty case first, before any division:

def average_word_length(words):
    if not words:
        return 0
    return sum(len(w) for w in words) / len(words)

Returning 0 for empty input is a deliberate choice: it gives callers a safe number instead of an exception. The wider habit is validate at the boundary, handle the degenerate case the moment data enters a function, so the rest of the body can assume its input is well-formed.

Two small statistics: an average that survives empty input, and a length histogram.
from collections import Counter

def average_word_length(words):
    if not words:
        return 0
    return sum(len(w) for w in words) / len(words)

def length_distribution(words):
    return Counter(len(w) for w in words)

sample = ['a', 'be', 'cat', 'do']
print('average:', average_word_length(sample))
print('distribution:', dict(length_distribution(sample)))
print('empty case:', average_word_length([]))
flowchart LR
  T["text"] --> S["count sentences"]
  T --> W["average word length"]
  S --> R["reading level label"]
  W --> R
  style R fill:#3776ab,color:#fff
A reading level combines sentence count and word length into one label.

From numbers to a reading level

A single average hides whether the text leans short and simple or long and technical. The reading-level heuristic here leans on a plain observation: longer words, on average, mark denser writing. Four characters per word is a rough watershed; below it the prose reads plainly, above it the vocabulary grows demanding.

No single number captures readability, and this rule ignores sentence length, syllables, and grammar entirely. Real readability formulas, like Flesch-Kincaid, combine several of those signals. What this milestone gives you is the shape of an analysis: take clean tokens, compute summary numbers, and combine them into a judgment a reader can act on. The same shape scales to far richer metrics once you add more signals.

Exercise

Write average_word_length(words) that returns the average length of the words in the list. Return 0 if the list is empty, so it never crashes on division by zero.

def average_word_length(words):
    # guard the empty case, then divide total letters by the word count
    pass
Exercise

Write length_distribution(words) that returns a collections.Counter mapping each word length to how many words have that length. Import Counter from collections.

from collections import Counter

def length_distribution(words):
    # count how many words have each length
    pass
Exercise

What does this print? It averages the two word lengths.

words = ["cat", "dogs"]
print(sum(len(w) for w in words) / len(words))
Exercise

This average_word_length crashes on an empty list with ZeroDivisionError, because it divides before checking the length. Add a guard so it returns 0 for an empty list instead of crashing.

def average_word_length(words):
    return sum(len(w) for w in words) / len(words)   # bug: crashes on []
Exercise

Combine the pieces into a reading-level estimate. Write reading_level(text) that splits text into words with .split(), returns "easy" if there are no words, and otherwise returns "hard" if the average word length is 4.0 or greater, else "easy". So reading_level("a be cat") returns "easy" (avg 2.0) and reading_level("elephant dinosaur") returns "hard" (avg 8.0).

def reading_level(text):
    # split, compute average word length, then classify
    pass

Recap

  • Average word length is sum(len(w) for w in words) / len(words), a float.
  • A generator expression inside sum avoids building an intermediate list.
  • Counter(len(w) for w in words) gives the length distribution.
  • Always guard the empty case before dividing, or ZeroDivisionError crashes the call.
  • Derive every statistic from the token list, never from the raw string.

Milestone 4 moves from counting characters to extracting structure, pulling emails, dates, and entities out of text with regular expressions.

Checkpoint quiz

What does sum(len(w) for w in ['a', 'bb', 'ccc']) return?

Why must you guard the empty list before computing sum(...) / len(words)?

Go deeper — technical resources