Module 10 · Capstone Project Milestones ⏱ 17 min

Milestone 1: Text Ingestion & Cleaning

By the end of this lesson you will be able to:
  • Normalize text by folding every character to lowercase
  • Strip punctuation from a string with str.translate and str.maketrans
  • Avoid the str.strip trap that only trims the two ends of a string

Real text is messy. Take the sentence The cat chased the CAT, and the cat won. A human sees the word the three times and cat three times. A naive program that splits on whitespace sees six different tokens: The, cat, the, CAT,, the, and cat. Capital letters and stray punctuation hide the fact that these are the same words repeated.

Before you can count words, measure their lengths, or rank the most common one, you must clean the text: fold every letter to one case, and strip the punctuation that clings to words. This milestone builds the two cleaning functions the whole project depends on. Skip this step and your word counts are wrong, your frequency table fills with near-duplicates, and every later statistic inherits the error.

flowchart LR
  T["raw text"] --> L["lowercase"]
  L --> P["strip punctuation"]
  P --> C["clean tokens"]
  style P fill:#3776ab,color:#fff
  style C fill:#3776ab,color:#fff
The cleaning pipeline: fold case, then delete punctuation, leaving uniform tokens.

Normalize the case

The first cleaning pass is the simplest: convert every letter to lowercase with text.lower(). Once The, THE, and the all collapse to the, they are indistinguishable to the rest of the program, and your counts start meaning what a reader expects.

Two details catch beginners. First, lower() returns a new string; it never edits the original, because Python strings are immutable. That is why the result is almost always reassigned: text = text.lower(). Calling the method and throwing away its return value is a silent no-op that leaves every capital exactly where it was. Second, lowercase is the cleaning convention rather than title() or upper(), because lowercasing is the operation that makes the most variants agree.

flowchart LR
  A["The"] --> L["the"]
  B["THE"] --> L
  C["the"] --> L
  L --> D["one token"]
  style L fill:#3776ab,color:#fff
Three case variants all fold down to a single token after lowercasing.
lower() returns a cleaned copy; the original string is never modified.
raw = "The CAT and the Cat"
print("before:", raw)

clean = raw.lower()
print("after:", clean)

# lower() returns a new string; the original is never changed
print("raw:", raw)

Strip the punctuation

Punctuation is the second source of phantom duplicates. cat, cat., and (cat) are the same word once you delete the marks around them.

The right tool is str.translate, fed a table built by str.maketrans. Hand maketrans('', '', string.punctuation) three arguments: the characters to replace from (empty), the characters to replace to (empty), and the set to delete. The result is a table whose only effect is to drop every punctuation mark. string.punctuation is a ready-made string of 32 characters, the period, comma, both quote marks, every bracket, the apostrophe, and the hyphen, so one call sweeps them all out:

import string
clean = text.translate(str.maketrans('', '', string.punctuation))

That breadth is the point. It is also a source of tradeoffs, because contractions and compound words do not survive a full strip unchanged.

translate with a maketrans delete-table strips every punctuation mark at once.
import string

raw = "Hello, world! Wait..."
clean = raw.translate(str.maketrans('', '', string.punctuation))
print(clean)

Order matters, and so do the tradeoffs

Cleaning is a pipeline, and the order of its stages is deliberate: lowercase first, then strip punctuation. Either order gives the same result here, because case-folding never creates or removes punctuation, but building the habit of normalize, then strip keeps the pipeline predictable as it grows into more stages.

One tradeoff is worth naming now. Removing every punctuation mark also removes apostrophes and hyphens, so don't becomes dont and state-of-the-art becomes stateoftheart. For a first analyzer this is an acceptable simplification: the counts stay consistent, which is what matters. If you later need to keep contractions and compounds intact, you will reach for regular expressions, the topic of Milestone 4, which can split on word boundaries instead of deleting characters.

A reusable clean_text helper: lowercase and strip in one expression.
import string

def clean_text(text):
    return text.lower().translate(str.maketrans('', '', string.punctuation))

print(clean_text("Hello, World!"))
print(clean_text("The CAT; the dog."))

A worked example

Trace one sentence end to end. Start with the raw string The CAT, the cat!:

  1. After lowercase: the cat, the cat!
  2. After strip punctuation: the cat the cat
  3. After split: ['the', 'cat', 'the', 'cat']

Four tokens appear where the raw string hid two words behind capitals and marks. The comma and the exclamation vanished together in a single translate pass, not a chain of replace calls. Every later milestone feeds output like this into its own function, so the cleaning is written and tested exactly once.

flowchart LR
  subgraph S["strip: edges only"]
    A1["hi, there!"] --> A2["hi, there"]
  end
  subgraph T["translate: every char"]
    B1["hi, there!"] --> B2["hi there"]
  end
  style A2 fill:#b45309,color:#fff
  style B2 fill:#3776ab,color:#fff
str.strip removes characters only at the two ends; str.translate removes them everywhere.
Exercise

Write normalize(text) that returns the text converted entirely to lowercase.

def normalize(text):
    # return the lowercased text
    pass
Exercise

Write remove_punctuation(text) that deletes every punctuation character from text. Use str.translate with a delete-table built by str.maketrans('', '', string.punctuation). Remember to import string.

import string

def remove_punctuation(text):
    # build a delete-table with str.maketrans, then call text.translate
    pass
Exercise

What does this print? The text is lowercased, then every punctuation mark is deleted.

import string

s = "Hello, World!"
print(s.lower().translate(str.maketrans('', '', string.punctuation)))
Exercise

This remove_punctuation is meant to delete every punctuation mark, but it only trims the two ends, so interior punctuation survives. Replace strip with the translate + maketrans approach so marks are removed everywhere.

import string

def remove_punctuation(text):
    return text.strip(string.punctuation)   # bug: edges only
Exercise

Combine the two passes into a tokenizer. Write clean_words(text) that lowercases the text, strips all punctuation, and returns the result split into a list of words using .split() with no arguments. So clean_words("Hello, World!") returns ["hello", "world"].

import string

def clean_words(text):
    # lowercase, then strip punctuation, then split into words
    pass

Recap

  • Real text mixes case and punctuation; clean it first, or every later count is wrong.
  • text.lower() returns a new lowercased string, so reassign the result or nothing changes.
  • str.translate(str.maketrans('', '', string.punctuation)) removes every punctuation mark, anywhere in the string.
  • str.strip only trims the two ends; never use it for whole-string cleaning.
  • Lowercase first, then strip punctuation, then split, as one predictable pipeline.

Milestone 2 takes this cleaned text, breaks it into tokens, and counts how often each one appears.

Checkpoint quiz

Why does calling text.lower() and ignoring its return value do nothing?

What does 'a.b.c'.strip(string.punctuation) return?

Go deeper — technical resources