Regular Expressions (re)

Sooner or later every program has to wrestle with text: pull a price out of a sentence, check whether a string looks like an email, or replace every date in a document. Doing this with string methods alone means nested loops, fragile index arithmetic, and code that collapses the moment the input changes slightly.

Regular expressions — regex for short — are a miniature language for describing patterns in text. Instead of saying 'find the colon, then move forward three characters', you write a pattern that says 'a dollar sign followed by one or more digits'. The re module compiles that description into an efficient search engine that runs over your text and hands back every match.

Regex is not always the right tool — for simple fixed replacements, ordinary string methods are clearer — but once a pattern involves choice, repetition, or position, a regex is usually shorter, faster, and far easier to maintain.

flowchart LR
  P["Pattern\n#quot;\d+#quot;"] --> C["Compile"]
  C --> E["Engine"]
  T["Text"] --> E
  E --> M["Match object\nor None"]
  style E fill:#3776ab,color:#fff

A pattern is compiled into an engine that scans text and returns match objects.

Literals and metacharacters

A pattern is built from two kinds of pieces. Literals match themselves exactly: the pattern cat finds the letters c-a-t in that order. Metacharacters are special symbols that stand for whole classes of characters or control how many times something may appear.

Here are the essentials:

. matches any single character except a newline.
\d matches any digit (0-9).
\w matches any word character: letters, digits, or underscore.
+ means 'one or more' of the thing before it.
* means 'zero or more'.
? means 'zero or one' — the thing is optional.
Square brackets define a character class: [aeiou] matches any one vowel.

re.search(pattern, text) scans through the text and returns the first match, or None if nothing fits. re.findall(pattern, text) returns every non-overlapping match as a list of strings. Both are the workhorses you will reach for most often.

Find all digits and the first word with metacharacters.

import re

text = 'Prices: $12, $5, and $100'

# Find all dollar amounts
amounts = re.findall(r'\$\d+', text)
print(amounts)

# Search for the first number
m = re.search(r'\d+', text)
print(m.group() if m else 'none')

Reading a pattern left to right

A regex is read in the same direction as the text it scans. \d+ means 'one digit, then keep taking digits as long as they appear'. \w+@\w+\.\w+ is a naive email pattern: word characters, an @, word characters, a literal dot, word characters. It is not production-grade, but it demonstrates how each symbol maps onto a position in the string.

Repetition is greedy by default: + and * grab as many characters as they can while still allowing the rest of the pattern to match. That behaviour is the source of the classic bug you will fix in the exercises below. When you need the shortest possible match instead, add a ? after the quantifier: +? and *? become non-greedy.

Greedy .* grabs everything up to the last match; .*? stops at the first.

import re

html = '<b>hi</b> and <b>bye</b>'

# Greedy: .* grabs as much as possible
greedy = re.search(r'<b>(.*)</b>', html)
print('greedy:', greedy.group(1) if greedy else 'none')

# Non-greedy: .*? stops at the first match
shy = re.search(r'<b>(.*?)</b>', html)
print('non-greedy:', shy.group(1) if shy else 'none')

Capturing groups and substitution

Parentheses in a pattern do more than group — they capture whatever matched inside them. After a successful search, m.group(1) gives you the first capture, m.group(2) the second, and so on. This is how you extract pieces of a match without slicing the original string by hand.

re.sub(pattern, replacement, text) goes further: it replaces every match with a new string. You can refer back to captures with \1, \2, etc. This is ideal for reformatting: swap the order of two names, normalize dates, or mask credit-card digits while keeping the last four.

When you only need grouping without capturing — for example, to apply + to a whole phrase — use (?:...) instead of (...). It keeps the logic tidy and avoids wasting memory on groups you never read.

flowchart TD
  T["Price: $42.50"] --> P["Pattern\n#quot;\$(\d+)\.(\d+)#quot;"]
  P --> G1["group 1 = #quot;42#quot;"]
  P --> G2["group 2 = #quot;50#quot;"]
  style P fill:#3776ab,color:#fff

Parentheses capture parts of the match into numbered groups.

Extract captured groups and replace matches with re.sub.

import re

line = 'Item: Apple, Cost: $2.50'

# Extract the two parts after the colons
m = re.search(r'Item: (\w+), Cost: \$(\d+\.\d+)', line)
if m:
    print('Product:', m.group(1))
    print('Price:', m.group(2))

# Mask the digits in the price
masked = re.sub(r'\d+\.\d+', 'X.XX', line)
print(masked)

flowchart LR
  B["#quot;\d+#quot;\nbecomes d+"] --> X["broken"]
  G["r#quot;\d+#quot;\nstays \d"] --> Y["works"]
  style B fill:#b45309,color:#fff
  style G fill:#3776ab,color:#fff

Without the raw prefix, backslash sequences are interpreted by Python before the regex engine ever sees them.

Exercise

Write find_numbers(text) that returns a list of all number strings in text. Use re.findall with the pattern r'\d+'.

import re

def find_numbers(text):
    # Return a list of all number strings in text
    pass

import re

def find_numbers(text):
    return re.findall(r'\d+', text)

Exercise

Write redact_vowels(text) that returns a new string where every vowel (upper or lower case) is replaced with '*'. Use re.sub with a character class.

import re

def redact_vowels(text):
    # Replace every vowel with '*'
    pass

import re

def redact_vowels(text):
    return re.sub(r'[aeiouAEIOU]', '*', text)

Exercise

This function is meant to return the text inside the FIRST <b>...</b> tags, but it uses a greedy .* and grabs everything up to the last </b> instead. Fix the pattern to use a non-greedy quantifier.

import re

def first_bold(html):
    m = re.search(r'<b>(.*)</b>', html)
    return m.group(1) if m else ''

import re

def first_bold(html):
    m = re.search(r'<b>(.*?)</b>', html)
    return m.group(1) if m else ''

Exercise

Predict exactly what this script prints.

import re

text = 'Price: $42.50, Tax: $3.75'
amounts = re.findall(r'\$\d+\.\d+', text)
print(amounts)

Your prediction — what does this print?

Exercise

Write extract_extension(filename) that returns the file extension without the leading dot — for example, 'pdf' for 'report.pdf'. If there is no extension, return None. Use re.search with a capturing group and the end-of-string anchor $.

import re

def extract_extension(filename):
    # Return extension without dot, or None
    pass

import re

def extract_extension(filename):
    m = re.search(r'\.(\w+)$', filename)
    return m.group(1) if m else None

Recap

Regex is a declarative pattern language inside the re module. Use it when string methods become too verbose.
re.search finds the first match; re.findall returns every match; re.sub replaces them.
Metacharacters like ., \d, \w, +, *, ?, and character classes [...] describe flexible patterns.
Parentheses capture matched text into groups you can read with .group(n) or reuse in substitutions.
Use raw strings r"..." for every pattern so backslashes survive intact into the regex engine.
When a pattern is simple and fixed, ordinary string methods are still clearer. Regex shines when the shape of the text matters more than its exact value.

In the capstone project you will use these skills to pull emails and dates out of raw documents — the exact task regex was invented for.

Literals and metacharacters

Reading a pattern left to right

Capturing groups and substitution

Recap

Checkpoint quiz

Go deeper — technical resources