With clean text in hand, the next job is to turn it into tokens and count how often each one appears. A token is one cleaned word, and the frequency table that counts them is the heart of the analyzer, the structure every later milestone reads from.
You could build that table by hand, looping over the words and incrementing a dict one key at a time. It works. But Python ships a tool built exactly for this job: collections.Counter. A Counter is a dictionary specialized for counting, with methods for the questions you actually ask, like which word appeared most? Reaching for it turns five lines of bookkeeping into one, and the result is faster to read and harder to get wrong.
flowchart LR W["list of words"] --> C["Counter(...)"] C --> F["freq table<br/>word -> count"] style C fill:#3776ab,color:#fff style F fill:#3776ab,color:#fff
Counting with Counter
Counter lives in the collections module. Hand it any iterable of items and it tallies how many times each one appears, returning a dict-like object whose values are the counts.
from collections import Counter
words = ['the', 'cat', 'the', 'bird', 'the']
freq = Counter(words)
# Counter({'the': 3, 'cat': 1, 'bird': 1})
Because a Counter is a dictionary, everything you already know about dicts applies: freq['the'] reads a count, freq.get('xyz', 0) returns 0 for an unseen word, and len(freq) is the number of distinct words. The difference is that Counter does the tallying for you in one call, and it adds a handful of counting-specific methods a plain dict lacks.
from collections import Counter
words = ['the', 'cat', 'the', 'bird', 'the', 'cat']
freq = Counter(words)
print('the:', freq['the'])
print('cat:', freq['cat'])
print('dragon:', freq.get('dragon', 0))
print('distinct:', len(freq))
Missing keys return zero
A plain dict raises KeyError the moment you read a key it does not hold. A Counter overrides that: subscripting a missing key returns 0 instead of raising, because a word you have never seen has, sensibly, a count of zero. That is why the demo prints dragon: 0, and why with a Counter you can drop the .get entirely: freq['dragon'] alone returns 0.
The convenience is real. With a Counter you can look up any word without first checking whether it exists, which keeps counting code short and crash-free. One small duty comes with it: a pure read never adds the key, but an assignment like freq['dragon'] = 5 does, so treat reads as free and be deliberate about writes. This default-zero behavior is the feature that makes a Counter feel purpose-built for counting.
Finding the most frequent words
The question that motivated counting is which words appear most? Counter answers it directly with most_common(n), which returns the top n entries as a list of (word, count) pairs, ranked from most to least frequent.
freq.most_common(2)
# [('the', 3), ('cat', 2)]
Read the return type carefully: it is a list of tuples, not a list of words. Each pair holds the word and its count together, which is exactly what you need to print a ranked report. Asking for most_common() with no argument returns every entry ranked, not just the top few.
from collections import Counter
words = ['the', 'cat', 'the', 'bird', 'the', 'cat']
freq = Counter(words)
top = freq.most_common(2)
print(top)
for word, count in top:
print(word, '->', count)
flowchart LR
subgraph W["Counter of a list"]
A1["to, be, to"] --> A2["to: 2, be: 1"]
end
subgraph C["Counter of a string"]
B1["to be to"] --> B2["counts each letter"]
end
style A2 fill:#3776ab,color:#fff
style B2 fill:#b45309,color:#fff
The trap: counting characters instead of words
Here is the mistake every beginner makes once. They clean the text into a single string and hand that string straight to Counter, expecting word counts:
Counter('to be or not to be') # counts letters, not words!
Counter counts the items it is given. Given a list, it counts list elements. Given a string, it counts characters, because a string is an iterable of characters. The result is full of single letters and a count for every space. The fix is to split first, so Counter receives words: Counter(text.split()). The lesson is general: know what kind of iterable you are counting before you call it.
from collections import Counter
def tokenize(text):
return text.lower().split()
text = 'to be to be to'
freq = Counter(tokenize(text))
print(freq.most_common(2))
print('total words:', sum(freq.values()))
flowchart LR F["freq table"] --> M["most_common(2)"] M --> L["a ranked list of pairs"] L --> P1["to: 3"] L --> P2["be: 2"] style M fill:#3776ab,color:#fff
Write tokenize(text) that lowercases text and splits it into a list of words on whitespace. So tokenize("To Be") returns ["to", "be"].
def tokenize(text):
# lowercase, then split into words
pass
text.lower() folds the case; .split() with no arguments splits on any whitespace.
Chain them: text.lower().split().
def tokenize(text):
return text.lower().split()
Write count_words(words) that takes a list of words and returns a collections.Counter of their frequencies. Import Counter from collections.
from collections import Counter
def count_words(words):
# build and return a Counter
pass
Counter(words) builds the frequency table in one call.
Counter(a=2, b=1) is a handy way to write an expected Counter in a test.
from collections import Counter
def count_words(words):
return Counter(words)
What does this print? Counter is given a string, not a list.
from collections import Counter
c = Counter("aab")
print(c["a"])
A string is an iterable of characters, so Counter counts letters.
The letter 'a' appears twice in 'aab'.
This word_counts is meant to count how often each word appears, but it counts characters instead, because it hands the raw string to Counter. Fix it so Counter receives a list of words, by splitting the text first.
from collections import Counter
def word_counts(text):
return Counter(text) # bug: counts characters
Counter counts whatever items it is given; a string is an iterable of characters.
Call text.split() first so Counter counts words, not letters.
from collections import Counter
def word_counts(text):
return Counter(text.split())
Bring cleaning and counting together. Write top_n_words(text, n) that lowercases text, splits it into words, counts them with Counter, and returns the n most common as a list of (word, count) pairs. So top_n_words("to be to be to", 1) returns [('to', 3)].
from collections import Counter
def top_n_words(text, n):
# lowercase, split, count, then return the n most common
pass
Counter(text.lower().split()) builds the frequency table.
Return freq.most_common(n) for the ranked list of pairs.
from collections import Counter
def top_n_words(text, n):
freq = Counter(text.lower().split())
return freq.most_common(n)
Recap
- A token is one cleaned word; the frequency table counts them.
collections.Countertallies an iterable in one call and behaves like a dict.Counter.most_common(n)returns a list of (word, count) pairs, ranked.- Hand
Countera list to count words, or a string to count characters; that is the classic trap. - Reach for
Counterover a manual tally loop unless you are showing how it works inside.
Milestone 3 turns these counts into statistics: average word length, length distributions, and a reading-level estimate.