Why streams beat lists
Suppose you must process a log file with ten million lines. Reading it all into a list works, once. On a larger file it exhausts memory and the program crawls or crashes, even though your loop only ever looks at one line at a time. The list eagerly builds the entire collection up front; your computation needs only the current item.
A generator is the lazy alternative: it produces values one at a time, on demand, and forgets each one the moment it has been consumed. You can iterate over a sequence of any size because you never hold more than a single value in flight. The same idea powers range, file reading, and database cursors, anything that might be too big to materialise all at once.
flowchart LR L["list<br/>eager: every value built up front"] --> M["all in memory now"] G["generator<br/>lazy: yields one value at a time"] --> O["produced on demand"] style L fill:#b45309,color:#fff style G fill:#3776ab,color:#fff
The protocol: iterables, iterators, next
Three words that beginners blur together. An iterable is anything you can loop over, a list, a string, a dict. An iterator is the object that actually hands out the values one by one. You turn an iterable into an iterator with iter(), and you pull the next value out with next().
it = iter([10, 20, 30])
next(it) # 10
next(it) # 20
When the iterator runs dry, next() raises StopIteration. A for loop is just sugar over this: it calls iter() once, then next() repeatedly, catching StopIteration to know when to stop. Knowing these mechanics makes errors like "X is not iterable" suddenly readable.
nums = [10, 20, 30] it = iter(nums) print(next(it)) print(next(it)) print(next(it))
yield pauses instead of returning
A generator function looks like an ordinary function with one keyword swapped: where return ends the call, yield pauses it. Calling the function returns a generator object immediately, without running its body. Each next() runs forward to the next yield, hands back the value, and freezes the function, local variables included, until it is called again.
def count_up_to(limit):
n = 1
while n <= limit:
yield n
n += 1
Calling count_up_to(3) does no work yet; it hands back a generator waiting at the start. The loop for v in count_up_to(3) drives it, pulling 1, 2, 3 in turn. State survives between pulls, which is what lets a generator build each value from the previous one.
flowchart LR C["next()"] --> Y["run forward to yield"] Y -->|"returns value, pauses"| P["frozen state"] P --> C style Y fill:#3776ab,color:#fff style P fill:#1e293b,color:#fff
def count_up_to(limit):
n = 1
while n <= limit:
yield n
n += 1
for value in count_up_to(3):
print(value)
Laziness makes infinity cheap
Because a generator produces values only when asked, an infinite generator is perfectly safe; you simply stop pulling. A Fibonacci generator can yield forever inside a while True loop, and nothing happens until a consumer asks for a value. The loop ends the moment you stop asking. Try that with a list and the program hangs until memory runs out.
This is the deepest payoff of the lazy model: the producer and the consumer are decoupled. The generator does not need to know how many values you want; the loop does not need to know how they are made. Each side does its own job, and neither has to hold the whole sequence in memory.
Generator expressions: the one-liner
When the body is a single expression, skip the def and write a generator expression, comprehension syntax wrapped in parentheses. (x * x for x in range(5)) builds a lazy stream of squares with no list ever created.
Pass it straight to a consumer like sum, max, or list. sum(x * x for x in range(5)) computes 30 without ever materialising [0, 1, 4, 9, 16]. Reach for this form when you only need to fold a sequence down to one value; reach for the generator function when each step needs real logic, running state, or several statements.
squares_list = [x * x for x in range(5)] squares_gen = (x * x for x in range(5)) print(sum(squares_list)) print(sum(squares_gen)) print(type(squares_gen).__name__)
Write a generator count_down(n) that yields n, then n-1, and so on down to 1. So list(count_down(3)) is [3, 2, 1]. Use yield inside a loop.
def count_down(n):
# while n > 0: yield n; then n -= 1
pass
A while loop that runs while n > 0 covers the count-down.
yield the current n, then subtract one before the next turn.
def count_down(n):
while n > 0:
yield n
n -= 1
What does this print? A generator yields two values; next() pulls them one at a time.
def gen():
yield "a"
yield "b"
g = gen()
print(next(g))
print(next(g))
The first next() runs to the first yield and returns 'a'.
The second next() resumes and runs to the second yield, returning 'b'.
Write a generator running_total(nums) that yields the running sum as it walks the list. For [1, 2, 3, 4] it yields 1, then 3, then 6, then 10. Keep a total variable that survives between yields.
def running_total(nums):
total = 0
# for each n: update total, then yield it
pass
Add each n to total before yielding.
total persists across yields because the generator's frame is frozen, not discarded.
def running_total(nums):
total = 0
for n in nums:
total += n
yield total
This is meant to yield the square of every number, but list(squares([1, 2, 3])) returns [1] and stops. The bug is the keyword: return ends the whole call on the first item. Replace it with the one that pauses instead.
def squares(nums):
for n in nums:
return n * n # bug: stops the generator on the first item
return finishes the function and hands back one value.
yield pauses after each value so the loop can keep going.
def squares(nums):
for n in nums:
yield n * n
Write a generator flatten(nested) that takes a list of lists and yields every element one at a time, in order. So list(flatten([[1, 2], [3, 4]])) is [1, 2, 3, 4]. Use two loops and a yield.
def flatten(nested):
# for each row, for each item: yield it
pass
Outer loop over rows, inner loop over the items in each row.
yield item (not return), so all values stream out lazily. 'yield from row' is the one-line shortcut.
def flatten(nested):
for row in nested:
for item in row:
yield item
Recap
- An iterable is loopable; an iterator is the object that yields its items via
iter()andnext(), ending withStopIteration. - A generator function uses
yieldto pause and resume; its local state is kept between pulls. - A generator expression
(expr for x in src)is the lazy cousin of a list comprehension: no list, no memory. - Generators are single-use; once exhausted they yield nothing, so rebuild the generator or materialise with
list()to reuse the values.
Next you will reach for the collections module, which offers ready-made containers for the counting and grouping jobs you currently write by hand.