Week 3 — Iterators & Generators

Implement the iterator protocol from scratch, write memory-efficient generators with yield, compose pipelines with yield from, and use generator's send() for two-way communication.

iteratorgeneratoryieldyield fromprotocol

Duration

⏱ 2.5 hours

Level

📊 Advanced

Prerequisite

🎯 Intermediate Week 2

OUTCOME

Build a lazy CSV reader and a coroutine-based data pipeline

What you'll learn

1Implement __iter__ and __next__ in a custom class
2Distinguish iterables from iterators
3Write generator functions and use them in pipelines
4Delegate to sub-generators with yield from
5Use send() and throw() for coroutine communication

1. Iterator Protocol

python

class Counter:
    def __init__(self, start, stop):
        self.current = start
        self.stop = stop

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        val = self.current
        self.current += 1
        return val

for n in Counter(1, 5):
    print(n, end=' ')   # 1 2 3 4

2. Generator Functions

python

def range_gen(start, stop, step=1):
    current = start
    while current < stop:
        yield current
        current += step

# Same as above but lazy — no list created
for n in range_gen(0, 10, 2):
    print(n, end=' ')   # 0 2 4 6 8

# yield from
def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain([1,2], [3,4], [5])))  # [1,2,3,4,5]

3. Coroutines with send()

python

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value

acc = accumulator()
next(acc)           # prime the generator
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5))  # 35

💻 Examples

Run these examples and check the output yourself.

01_lazy_csv.py— Memory-efficient lazy CSV reader generator

CODE

import csv
from pathlib import Path

def lazy_csv(path, batch_size=100):
    """Yield rows from a CSV file in batches."""
    with open(path, newline='', encoding='utf-8') as f:
        reader = csv.DictReader(f)
        batch = []
        for row in reader:
            batch.append(row)
            if len(batch) >= batch_size:
                yield batch
                batch = []
        if batch:
            yield batch

# Process a large file without loading it all
for batch in lazy_csv('data.csv'):
    print(f'Processing {len(batch)} rows...')
    # process batch here

📝 Exercises

Try them yourself first, then open the solution to compare.

Exercise 1

Infinite sequence generators

Goal: Write three infinite generators: naturals(), primes(), and fibonacci().

Requirements

Each yields values indefinitely
Use itertools.islice to take N values
All work correctly with next() and for loops

▶Toggle solution

SOLUTION

from itertools import islice

def naturals(start=1):
    n = start
    while True:
        yield n
        n += 1

def primes():
    def is_prime(n):
        if n < 2: return False
        return all(n % i for i in range(2, int(n**0.5)+1))
    for n in naturals(2):
        if is_prime(n):
            yield n

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a+b

print(list(islice(naturals(), 5)))
print(list(islice(primes(), 10)))
print(list(islice(fibonacci(), 10)))

▶ Output

[1, 2, 3, 4, 5]
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Example code / lecture materials

All lecture materials and example code are openly available on GitHub.

View on GitHub ↗