34 · Generators with yield
Write lazy sequences with yield — a function that produces values on demand, uses constant memory regardless of sequence length, and composes cleanly.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
34 · Generators with yield
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.A **generator function** (any `def` with `yield`) returns a **generator object** when called — it does **not** run the body immediately. Each call to `next()` on the generator resumes execution from where `yield` last paused it, emitting one value. When the function returns (or falls off the end) Python automatically raises `StopIteration`. The generator is an iterator — it plugs into `for`, `zip`, `list()`, and everything else.
Without this:
Without generators you'd build giant lists to represent sequences — a million-element range would consume millions of bytes before your loop even starts. Generators allow streaming computation: the next value is computed only when asked for, which is the foundation of all memory-efficient data pipelines.
Generators exist to solve two problems simultaneously:
- Memory — if you're producing a long sequence, there's no reason to hold all values in RAM at once. A generator yields one value, discards it, computes the next.
- Composition — generators chain together without intermediate allocations.
gen_b(gen_a(source))pulls values through a pipeline of transformations one at a time.
The syntax is simple: put yield anywhere inside a def and the function becomes a generator function. Calling it returns a generator object; the body hasn't run yet. Every next() call resumes the body until the next yield, then suspends again. There are three ways to write generators in Python: generator functions (def + yield), generator expressions ((...)), and yield from for delegation.
Python (in browser)
Calling `counts_up_to(3)` does not execute a single line of the body — the `for i in range(n)` hasn't started. Only when `next(g)` is called does execution enter the function, run until `yield i`, emit the value, and then freeze.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The generator expression `(x*x for x in range(N))` creates a tiny object (around 200 bytes) regardless of how big `N` is. The equivalent list comprehension allocates memory for every element upfront. We use 1M here so the cell runs comfortably in your browser; the same generator with `range(1_000_000_000)` would be just as small. For sequences you only partially consume, the generator is always the right choice.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`yield from f` delegates iteration to the file object itself — each `next()` call reads exactly one line from disk without buffering the whole file. This is the idiomatic pattern for streaming file processing in Python: wrap the open/read logic in a generator and callers never see file handles.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
`yield from iterable` is shorthand for `for x in iterable: yield x` but also correctly threads `send()` and `throw()` calls through the chain (advanced usage). The pipeline pattern here is the Python equivalent of Unix pipes: each stage is independent, composable, and memory-constant.
Does calling a generator function immediately execute its body?
- A generator function is any `def` containing `yield`. Calling it returns a generator object — the body does not run until `next()` is called.
- Generator expressions `(expr for x in iterable)` are the one-liner equivalent of generator functions and use constant memory.
- `yield from iterable` delegates to another iterable, making it easy to compose generators into pipelines without intermediate allocations.
- A 10M-element generator expression occupies ~200 bytes; the equivalent list occupies ~80 MB. For sequences consumed once or partially, prefer the generator.
Streaming datasets that don't fit in RAM are modelled as generators: `def batches(path, batch_size): ... yield batch` is the pattern behind PyTorch `IterableDataset` and TensorFlow `tf.data.Dataset`. HuggingFace `datasets.IterableDataset` is a generator-based API for datasets too large to cache locally. Even NumPy's `np.load(..., mmap_mode='r')` returns a memory-mapped array that acts like a lazy generator over disk-resident data.
If you remove it: Without generators, loading a 100 GB dataset for fine-tuning would require materialising the full contents in RAM before training can begin — an impossible constraint on most machines. The generator model is what makes it feasible to train on data that is orders of magnitude larger than available memory.