PHP Generators in Practice: Processing Data That Does Not Fit in Memory

A script that works fine against 10,000 rows and dies with Allowed memory size exhausted against 2 million is one of the most predictable failures in PHP. The usual reflex is to raise memory_limit until the error stops. That buys time and hides the real issue: the code holds the entire data set in memory at once when it only ever looks at one row at a time.

Generators fix that shape. They let a function produce values one at a time, pausing between each, so peak memory stays flat no matter how many rows flow through. The mechanism has been in PHP since 5.5 and is well documented — what is less well covered is how to compose generators into real pipelines, and the specific ways they bite when you assume they behave like arrays.

What Actually Changes in Memory

Consider reading a CSV export and filtering it. The array version builds the complete result set before the first row is inspected:

<?php

// Eager: every row lives in memory simultaneously.
function readCsvEager(string $path): array
{
    $rows   = [];
    $handle = fopen($path, 'rb');
    fgetcsv($handle);                       // discard the header row

    while (($row = fgetcsv($handle)) !== false) {
        $rows[] = $row;
    }
    fclose($handle);

    return $rows;                           // peak memory scales with row count
}

// Lazy: one row is live at any moment.
function readCsvLazy(string $path): Generator
{
    $handle = fopen($path, 'rb');
    if ($handle === false) {
        throw new RuntimeException("Cannot open {$path}");
    }

    try {
        fgetcsv($handle);                   // discard the header row

        while (($row = fgetcsv($handle)) !== false) {
            yield $row;                     // pause here until the caller asks again
        }
    } finally {
        fclose($handle);                    // runs even if the consumer breaks early
    }
}

The finally block matters more than it looks. If the consumer stops iterating halfway through — a break, an exception, or simply never finishing the loop — PHP destroys the generator and runs the finally, closing the file handle. Without it, abandoned generators leak descriptors in long-running workers.

Measuring the difference is worth doing once so the numbers stop being abstract:

<?php

$path = __DIR__ . '/orders.csv';           // ~2 million rows

$before = memory_get_peak_usage(true);
$count  = 0;
foreach (readCsvLazy($path) as $row) {
    $count++;
}
$lazyPeak = memory_get_peak_usage(true) - $before;

printf("lazy:  %d rows, peak delta %.2f MB%s", $count, $lazyPeak / 1048576, PHP_EOL);

// The eager version on the same file will either report a peak in the
// hundreds of megabytes or fatal out, depending on memory_limit.

The lazy figure stays essentially flat as the file grows. That is the whole point: memory becomes a function of row size, not row count.

Composing Stages Into a Pipeline

A single generator is mildly useful. Generators become genuinely powerful when each stage of a transformation is its own generator that accepts an iterable and yields an iterable. Stages then chain without any intermediate collection ever existing.

<?php

declare(strict_types=1);

/**
 * Each stage takes an iterable and returns a Generator, so stages compose
 * freely and nothing is materialised between them.
 */
final class Pipeline
{
    public static function map(iterable $source, callable $fn): Generator
    {
        foreach ($source as $key => $value) {
            yield $key => $fn($value);
        }
    }

    public static function filter(iterable $source, callable $predicate): Generator
    {
        foreach ($source as $key => $value) {
            if ($predicate($value)) {
                yield $key => $value;
            }
        }
    }

    /** Group the stream into fixed-size chunks - essential for bulk inserts. */
    public static function chunk(iterable $source, int $size): Generator
    {
        if ($size < 1) {
            throw new InvalidArgumentException('Chunk size must be >= 1');
        }

        $buffer = [];
        foreach ($source as $value) {
            $buffer[] = $value;
            if (count($buffer) === $size) {
                yield $buffer;
                $buffer = [];               // release the chunk immediately
            }
        }

        if ($buffer !== []) {
            yield $buffer;                  // final partial chunk
        }
    }

    /** Stop after N items without consuming the rest of the source. */
    public static function take(iterable $source, int $limit): Generator
    {
        $taken = 0;
        foreach ($source as $key => $value) {
            if ($taken++ >= $limit) {
                return;
            }
            yield $key => $value;
        }
    }
}

Wired together, an ETL job that would otherwise need a queue and a scratch table becomes a readable chain:

<?php

$rows      = readCsvLazy(__DIR__ . '/orders.csv');
$parsed    = Pipeline::map($rows, static fn(array $r): array => [
    'id'       => (int) $r[0],
    'email'    => strtolower(trim($r[1])),
    'total_cents' => (int) round((float) $r[2] * 100),
    'placed_at'   => $r[3],
]);
$valid     = Pipeline::filter($parsed, static fn(array $o): bool =>
    $o['id'] > 0 && filter_var($o['email'], FILTER_VALIDATE_EMAIL) !== false
);
$batches   = Pipeline::chunk($valid, 500);

$pdo = new PDO('mysql:host=localhost;dbname=shop', 'user', 'pass', [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
]);

$inserted = 0;
foreach ($batches as $batch) {
    $placeholders = implode(',', array_fill(0, count($batch), '(?,?,?,?)'));
    $stmt = $pdo->prepare(
        "INSERT INTO orders (id, email, total_cents, placed_at) VALUES {$placeholders}"
    );

    $params = [];
    foreach ($batch as $order) {
        array_push($params, $order['id'], $order['email'],
                            $order['total_cents'], $order['placed_at']);
    }

    $stmt->execute($params);
    $inserted += count($batch);
}

printf("Inserted %d orders, peak memory %.2f MB%s",
    $inserted, memory_get_peak_usage(true) / 1048576, PHP_EOL);

Nothing in that chain runs until the foreach over $batches starts pulling. Each iteration drags exactly one row through every stage. Peak memory is bounded by the chunk size — 500 rows — regardless of whether the file has ten thousand rows or fifty million.

That property is what makes generators valuable in constrained runtimes. A Lambda function with a fixed memory allocation can stream a multi-gigabyte S3 object it could never load; the deployment side of that pattern is covered in the guide to serverless PHP with Bref on AWS Lambda.

The yield from Key Collision

yield from delegates to another traversable, splicing its values into the outer generator. It is the natural way to flatten nested sources, and it carries a trap that silently destroys data.

It preserves the inner generator’s keys. Each delegated source restarts its keys at zero, so a flattened stream contains repeated keys — and any consumer that keys by them keeps only the last value.

<?php

function fileLines(string $path): Generator
{
    $handle = fopen($path, 'rb');
    try {
        $lineNumber = 0;
        while (($line = fgets($handle)) !== false) {
            yield $lineNumber++ => rtrim($line, "\r\n");   // keys restart at 0
        }
    } finally {
        fclose($handle);
    }
}

function allLines(array $paths): Generator
{
    foreach ($paths as $path) {
        yield from fileLines($path);        // inner keys leak through
    }
}

$paths = ['a.txt', 'b.txt', 'c.txt'];       // 100 lines each

// Iterating is fine - foreach does not care about duplicate keys.
$counted = 0;
foreach (allLines($paths) as $line) {
    $counted++;
}
echo $counted, PHP_EOL;                     // 300, as expected

// Collecting is NOT fine - later keys overwrite earlier ones.
echo count(iterator_to_array(allLines($paths))), PHP_EOL;        // 100 - data lost!

// The fix: discard keys during collection.
echo count(iterator_to_array(allLines($paths), false)), PHP_EOL; // 300 - correct

The second argument to iterator_to_array() defaults to true, meaning “preserve keys”. Passing false reindexes and keeps everything. Any pipeline stage that might see delegated input should either yield values without keys or generate globally unique ones.

Return Values and Two-Way Communication

A generator can return a value alongside everything it yields, retrievable with getReturn() once iteration completes. This is the clean way to surface summary statistics without a by-reference parameter or a mutable counter captured in a closure.

<?php

/** Yields valid rows; returns a tally of what it rejected. */
function validateRows(iterable $rows): Generator
{
    $accepted = 0;
    $rejected = [];

    foreach ($rows as $index => $row) {
        if (!isset($row['email'], $row['id'])) {
            $rejected[$index] = 'missing required field';
            continue;
        }
        $accepted++;
        yield $row;
    }

    return ['accepted' => $accepted, 'rejected' => $rejected];
}

$validator = validateRows($parsedRows);

foreach ($validator as $row) {
    processRow($row);
}

// Only legal AFTER the generator has finished. Calling it early throws
// "Cannot get return value of a generator that hasn't returned".
$summary = $validator->getReturn();
printf("accepted %d, rejected %d%s",
    $summary['accepted'], count($summary['rejected']), PHP_EOL);

Generators also accept values pushed back in via send(), which turns them into simple coroutines. In practice this is the least-used capability, because once you want real cooperative multitasking, PHP Fibers handle it far better — Fibers can suspend from anywhere in a nested call stack, while a generator can only pause at its own yield statements.

Where Generators Stop Working

Generators trade capability for memory, and the things they give up cause real bugs when code assumes array semantics.

The one-shot restriction catches people most often. A generator is consumed as it runs, and rewinding an already-started generator throws an exception:

<?php

$rows = readCsvLazy('orders.csv');

foreach ($rows as $row) { /* ... */ }
foreach ($rows as $row) { /* Exception: Cannot traverse an already closed generator */ }

// Pass a factory rather than a generator when the data must be read twice.
$makeRows = static fn(): Generator => readCsvLazy('orders.csv');

foreach ($makeRows() as $row) { /* first pass  */ }
foreach ($makeRows() as $row) { /* second pass - a fresh generator */ }

Sorting deserves its own warning. A sort needs every element before it can emit the first one, so any sort inside a pipeline silently reintroduces the memory profile you built the pipeline to avoid. If ordering matters, push it into the ORDER BY of the query producing the stream, or accept the buffering cost deliberately rather than by accident.

There is also a cost worth knowing about: generators are slower per element than a plain array loop. Each yield is a context switch that saves and restores execution state. On a few thousand items where memory is not a concern, an array is both faster and simpler. Generators pay off when the data set is large, unbounded, or arriving from a stream — not as a blanket replacement for arrays.

Choosing the Right Abstraction

Generators are one option among several for producing sequences, and the right pick depends on whether the source is re-readable and whether callers need array behaviour.

• Plain array — the data is small and bounded, and callers want count(), sorting, and repeated traversal. Do not over-engineer this case.
• Generator function — a one-shot stream from a file, socket, or paginated API. The default choice for ETL work.
• IteratorAggregate — a domain object that should be iterable more than once. Return a fresh generator from getIterator() and the re-traversal problem disappears.
• SplFixedArray — a known-length numeric data set where you need random access but want lower overhead than a standard array.

The IteratorAggregate pattern is the one most codebases should use more:

<?php

/** Iterable as many times as you like - each call builds a new generator. */
final class OrderExport implements IteratorAggregate
{
    public function __construct(private readonly string $path) {}

    public function getIterator(): Generator
    {
        $handle = fopen($this->path, 'rb');
        try {
            fgetcsv($handle);
            while (($row = fgetcsv($handle)) !== false) {
                yield $row;
            }
        } finally {
            fclose($handle);
        }
    }
}

$export = new OrderExport(__DIR__ . '/orders.csv');
foreach ($export as $row) { /* works */ }
foreach ($export as $row) { /* still works - new generator each time */ }

The full semantics of yield, delegation, and generator return values are specified in the PHP manual’s generators chapter, which is worth reading once end to end rather than in fragments.

Rules That Hold Up

A few habits prevent most generator bugs before they happen:

• Wrap every resource a generator opens in try/finally, so abandoned iteration still closes it.
• Type parameters as iterable rather than array, so a function accepts both without change.
• Pass false as the second argument to iterator_to_array() unless you have specifically verified the keys are unique.
• Return factories, not generator instances, from anything a caller might iterate twice — or implement IteratorAggregate.
• Chunk before writing. Row-at-a-time inserts turn a memory problem into a latency problem.
• Keep sorts and aggregations in the database where the data already lives.

The underlying shift is thinking in streams rather than collections. Once a function accepts an iterable and yields an iterable, it stops caring whether the source is ten rows or ten million — and the memory limit stops being a number you tune.