PHP — Generators

A Guide to PHP Generators

A Guide to PHP Generators: Enhancing Performance and Memory Efficiency

History

In PHP, generators are a powerful feature introduced in PHP 5.5 that allows you to iterate over a set of data without needing to create an array in memory. They provide a memory-efficient way to iterate through large sets of data by generating values on-the-fly.

Introduction

Often you need to process a big array of data, and often you will get the error from php saying:

Fatal error: Allowed memory size of xxxxxx bytes exhausted (tried to allocate xxxxx bytes) in your_script.php on line xx

The message indicates that the PHP script has exceeded the allowed memory limit. The tried to allocate part provides information about the amount of memory PHP attempted to allocate when the error occurred.

To address this issue there are several ways.

1. Increase memory limit

ini_set('memory_limit', '256M');  // Set memory limit to 256 megabytes

2. Use unset

foreach ($largeArray as $key => $value) {
    // Process $key => $value
    unset($largeArray[$key]);  // Free up memory for the processed element
}

3. Use generators
Instead of loading all array in memory and processing them, this loads and process one item at a time.

In this article, we will tackle in more detail the 3rd option.

Syntax

To create a generator, you define a function with the yield statement. The generator function will pause its execution each time it encounters a yield, allowing you to generate values lazily, meaning it only has a single reference in memory at all times.

Let's take a normal function that returns an array and converts it into a generator.

function exampleArray(int $from = 1, int $to = 1000): array
{
  $data = [];
  for ($i = $from; $i <= $to; $i++) {
    $data[] = $i;
  }
  return $data;
}

foreach (exampleArray() as $value) {
  echo "Current value:  <strong>{$value}</strong> <br>";
}

# RESULT:
// Current value: 1
// Current value: 2
// Current value: 3
// ......
// Current value: 1000 

If we use the above function it will return the result as it is displayed above. Now if you try to increase the $to variable to a very large number, then the system would generate a Fatal error as below:

Fatal error: Allowed memory size of xxxxxx bytes exhausted (tried to allocate xxxxx bytes) in your_script.php on line xx

Now let's update the above function into a generator and try again.

function exampleGenerator(int $from = 1, int $to = 1000): Iterator {
  for ($i = $from; $i <= $to; $i++) {
    // Note that $i is preserved between yields.
    yield $i;
  }
}

foreach (exampleGenerator() as $value) {
  echo "Current value:  <strong>{$value}</strong> <br>";
}

# RESULT:
// Current value: 1
// Current value: 2
// Current value: 3
// ......
// Current value: 1000

PS: Generator return types can only be declared as Generator, Iterator or Traversable

Now if we call the above function with a large number, we won’t get the same error anymore because in this case, only one data set is kept in memory at a time.

Using yield with keys

When you yield a result, there is an implicit numeric 0-based key iterating the result. You can however yield both a key and a value by adding the => arrow operator.

Considering the multi-dimensional array below:

$users = [
    [
        'id' => 1,
        'username' => 'john_doe',
        'email' => 'john.doe@example.com',
    ],
    [
        'id' => 2,
        'username' => 'jane_smith',
        'email' => 'jane.smith@example.com',
    ],
    [
        'id' => 3,
        'username' => 'alice_jones',
        'email' => 'alice.jones@example.com',
    ],
];

function exampleGeneratorMultiKey(array $input = []): Iterator {
    foreach ($input as $key => $value) {
        yield $value['id'] => $value;
    }
}

foreach (exampleGeneratorMultiKey($users) as $key => $value) {
    echo "Current key: {$key}, and value: {$value['username']} <br>". PHP_EOL;
}

# Result
Current key: 1, and value: john_doe
Current key: 2, and value: jane_smith
Current key: 3, and value: alice_jones

Sending values back to the generator

Sometimes might be needed to send values back to the generator function, and you can do that. For example, consider you are processing a big array of data, and if certain criteria are met you want to exit from the generator.

function dynamicGenerator($max) {
    for ($i = 0; $i < $max; $i++) {
        $input = yield $i;

        if ($input === 'skip') {
           continue;
        }
        
        // this will break and return a value
        if ($input === 'break') {
           return;
        }
    }
}

$generator = dynamicGenerator(15);

foreach ($generator as $value) {
    echo $value . PHP_EOL;

    // Sending a value to the generator
    // to skip the next iterator
    if ($value === 3) {
        $generator->send('skip');
    }
    
    // Sending a value to the generator
    // to exit the iteration
    if ($value === 6) {
        $generator->send('break');
    }
}

# Output
0 1 2 3 5 6

In this example above, the generator produces values from 0 to 15. When the value 3 is encountered, the calling code sends a skip to the generator to inform it to skip the next iteration. When the value 6 is encountered the calling code sends a break to the generator to notify it to exit.

Returning values from the generator

As we saw in the example above inside a generator function we can also return a value, and we can get that value returned from the generator using the following command: $generator->getReturn() So the return value will be returned once the generator has been finished executing.

function gen() {
    yield 1;
    yield 2;
    yield 3;
    return 'done';
}

$gen = gen();

foreach ($gen as $value) {
    echo $value . PHP_EOL;
}
echo $gen->getReturn() . PHP_EOL;

# Return
1 2 3 'done'

Key takeaways

1. Generator return types can only be declared as Generator, Iterator, or Traversable (compile-time check)
2. yield can be called without an argument to yield a null value with an automatic key.
3. Generator functions are able to yield values by reference as well as by value.
4. You can yield values from another generator using yield from.
5. You can use yield with key => value.
6. Generators are only executed when you start to iterate them.
7. A Generator instance can only be traversed once

I will try to explain them all with examples.

function secondGenerator(): Iterator {
    yield 4;
    yield 5;
}

function firstGenerator(): Iterator {
    yield 1;
    yield 2;
    yield 3; // yield a value
    yield;   // yield null value, CAUTION with this one because works differently witha while() loop.
    yield from secondGenerator(); // yield another generator
    yield 'done' => 'done'; // yield key and vlue pare
    return 'completed'; // return a value
}

$generator = firstGenerator();
foreach ($generator as $value) {
    echo 'value: ' .$value . PHP_EOL;
}
echo 'RETURN: ' . $generator->getReturn() . PHP_EOL;

# Output
value: 1
value: 2
value: 3
value:
value: 4
value: 5
value: done
RETURN: completed

As we said in point 7 above a generator can be traversed only once, so if we try and traverse the same generator again we will get a fatal error:

Fatal error: Uncaught Exception: Cannot traverse an already closed generator in your_script.php on line xxx

$generator = firstGenerator();
foreach ($generator as $value) {
    echo 'value: ' . $value . "<br>" . PHP_EOL;
}
echo 'RETURN: ' . $generator->getReturn() . "<br>" . PHP_EOL;

# the first traverse will work

foreach ($generator as $value) {
    echo 'value: ' . $value . "<br>" . PHP_EOL;
}

# this will throw an error
Fatal error: Uncaught Exception: Cannot traverse an already closed generator in your_script.php on line xxx

Use cases

A use case is when processing a large data set. For example, consider when you are importing data or processing data from a large CSV file.

function readFile($filename) {
    $file = fopen($filename, 'r');
    
    while (!feof($file)) {
        $data = fgets($file);
        yield process($data);
    }
    
    fclose($file);
}

foreach (readFile('large_data.txt') as $result) {
    // Process each piece of data
    echo $result . PHP_EOL;
}

PS: keep in mind that the methods you can use on generator functions are the ones related to iterators.

final class Generator implements Iterator {
  /* Methods */
  public current(): mixed
  public getReturn(): mixed
  public key(): mixed
  public next(): void
  public rewind(): void
  public send(mixed $value): mixed
  public throw(Throwable $exception): mixed
  public valid(): bool
  public __wakeup(): void
}

Conclusion

PHP generators provide an elegant and memory-efficient solution for working with large datasets, generating sequences, and improving the performance of your code. By utilizing the yield statement, you can create more readable and scalable code, avoiding unnecessary memory consumption. However, it's crucial to be mindful of properly closing generators and preventing infinite loops to ensure your code remains robust and efficient.

Feel free to Subscribe for more content like this 🔔, clap 👏🏻 , comment 💬, and share the article with anyone you’d like

And as it always has been, I appreciate your support, and thanks for reading.