How I have used PHP generators in practice for the first time

A Generator in PHP is not widely used construct. Although I knew how they work I have never used them in production applications… for now. Last time I managed to solve a memory problem using the generators, so let me show you how they can be used in real world issues.

Introduction: What is a Generator?

A Generator is an object returned by a function with yield keyword inside. It’s an iterator that returns next element when we call yield in the function. Because it implements Iterator interface, we can use the iterator in loops for example.

Let’s consider following code snippet with a very simple generator:

We declare a function with yield inside. Because of that it’s not really a function now but a generator. When we use the $sequence in foreach, the first iteration calls the function body and gets first occurrence of yield that returns the $number element. Then function execution is stopped, foreach body is executed and a new iteration starts. New iteration calls next() method on the iterator and next yield is fetched and so on. Simple, right?

So let’s start the real game.

The problem

Imagine you have an application using CQRS (Command-Query Responsibility Segregation) and two database. One for writing (Write Model, relational database like MySQL) and one for reading (Read Model, NoSQL database like ElasticSearch or MongoDB). Every time a Domain Model is persisted in Write Model, an event is sent to the Projection Builder which creates a Read Model entity from the Write Model data. Moreover it’s not one-to-one map. A Read Model entity differs from Domain Model and can include data from other Domain Models (because Read Model is not relational, we embed related models).

It’s not a problem. When persisting normal traffic changes everything works perfect. The problem occurs if you need to add additional field to the Read Model and be sure every entity has proper value there. The only way to do this is to rebuild all Projections. If your database is large, this can be a not straightforward issue to solve.

Our code wasn’t very clever. We just loaded whole Write Model to an array and used foreach to build every projection using existing code. It worked great on development environment and small sets of data, but on the production server PHP ran out of memory.

A Generator to the rescue

We wanted to achieve two goals:

• Load the whole Write Model in small chunks
• Don’t need to rewrite the Projection Builder

So we needed to create a method in Repository that will return an Iterator and load small parts of data on demand. Generator fits perfectly for this particular task.

We created an AggregateGenerator responsible for fetching the data.

As you can see, it accepts RepositoryInterface and number of entries per chunk. Every chunk is yielded to generator and then it’s removed from memory, so new can be loaded. Moreover, we add a method to the repository that returns the generator of current aggregate.

Then in our Projection Builder we only needed to replace fetchAll() call with fetchAllAsIterator(). Hooray! The issue is solved and we don’t longer fear the sentence: “Guys, this change will require rebuild of all projections”