Use a Generator to batch process TYPO3 repository entries

When processing too many entries in a TYPO3 extbase repository, you end up running out of memory because the persistenceManager clings to every processed entry, unless you clearState your way out of this mess. To work around this issue you can batch process the entries you are processing by making your nice list iteration ugly.

Generator to the rescue. Instead of doing the same work over and over again, obfuscating your clean code, you can write a generic Repository-Trait that you can use for every occasion where you have to batch process all entries in your repository.

trait Batchable
{
    /**
     * @param int $batchSize
     * @param bool $clearState
     * @return \Generator
     */
    public function findAllBatched($batchSize, $clearState = true)
    {
        $count  = $this->findAll()->count();
        $offset = 0;
        if ($count > 0) {
            while ($count > $offset) {
                $objects = $this->findAllSelective($batchSize,               
                               $offset);
                foreach ($objects as $object) {
                    yield $object;
                }
                $offset += $batchSize;
                if ($clearState) {
                    $this->persistenceManager->persistAll();
                    $this->persistenceManager->clearState();
                }
            }
        }
    }
     /**
     * @param int $limit
     * @param int $offset
     * @return \TYPO3\CMS\Extbase\Persistence\QueryResultInterface
     */
    public function findAllSelective($limit = null, $offset = null)
    {
        $query = $this->createQuery();
        if (isset($limit)) {
            $query->setLimit($limit);
        }
        if (isset($offset)) {
            $query->setOffset($offset);
        }
        return $query->execute();
    }
}

This can be further improved by providing a callback instead of using the findAllSelective specifically giving you the ability to batch-process every kind of database query, but I leave this in a more simple state for this example.

Now all you have to do — including adding the use statement for the Trait to your Repository — to process every single entry in your whole repository without running the risk of running out of memory is:

foreach($this->myRepository->findAllBatched(50) as $entry){
    // Do horrible, horrible things to the poor $entry
}