CakePHP — Run background jobs using queued events

At SprintCube, we build large-scale backend applications for our clients. These applications perform heavy operations that are time-intensive. Those processes are not desirable to do so during a request, as the resulting slowness is perceived directly by the application’s users. Instead of performing that kind of task that takes longer such as image processing, the sending of an email, or any kind of background synchronization, should be carried out as a background task/job.

If you already are using CakePHP, you are familiar with its event system. If not, read more about its Events System.

In a typical application, we fire an event on certain activities and our predefined event listener catches that event and perform the job. For example, sending a welcome email on successful user registration.

// in src/Model/Table/UsersTable.php
namespace App\Model\Table;
use Cake\Event\Event;
use Cake\ORM\Table;
class UsersTable extends Table
{
    public function register($user)
    {
        if ($this->save($user)) {
            $event = new Event('User.register', $this, [
                'user' => $user
            ]);
            $this->eventManager()->dispatch($event);
            return true;
        }
        return false;
    }
}

And suppose we have a listener which sends a welcome email to the newly registered user, and also another one which updates the total users count for dashboard statistics, and so on. This makes the request very slow and a bad experience for the user.

To avoid this kind of situation and make the user experience better, we use the background jobs in our application using Events and josegonzalez/cakephp-queuesadilla plugin. It provides a nice set of functionality to add jobs in the queue and a worker to perform queued jobs.

This article describes how we use this plugin to implement background jobs in our CakePHP applications.

Installation & Configuration of plugin

You can install the plugin via Composer.

composer require josegonzalez/cakephp-queuesadilla

Read more here: https://cakephp-queuesadilla.readthedocs.io/en/latest/installation.html

After installing the plugin, load the plugin via the appropriate method. In your src/Application.php

public function bootstrap()
{
    ...
    $this->addPlugin('Josegonzalez/CakeQueuesadilla');
    ...
}

In your config/app.php add a configuration array to use in the plugin.

...
'Queuesadilla' => [
    'default' => [
        'url' => 'mysql://root:password@localhost:3306/database?queue=default&timeout=1'
    ]
]
...

You can read more about different configuration options here: https://cakephp-queuesadilla.readthedocs.io/en/latest/configuration.html

And consume this configuration in plugin by adding the following line in your config/bootstrap.php

use Josegonzalez\CakeQueuesadilla\Queue\Queue;
Queue::setConfig(Configure::consume('Queuesadilla'));

And apply the database migration to generate the default jobs table,

bin/cake migrations migrate --plugin Josegonzalez/CakeQueuesadilla

Now the plugin is set up and ready to use!

Queue Manager

We use a special QueueManager class to allow our application to add jobs into the queue.

Create a file under src/Queue/QueueManager.php with the following content.

<?php
namespace App\Queue;
use Cake\Event\Event;
use Cake\Event\EventManager;
use Josegonzalez\CakeQueuesadilla\Queue\Queue;
use josegonzalez\Queuesadilla\Job;
class QueueManager
{
    /**
     * Places an event in the job queue
     *
     * @param Event $event Cake Event
     * @param array $options Options
     * @return void
     */
    public static function queue(Event $event, array $options = [])
    {
        Queue::push(
            '\App\Queue\QueueManager::dispatchEvent',
            [get_class($event), $event->getName(), $event->getData()],
            $options
        );
    }
    /**
     * Constructs and dispatches the event from a job
     *
     * ### Data array
     * - 0: event FQCN
     * - 1: event name
     * - 2: event data array
     *
     * @param Job\Base $job Job
     * @return void
     */
    public static function dispatchEvent($job)
    {
        $eventClass = $job->data(0);
        $eventName = $job->data(1);
        $data = $job->data(2, []);
$event = new $eventClass($eventName, null, $data);
        EventManager::instance()->dispatch($event);
    }
}

We are now ready to add tasks into queue!!

Add events to the queue

Let’s take the same example of sending a welcome email on user registration event.

// in src/Model/Table/UsersTable.php
namespace App\Model\Table;
use App\Queue\QueueManager;
use Cake\Event\Event;
use Cake\ORM\Table;
class UsersTable extends Table
{
    public function register($user)
    {
        if ($this->save($user)) {
            $event = new Event('User.register', $this, [
                'eventOptions' => [
                    'userId' => $user->id
                ]
            ]);
            QueueManager::queue($event);
            return true;
        }
        return false;
    }
}

This will add User.register event to the queue and pass the userId as a parameter. This way we can bypass the mail sending process and send the response to the user instantly.

Our listener will handle the process of sending an email.

You can use this QueueManager in your controller’s action, a model’s function or callback.

Setup event listener

Let’s set up our event listener which are responsible to catch the event and perform the different operations.

Create a file under src/Listener/UserEventsListener.php with the following content,

<?php
namespace App\Listener;
use Cake\Event\EventListenerInterface;
use Cake\ORM\TableRegistry;
class UserEventsListener implements EventListenerInterface
{
    /**
     * Returns a list of events this object is implementing.
     *
     * @return array associative array or event key names pointing to the function
     * that should be called in the object when the respective event is fired
     */
    public function implementedEvents()
    {
        return [
            'User.register' => 'sendWelcomeEmail'
        ];
    }
    /**
     * Sends a welcome email to new user
     *
     * eventOptions
     *  - userId: User ID
     *
     * @param \Cake\Event\Event $event Event
     * @param array $eventOptions Event options
     * @return void
     */
    public function sendWelcomeEmail($event, $eventOptions)
    {
        $usersTable = TableRegistry::get('Users');
        $user = $usersTable->get($eventOptions['userId']);
       
       // Add your email sending logic here
    }
}

Now register this event listener in our application by adding the following line in config/bootstrap.php

use App\Listener\UserEventsListener;
Cake\Event\EventManager::instance()->on(new UserEventsListener());

Setup the worker

We have events, job queue and their listeners ready. Now we need a worker which picks up the job from queue and fires the event.

For this, let’s create a new Shell under src/Shell/QueueShell.php . This will extend the plugin’s QueuesadillaShell class, as we need to override getWorker method.

<?php
namespace App\Shell;
use Cake\Datasource\ConnectionManager;
use Josegonzalez\CakeQueuesadilla\Shell\QueuesadillaShell;
/**
 * Queue shell command.
 */
class QueueShell extends QueuesadillaShell
{
    /**
     * Retrieves a queue worker
     *
     * @param \josegonzalez\Queuesadilla\Engine\Base $engine engine to run
     * @param \Psr\Log\LoggerInterface $logger logger
     * @return \josegonzalez\Queuesadilla\Worker\Base
     */
    public function getWorker($engine, $logger)
    {
        $worker = parent::getWorker($engine, $logger);
        $worker->attachListener('Worker.job.success', function ($event) {
            ConnectionManager::get('default')->disconnect();
        });
        
        $worker->attachListener('Worker.job.failure', function ($event) {
            ConnectionManager::get('default')->disconnect();
        });
        return $worker;
    }
}

Setup system service

We now need to make this worker running in the background all the time. We use Linux and system services for this.

So, let’s create a new service under /etc/systemd/system/app_default_queue.service

[Unit]
Description=Default Queue Processor
After=multi-user.target
[Service]
Type=simple
ExecStart=/path_to_app/bin/cake queue --config default
Restart=always
[Install]
WantedBy=multi-user.target

Now enable and start this service.

$ cd /etc/systemd/system
$ sudo systemctl enable app_queue_default
$ sudo systemctl start app_queue_default // to start
$ sudo systemctl stop app_queue_default // to stop

Also, make sure that our cake shell is executable. If not, make it via,

$ chmod u+x [path_to_app]bin/cake

Done.

Now you can queue events and listeners as per your need and our worker will execute them in the background.

Happy coding!!

- The classQueueManager is originally developed by Jeremy Harris
- This article is based on CakePHP 3.x version.