Yo yo yo… Include Gearman and Redis in your PHP app

What can we do better ?

There comes a time when we don’t want the user to wait for a response but needs to do lot of processing(third party API calls, sending bulk messages, encoding uploaded videos, resizing uploaded pictures, sending push notifications etc).

How to do this ?

A general Request-Response follows the below pattern.

1. Get a HTTP request
2. Process it
3. Return the response

During the second step we may need to do any of those time consuming, resource hungry tasks. Some tasks, you may want to do on another machine. There are many ways to do these. IMHO all these have similar pattern of doing things.

Somewhere you submit a job(time consuming and/or resource hungry), something holds these jobs, at some other place somethings work this job. We need to do things asynchronously. We need a way to communicate between processes. Here comes inter-process communication, Message queues provides a way to do IPC.

Message queues are implemented in many ways, using different protocols. Most of them are based on polling and some of them are based on publish-subscribe. The main difference between these to types is how we get the job. The worker(which does the job) polls a queue to get a job or message in the first case and in later the job or the message is pushed(AMQP, ZMQ) to the worker.

All these are okay, what about the two bulls(redis and gearman) ?

Redis is an advanced key-value cache, store, data structure server with many advanced features like pub-sub, keys with a limited time-to-live and many more.

Redis can be used as message queue which we can use to do background processing. On the other hand Gearman is a job server(One thing it does perfectly). Gearman can also be used to do job distribution, to call functions between languages etc.

Redis can do all the things that Gearman can do, but the problem with redis is that there are many implementations(no standard protocol) to do the same thing. Sometimes your jobs are submitted from a different language than the workers. If you use any implementation based on redis there may be a problem with things(if the implementations do not match for the languages).

Redis can be used in lot more cases. (As a common session storage ?)

There are lot of ports(https://github.com/chrisboulton/php-resque) to the resque. There are bundles for Symfony. One of them is https://github.com/michelsalib/BCCResqueBundle. There are Gearman bundle(https://github.com/mmoreram/GearmanBundle).

With Gearman and Redis you can do foreground processing and background processing. With Gearman you can do parallel processing where you can execute multiple jobs in parallel as tasks. Great for speeding up lots of small calls which done in serial take a long time.

The workers run as process(multiple workers can do same job). If your workers are written in PHP, there will be a problem after long time(memory usage). Use supervisor to restart them after a number of runs. If your workers using Doctrine or Symfony components please read this.

One new thing I wanted to share is how to do a delayed job using Gearman. There is no native support to do this. Using redis we can submit a job to Gearman after a delay. Use redis to enqueue a delayed job, in the worker submit a Gearman job ☺.

PHP Snippet:

<?php

// get resque
$resque = $this->get('bcc_resque.resque');

// create your job
$job = new MyJob();
$job->args = array(
 'file' => '/tmp/file',
 'content' => 'hello',
);

// enqueue your job to run at a specific \DateTime or int unix timestamp
$resque->enqueueAt(\DateTime|int $at, $job);

// or

// enqueue your job to run after a number of seconds

$resque->enqueueIn($seconds, $job);

=======================================

<?php

namespace My;

use BCC\ResqueBundle\Job;

class MyJob extends Job
{

 private $gearman; 

 public function setGearman($gearman){ 

 $this->gearman = $gearman;

 }

 public function run($args)
 {

  //prepare a job with $args

  $gearman = $this->gearman->doBackgroundJob($jobName, $data_to_be_passed);
 }
}