Laravel Horizon — tinkering with `balance` config and process creation

This is a post about me tinkering with Laravel Horizons balance config option. Contains my thought process and a bunch of screenshots. You should probably start by reading the more condensed version first:

Laravel Horizon — Number of Workers and Job execution Order

Strategy: balance=simple

// config/horizon.php
...
    'local' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['high', 'default'],
            'balance' => 'simple',
            'processes' => 1,
            'tries' => 0
          ,
        ],
    ],
...

When I run php artisan horizon to start the Horizon service, things get processed in an unexpected order. What’s going on? I said I only want 1 process, but things are being executed at the same time from both queues.

I’m using Homestead for my local environment. Let’s take a look at how these processes are being spawned by Horizon.

1. Login to Homestead with homestead ssh This only works if you are using homestead AND you setup this function.
2. Now we will run top to view the processes.

Instead of just running top we can filter the results to focus just on the php related processes with this command:

# This command will get all processes with `php` in the command.
top -c -p $(pgrep -d',' -f horizon:)

For my horizon config with balance=simple and processes=1 here is what I get:

Not what I expected, but cool! Let’s have a look at the commands it generated:

balance=simple AND processes=1

In the first command it says --max-processes=1 --min-processes=1 . Looks like Horizon’s simple balance strategy is actually setting a min and max and it will spawn workers as needed for both queues. That could come in handy.

Let’s increase processes to 2. With this, I would expect to now have two processes per queue.

// config/horizon.php
...
'local' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['high', 'default'],
        'balance' => 'simple',
        'processes' => 2,
        'tries' => 0
      ,
    ],
],
...

restarted Horizon and….

balance=simple and processes=2

Not what I expected. You can see that --max-processe=2 now, so that’s good, but what does it mean? Let’s put some jobs in our queue and see if horizon automatically spawns the max processes to get things rolling.

The results? Nothing! I end up getting only one process per queue and just to confirm, only one job is being processed per queue. My highqueue only has a single job, once it completed the worker did not switch to my default queue. That’s fine, I think that is expected and the auto strategy will handle moving workers from one queue to another.

Since I’m not sure what is going on, let’s experiment

'processes' => 3,

which gives us:

balance=simple AND processes=3

Now we are getting somewhere. max-processes=3 and we do in fact only have 3 processes. 2 are dedicated to the high queue and 1 process is dedicated to the default queue.

Looks like Horizon is obeying the max-processes, but it will also make sure that you have least 1 worker for each queue. Let’s try a few more things to confirm this

'processes' => 7,

balance=simple AND processes=7

Looks that confirms my theory. Let’s make processes=2 and add a few more queues just for fun.

'queue' => ['high', 'default', 'emails', 'notifications']
'balance' => 'simple',
'processes' => 2,

As expected, even though we have a max-processes of 2 we get 4 workers running:

Balance=auto

Let’s give balance=auto a try and see what it does. We will start off with this config:

// config/horizon.php
...
'local' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['high', 'default'],
        'balance' => 'auto',
        'processes' => 1,
        'tries' => 0
      ,
    ],
],
...

which gives us:

balance=auto AND processes=1

Looks pretty similar to the balance=simple strategy when processes=1

Since auto is supposed to reallocate workers based on how many items are in each queue. We will put 10 jobs in the default queue and 1 job in high to see what happens. The jobs are pretty simple, they just sleep for 10 seconds.

// my test job, when I dispatch, I give it a unique ID so I can
// track it in the log.
private $myId = null;
public function __construct($myId)
{
  $this->myId = $myId;
  Log::info('constructed: ' . $myId );
}
public function handle()
{
  Log::info('started: ' . $this->myId);
  sleep(10);
  Log::info('finished: ' . $this->myId);
}

And, it works exactly the same as the simple strategy. There is still one process for each queue. Maybe this is because we need more jobs before the Horizon thinks it should reallocate the worker. Or maybe it is because processes is set to 1, and Horizon will only allow 1 per queue but at least 1 per queue, this seems more likely. Let’s test it.

'processes' => 2,

Increasing to 2 didn’t change anything. Let’s try dispatching 2,000 default jobs and 1 high priority and see what happens

Nothing happened! Still 1 process per queue. Let’s try this:

'processes' => 3,

Finally something different! Now we have 2 processes for default and 1 for high Note this is a little different than the simple strategy. At this point we still have a just over 1,000 jobs to process so Horizon is dedicating more more workers to that queue even though the other queue is defined as a higher priority queue.

And just to further confirm this, let’s increase the processes to 6, add 1,000 default jobs, then after a minute, add 1,000 high priority jobs

'processes' => 6,

After adding 1,000 default jobs:

balance=auto AND processes=6 AND added 1,000 default priority jobs while there is only 1 high priority job

balance=auto AND processes=6 AND added 1,000 high priority jobs while there are about 980 default priority jobs

This looks great! Horizon is switching the workers from one queue to another based on the number of jobs in each queue. Even if there are 0 high priority jobs, Horizon makes sure there is at least 1 worker for that queue.

This is super helpful to know. If you setup 10 queue and set your processes to 1, you will actually have 10 processes running at all times. Even if you set your processes to 9, you will still have 10 running, 1 for each queue and the balance=auto feature won’t do any automatic switching unless you have excess processes