Multi Server Maintenance Mode
A few days ago, Chris Fidao released the latest batch of videos in his fantastic series “Scaling Laravel”. This particular batch covered the process of creating and configuring the following server types:
- Load Balancer
- Application Server (2 or more)
- Queue Worker Server (1 or more)
- Database Server
- Cache Server (Redis)
- CRON Server (potentially unnecessary with Laravel 5.6).
But what particularly interested me, was he performed the above steps using Laravel Forge. If you’re not a server admin by trade, Forge truly is a god send. Chris was able to demonstrate this by having the entire, horizontally-scaled solution up and running within half an hour (minus server provision time).
You can watch it here: https://serversforhackers.com/scaling-laravel/forge
The videos answered all of my questions, except one… how to deal with maintenance mode in such a configuration?
PROMO: Check out our startup https://ceed.io — A template-driven script / code generator with support for functions, conditionals, iterations, schemas and more! Create scripts to generate common files such as forms, repositories, factories, validations etc. then execute them through the browser, cURL or in your IDE. Save hours of time by producing code to your design & style, while eliminating bugs!
Maintenance Mode Overview
If you’re not familiar with maintenance mode, check out the official docs: https://laravel.com/docs/5.5/configuration#maintenance-mode
In a perfect world, you would be able to push your latest code up to your source control system (typically
git these days) and have Forge deploy it without ever having to put your system into maintenance mode.
If you’re running a single server, that’s generally been my experience, as you’re updating a single source. Once it’s done, it’s available everywhere.
However, what happens when multiple servers are involved? Typically this kind of deployment does not always happen in sync so, for example, application server 1 could finish deployment before application server 2.
Depending on how the load balancer routes your traffic, a user could start a session on one server and then be bounced to the second (which does not possess feature X). The result is a frustration… “it was here a moment ago, what happened?”. This is likely followed by support tickets and wasted time.
In these situations, it really is beneficial to put each relevant server into maintenance mode before performing the upgrade. So, how do we go about doing this in an efficient, controlled manner?
Those looking quickly at this issue, might just be tempted to place the relevant maintenance mode commands:
php artisan down # start maintenance
php artisan up # finish maintenance
At the beginning and end of their deployment script e.g:
# set working directory
# enable maintenance mode (if possible)
if [ -f artisan ]
php artisan down
# retrieve latest code, run composer and reload PHP-FPM
git pull origin master
composer install — no-interaction — prefer-dist — optimize-autoloader
echo "" | sudo -S service php7.2-fpm reload
# post deployment
if [ -f artisan ]
# perform any migrations
php artisan migrate — force
# disable maintenance mode (if running)
php artisan up
While this will make sure that the server is not available / won’t process any queued jobs, you still run the risk that not all servers are put in maintenance mode at the same time. Ideally, you would want a setup like this:
ALLrelevant servers into maintenance mode,
- Perform code deployment to
ALLrelevant servers out of maintenance mode.
Since we can’t do that through the deployment script, it isn’t a good option.
Another option might be to implement a solution directly into your codebase, so that it can handle a multi-server deployment / upgrade.
While researching this article, I came across one written by Michael Raypold: http://www.raypold.com/post/laravel-distributed-maintenance-mode/
His solution involves overriding the
up commands, as well as the
isDownForMaintenance() method, so that instead of looking in the server’s local storage directory for the file that indicates maintenance mode, Laravel would look in a memory cache (Redis).
This addresses the problem of instant maintenance mode across all servers, as once you’ve called
php artisan down on one server, the result is propagated across all of them. I was all set to implement this, but after some additional thought, I decided against it for the following reasons:
- It requires modification of Laravel internals. Whenever you do this, you run the risk of breaking your app when upgrading Laravel. You can’t rely on the release notes saving you, as internal changes to the framework might not have an outward appearance to someone who hasn’t made an override, and therefore, they’re not documented in the upgrade guide.
- It is dependant on the cache server always being available during the maintenance process. In many scenarios, this might not be a problem, but suppose you wanted to perform work on the cache server itself? Maybe you need to reboot it, maybe you need to resize it / up its specs? You could handle an unavailable cache server and just assume maintenance mode, but what if it just crashed? All your servers would be unavailable.
I felt, given these reasons, I risked introducing a potential headache rather than a solution to the problem I had.
The third possibility (and my preferred approach), is the one that, ironically, requires the least amount of effort… use a Forge recipe.
Forge allows you to create shell scripts (known as recipes). They’re intended to be run following the provisioning of a server, but fortunately, they’re not exclusively limited to this purpose and can be run at any time. In addition, if you have multiple servers, prior to running a recipe, Forge allows you to select which servers to run the script on… perfect for our needs.
Thus we can create two very simple scripts, one to start maintenance:
And a second one to finish it:
Now, you have full control and flexibility with regards to maintenance mode. If you’re pushing a code change that doesn’t require maintenance mode e.g. you’re including some CSS fixes or addressing typos, then you can just push your code up to your VCS and let Forge handle the deployment process.
Alternatively, if you’re pushing major changes, need to modify one or more of your servers, or even add new ones to the mix, you can run the recipe. Once the script has run, you can proceed with the upgrade, safe in the knowledge that your site is inaccessible and that queued jobs will not run.
As an aside, you might want to modify the first script so that when it calls
php artisan down, it includes the optional flags:
Honourable Mention (Option #4)
One remaining possibility, is to modify your load balancer and point it to a static maintenance mode page. The problem with this though, is that Forge doesn’t give you an interface to do this, and since this article is about addressing this problem through Forge, I’ve left it out of consideration.
There are some additional caveats to this approach too, as Michael pointed out in his article, so be sure to read it if you’re considering this approach.
I want to stress that my preferred approach is
NOT something you should reach for
EVERY time you deploy changes. It is heavy-handed, as it brings your entire production environment down. In many situations, you can deploy your code without using maintenance mode, and when you can, you should.
We all strive for zero-downtime and whenever an option exists that can get us closer to that goal, we should take it.
That being said, when you need to make major changes to your production environment, whether they be to your codebase or server architecture, the recipe is a good option to prepare your setup for the upgrade. Just remember to run the second script afterwards, and ideally, inform your users about the downtime in advance. Nobody likes surprises!