Image for post
Image for post

An annoying Capistrano, Unicorn
and Bundler issue

Kir Shatrov
Mar 22, 2015 · 3 min read

Authors: Kir Shatrov, Lead Developer at Evil Martians, current Capistrano maintainer; Kirill Kuznetsov, Lead of Operations Team at Evil Martians.

We’ve moved! Check out this post — and other posts by fellow martians—on our new blog, Martian Chronicles!

TL;DR: Always use an absolute path to Unicorn in Capistrano scripts; otherwise your application server will fail to restart after a number of deploys because of Capistrano release cleanups.

Last month, we came across an interesting problem: sometimes after a successful deploy of one of our Ruby on Rails 4 applications, a Unicorn server could not be restarted using zero downtime deployment signal. That affected all environments we run the application in — unfortunately, including production. As a result, we got annoying exceptions like ActionView::MissingTemplate in the production environment. Only a hard restart of Unicorn would fix that kind of exceptions.

We had no idea what could be the source of the issue, so we decided to check this case out using our monitoring system (Zabbix) to catch such incidents immediately and act on it.

Here is a simple Rails code snippet we've used to detect if Unicorn was properly restarted:

Note that the REVISION file is written by Capistrano on every deploy, so we can simply compare the contents of the file with the current application revision we have in memory.

And here is a snippet of Zabbix configuration we've used:

Next, we’ve created a Zabbix agent item (“Revision status check”) and used it for a trigger:

Now, how does this trigger work, exactly? It checks for a non-200 HTTP response and ensures that it persists during a period, to prevent false alerts during deploys and rebouncing. Google “Hysteresis in Zabbix” for more details, if you're interested.

The first alert we've got led us to the following output in the Unicorn log file:

And at that moment I’ve started to get a clear understanding of what’s going on.

Here is how zero-downtime Unicorn deploy works: it starts the master process and replaces it with a fork on the next deploy.

On the first deploy, bundle exec unicorn_rails command was mapped to /home/project/releases/1/vendor/bundle/ruby/2.1.0/bin/unicorn_rails, which started properly. But later, let’s say on 100th deploy, the original Unicorn still pointed to the /home/project/releases/1 directory from the first release. By default, Capistrano 3 keeps only the last five releases and cleans up older releases (deploy:cleanup task runs automatically by default), and obviously /home/project/releases/1 did not exist anymore.

Since /home/project/releases/1/vendor/bundle is just a symlink to /home/project/shared/vendor/bundle, we had to configure Bundler to map commands to the shared directory — which exists forever.

Here is a problem: BUNDLE_PATH is relative, and it maps to /home/project/releases/1/vendor/bundle, which would be destroyed later after a few deploys.

And the fix:

In Capistrano 3, this patch can be applied by setting the `bundle_path` option:

Done.

In spite of my history of contributions to Capistrano and Bundler, it was one of the most interesting bugs I’ve found recently.
We've decided to keep monitoring Unicorn revisions to catch any failed restarts in future.

Evil Martians

The Martian Chronicles

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store