Runbook: A Ruby DSL for Gradual System Automation

A simple runbook

A runbook outlines a list of steps required to perform an operation.

# restart_nginx.rbRunbook.book “Restart Nginx” do
description <<-DESC
This is a simple runbook to restart nginx
DESC
section “Restart Nginx” do
step “Stop Nginx”
step “Wait for requests to drain”
step “Start Nginx”
end
end
# Restart NginxThis is a simple runbook to restart nginx## 1. Restart Nginx1. [] Stop Nginx2. [] Wait for requests to drain3. [] Start Nginx

Adding automation

Moving past this initial outline, one can start to build automation into their runbook.

# restart_nginx.rbRunbook.book “Restart Nginx” do
description <<-DESC
This is a simple runbook to restart nginx and verify it starts successfully
DESC
section “Restart Nginx” do
server “app01.prod”
user “root”
step “Stop Nginx” do
note “Stopping Nginx…”
command “service nginx stop”
assert %q{service nginx status | grep “not running”}
end
step { wait 30 } step “Start Nginx” do
note “Starting Nginx…”
command “service nginx start”
assert %q{service nginx status | grep “is running”}
confirm “Nginx is taking traffic?”
notice “Make sure to report why you restarted nginx”
end
end
end

Features

Some of Runbook’s features include:

SSH integration

Runbook integrates with SSH using SSHKit to provide support for executing commands on remote servers, downloading and uploading files, and capturing output from remotely executed commands. You can control the parallelization strategy for execution, executing in parallel, serially, or in groups.

Runbook.book “Restart Nginx” do
section “Restart Services” do
servers (0..50).map { |n| “app#{n.to_s.rjust(2, “0”)}.prod”
parallelization(strategy: :groups, limit: 5, wait: 2)
step “Restart services” do
command “service nginx restart”
end
end
end

Dynamic control flow

We designed Runbook’s control flow to be dynamic; at any point you can skip steps, jump to any step (even a previous step), or exit.

Noop and auto modes

Runbook provides both a noop and an auto mode. Noop mode allows you to verify the operations your runbook will run before you execute it. Auto mode will execute your runbook, requiring no human interaction. Any prompts you have added to your runbook will use the provided default values, or the execution will immediately fail if prompts exist without defaults.

Runbooks can be executed in noop mode to describe what commands the runbook will execute

Execution lifecycle hooks

Runbook provides support for before, around, and after execution hooks. You can alter and augment your runbook behavior by hooking into the execution of entities and statements in your runbook. Hooks can be used to provide a rich set of behavior such as timing the execution of steps of a runbook or the runbook as a whole, tracking the frequency of execution of a runbook, and notifying Slack when a runbook has completed.

Runbook::Runs::SSHKit.register_hook(
:notify_slack_of_execution_time,
:around,
Runbook::Entities::Book
) do |object, metadata, block|
start = Time.now
block.call(object, metadata)
duration = Time.now — start
unless metadata[:noop]
message = “Runbook #{object.title}: took #{duration} seconds to execute!”
notify_slack(message)
end
end

First-class tmux support

Runbook.book “Restart Nginx” do
layout [[
[{name: :top_left, runbook_pane: true}, :top_right],
:middle,
{
name: :bottom,
directory: “/var/log”,
command: “tail -Fn 100 nginx.log”
},
]]
section “Setup monitoring” do
step do
tmux_command “watch ‘service nginx status’”, :top_right
tmux_command “vim /etc/nginx/nginx.conf”, :middle
end
end
end

Ruby commands

Runbook provides a ruby_command statement to dynamically define runbook statements and their arguments. You can, for example, hit a JSON endpoint to retrieve a list of servers and then execute a command on each of those servers. Because you are working in Ruby, you have access to all the parsing and processing capabilities it provides.

require 'json'Runbook.book "Restart Old Services" do
section "Restart week-old services" do
step do
server "monitor01.prod"
capture "curl localhost:9200/host_ages.json", into: :host_ages ruby_command do |rb_cmd, metadata|
old_hosts = JSON.parse(host_ages).select do |host|
host["started"] < 1.week.ago
end
old_host_names = old_hosts.map { |host| host["name"] }
old_host_names.each do |name|
command "shutdown -r now", ssh_config: {servers: [name], user: "root"}
end
end
end
end
end

Generators

Runbook provides generators, similar to Rails, for generating runbooks, runbook extensions, and runbook-focused projects. You can even define your own generators for including team-specific customizations in your generated runbooks.

Help instructions for the Runbook generate command

Adaptability

Runbook is designed to seamlessly integrate into existing infrastructure. It can be used as a Ruby library, a command line tool, or to create self-executable runbooks. Runbook adheres to universal interfaces such as the command line and ssh. Runbooks can be invoked via cron jobs and integrated into docker containers.

module MyRunbook::Extensions
module Aliases
module DSL
def s(title, &block)
section(title, &block)
end
end
end
Runbook::Entities::Book::DSL.prepend(Aliases::DSL)
end

Check it out

At Braintree, we use Runbook for automating our app deployment preflight checklists, on-call playbooks, system maintenance operations, SDK deployments, and more. We’ve found it to be instrumental in streamlining production operations, reducing human error, and increasing overall quality of life.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store