One parser to rule them all

Bash is a wonderful and terrible language. It can provide extremely elegant solutions to common text processing and system management tasks, but it can also drag you into the depths of convoluted workarounds to accomplish menial jobs.

Recently I ran into one of these situations when trying to parse varying inputs in a control script. The standard tools for argument parsing were breaking at every turn as I changed argument order and added flag options for flexibility. …

Use Git hooks to create your own push deployment pipeline

I recently got interested in supporting Heroku style push deployments in the AWS stack at Fullscreen, Inc. There are a few solutions for this in the wild like Dokku, but I wanted to use something flexible enough to grow with our deployment infrastructure that wasn’t dependent on container technology. The solution I came up with involves a deploy server, a Git hook and a bit of Bash.

Configuring the Server

Before I could have engineers start pushing code, I needed to setup a deployment server. The most basic setup involved a new instance with a “git” user for authenticated pushes over ssh. …

If you haven’t heard of it, AWS’ CloudFormation service allows you to define complex application stacks in JSON templates and offload all of the resource creation and management to the cloud. It’s usually a combination of your best friend and your worst enemy due to a lack of flexibility and transparency into what will happen when a template change is applied to a stack.

An especially annoying aspect of the JSON templates is that there is no way to leave breadcrumbs detailing why your resources are configured the way they are. JSON as a spec does not support comments, and in order to future proof their own service AWS doesn’t allow custom keys in resource declarations (e.g. …

One of SaltStack’s many strengths is it’s extensibility. It’s possible to write custom states, execution modules, grains, pillars, renderers and returners all in python and all with the power of SaltStack’s internal modules. Let’s take a look at writing a custom module that decorates the pillar with information from an external resource. For this exercise, we’ll create an external pillar that reads data from a Redis database.

Configuration

We’ll start by creating our external pillar module file. In your Salt base or root directory, create a folder called “modules” and a sub-folder called “pillar”:

/srv/salt $ mkdir -p modules/pillar

Next, we’ll create our pillar…

Provisioning and configuring multiple nodes with a tool like SaltStack is often trivial once your recipes are well defined. Things can become more complicated though when you start to deploy code or critical system configuration changes.

The SaltStack orchestration runner is a salt runner that executes commands across targeted minions from the salt-master. Because commands are not run by your minions and orchestration can make use of the requisite system, all kinds of possibilities open up for state enforcement and error handling.

The real power of the orchestration system becomes apparent when you start dealing with code deployments. In a typical application update, new code is deployed to new or existing machines, any application requirements are installed or updated and the application is gracefully reloaded. Each of these steps represents a failure point that could leave your machines running different versions of the same code. …

The internet is rife with promises of 100% availability when using HAProxy for load balancing. THEY ARE LIES!

When you instruct HAProxy to reload it’s configuration, the following occurs:

1. A new process is created that signals the old process to unbind it’s ports
2. The new process binds to the ports
3. Existing connections are finished on the old process, or passed on to the new process

Seems pretty straight forward but there’s a small window of time between steps 1 and 2 where neither process is bound to the configured ports and requests are rejected. …

The git commit log is one of the best tools for understanding the history and direction of a project. I’ve found there are a ton of situations where the log can make complex interactions with a repo a breeze.

Sometimes a branch you want to update can fall too far behind it’s base branch and a merge could introduce potentially unwanted commits. To find the difference in commits between two branches, use the —cherry-pick option:

git log --left-right --graph --cherry-pick --oneline master...production
< f472340 unshared commit in master branch
> 284f4b9 unshared commit in production branch

The —cherry-pick extracts commits that exist in one branch but not the other, while —left-right indicates which side the commit belongs to. …

If you haven’t heard about Ansible yet, it’s worth giving the docs a read before proceeding.

As my deployment strategies grow in complexity, I find that I really need Ansible to be aware of environment configuration. By default Ansible doesn’t provide a mechanism for managing multiple environments, but with a little hacking we can get a pretty comprehensive solution.

My directory structure looks something like this:

ansible/
 production/
  group_vars/
   some_group
  inventory
 roles/
  some_role/
 staging/
  group_vars/
   some_group
  inventory
 servers.yml

Each environment has it’s own directory that contains a dynamic inventory script and a group_vars directory. This allows two things to happen:

1. I can deploy to specific machines based on…