Adventures in webhooks

Automated deploys to a preview environment and blog posts that practically render themselves without a massive pivot in tech stack or increased cost.

A week ago I joined the startup EuMotus. As engineer one, I am inheriting a static website that has historically been manually updated and uploaded over SFTP. One of our first urgent tasks was to get a blog up, the details of which I will cover in a future post here. For now, I summarize:

While a full-featured CRM with WYSIWYG editing would have been great for our content creators Boris & Sophia, it would also have meant a significant amount of tech effort to set up and maintain, and that in competition with other priority claims on my team-of-one’s time. So for now we went with jekyll for its ease of setup, github for storage/version control, and prose.io for content editing. And after some tech massaging, the first post went live.

Give ‘em a branch

Already in the first days of blogging, as I worked on functional website changes and editors worked on posts, I ran into irksome code base collisions. This will very likely quiet down as our structure and flow normalize, yet the immediate impact was time wasted on the meticulous resolving of content merge conflicts.

To mitigate, I created a branch called blog-edits so edits could be made independently from master. For now, editors select the branch in prose.io and go to town in the _posts folder. For images, I created an includeable template that standardizes styling that they can {% include image.html ... %} in their posts, though these don’t render in the prose.io preview.

Two implications of this setup: (1) going live is a manual process of pulling down the master branch, merging blog edits, building the jekyll site, and uploading over SFTP, and (2) there’s no easy way for the editors to see their content with images and fully styled until published.

With an emphasis on low-hanging fruit, I set up a second quick-and-dirty web instance as a preview environment and began uploading previews of the full jekyll render for team review. This somewhat addressed implication 2, but the process remained a manual one and in fact doubled the work required to deploy end-to-end.

Lift with your legs

The more I can empower the editors to publish without me, the more I am free to focus on other engineering. In addition, automating the deploys to the preview environment saves me from all those manual steps for functional changes too. In previous companies, I’ve had teams of support people providing tools to streamline my workflows — not here. Like Fezzik and the Brute Squad, I am the DevOps, at least for now.

Here is how I bootstrapped a simple preview deployment flow from scratch:

1. Created an ssh key on the preview host to be used as a deploy key in the github repo and pulled the repo down into a known location on the preview host
2. Grabbed the adnanh/webhook tool to listen on the preview environment for deploy events
3. Set up a basic hooks.json file to run a redeploy.sh script that pulls the latest content from the repo, builds the jekyll project, and updates the web root
4. Added the webhook definition in the github repo setting
5. Added basic papertrail logging of the webhook process, a nice to have

Once again, with feeling

Adding the deploy key to github was familiar, like any other public/private key pair use.

To integrate the webhook across github, the preview environment, and papertrail, I started with a mock redeploy.sh script (that does an echo testing) and minimal hooks.json configuration file:

[{
    "id": "redeploy-webhook",
    "execute-command": "./redeploy.sh",
    "command-working-directory": "/path/to/webhooks"
}]

For now I am running the webhook tool in a detached screen session with the command stdbuf -oL -eL /path/to/gobin/webhook -hooks hooks.json -verbose >& /var/log/webhooks.log. stdbuf wraps the tool invocation and causes each line of stdout and stderr to flush to the log file immediately.

After listing the log file path /var/log/webhooks.log in /etc/log_files.yml and following papertrail’s quickstart, a simple sudo remote_syslog forks the process that watches the log file and uploads to their web interface.

Once I could verify that a commit in our repo caused a testing to show up in papertail, it was time to add the gas. I updated the redeploy.sh script:

cd /path/to/repo
git fetch origin
git checkout master
git branch -D deploy
git checkout -b deploy
git merge origin/blog-edits --no-commit
git commit -m "deploy"
bundle install
bundle exec jekyll build
rsync -r --delete _site/ /var/www/html/

Assuming there are no merge conflicts, this pulls the latest state from github, merges in any loose blog edits, renders the jekyll output, and then updates the web root in place (note the--delete flag). In a matter of seconds, an editor saves a change in prose.io and sees it fully rendered in the preview environment.

The Moral

Things I got to play with from scratch for the first time:

• github deploy keys and webhooks
• adnanh/webhook tool
• Papertrail

TODOs that we may or may not get to:

• fine-tuning the webhook behavior (e.g. only trigger a deploy on a push to the master and blog-edits branches)
• setting up the webhook as a systemd service so it starts up on system boot, replacing the screen session
• applying a similar flow for production deploy

This solution leaves a lot to be desired, but for this early-stage startup, it’s light and right for now. 🅳