Dotfiles distribution

When faced with managing dozens of servers, keeping your configuration files up to date (or existent) becomes a real chore.

Vedran Mikulic
Apr 10, 2013 · 5 min read
Photo by Tim Evans on Unsplash

By Hans Tool, Vedran Mikulic

At Booking.com we saw many interesting ways people had solved this problem, but all of them had at least one flaw: a manual step had to be taken. Either configuration files needed to be pulled from an external source or pushed to the new host. When the last hackathon came around we decided there had to be a better way to have your configuration files delivered to the host upon login.

The problem

  • Network accessibility — some of our servers are in segregated networks and cannot talk to each other
  • Permissions of files when using a git repository
  • That pesky manual step
  • Getting the configuration files to their right places (or linking them)

The goals

  • The configuration files should be in git repositories. The benefits are clear here. An additional benefit to us was that the server hosting our git repositories is accessible from (almost) all hosts in the network, at least for read-only (HTTP) access
  • The entire mechanism should be opt-in. We don’t want to disrupt anyone’s current setup or introduce unnecessary steps (more than absolutely necessary)
  • The overhead for this should be minimized in order to have the least possible disruption to the login process
  • There should be no (or as little as possible) manual steps

The “dotfiles” repositories

So a single repository was out of the question. We decided on separate repositories for each user. This made managing them a breeze because each user owned their own repo. When a user opts in to using this mechanism, a fresh repository is cloned from a template on the server. This serves many purposes:

  • The template repository contains some sane defaults for ssh and git which will make life easier for new developers and those who don’t generally bother with setting things up.
  • There is a well defined place where you can take a look at how your colleagues set up their work environment.
  • Git experts can manage their environment with sufficient granularity.

With persistence out of the way, we still needed to store preferences and reference those preferences upon login.

LDAP

We settled on some basic options and additional data. The final attribute looks something like this and is a mix of settings and data that facilitates a faster login.

labeledURI: move_files=1,verbose=1,repo=dotfiles/htool.git,uid=htool, sha=32d8e90447687d75e1fe741d33c550e9b28a6cdc,update=1,branch=master,enabled=1

The sha parameter is the magic that makes it all work. To get that parameter in LDAP, we wrote a simple git hook which updates the users LDAP record with the SHA1 of the latest commit.

Finally, we whipped up a new section on the employee profile page in the staff portal to allow people to set these options for themselves.

The login process

When we see that a user has opted in, there are a few main scenarios:

  • New host — We need to clone to ~/.dotfiles
  • Known host and there are local changes — Raise a warning about the local change
  • Known host and LDAP SHA1 matches local dotfiles SHA1 — We are done
  • Known host and LDAP SHA1 differs from local dotfiles SHA1 — We need to pull the changes

How do we know that there are changes? That’s where the sha contained in the settings string comes into play. If you have been paying attention, you'll remember that we populate this with a commit hash after each push to the repository. At this point all that's needed to test if the repository needs a pull is to compare the SHA1 from the LDAP attribute to the last local commit. By doing this we have no need to do a git fetch (or pull) on each login, but only when there are changes present.

After the git pull or clone, there is a final step of linking all the files to their right places. For this there is a script in the ~/.dotfiles repository itself (remember, it was cloned from a template), which mirrors the files from ~/.dotfiles/dotfiles into the users home directory. If the script finds the same files present in your home directory, they are either ignored or moved to ~/.dotfiles/bit-bucket, depending on your “move_files” setting.

The user is also free to add additional steps to that script (after all, it’s in their repository) or source an additional script from there.

And that’s all there is to it: a little bit of git, mixed in with a little bit of LDAP and Perl

Usage

This was a big pet peeve that was resolved and a hackathon well spent.


Would you like to be an Engineer at Booking.com? Work with us!

Booking.com Infrastructure

Core Infrastructure in Booking.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store