What can github teach us about continuous delivery?

So most people have probably heard of github — darlings of the open source world, pioneers of social coding and ubiquitous source control solution of choice for the vast majority of open source developers. If you work with any modern open source system these days, chances are that github is a site you are extremely familiar with. As an organisation though there may be some things about github that will surprise you;

  • github was founded in 2008 by three guys but now has a staff of almost 600 people.
  • Their staff are based all over the globe — they have people in Australia, Japan, Europe… one person in Hawaii… in fact only 44% of their staff are actually based in their San Francisco HQ.
  • They even have people who don’t have a fixed abode, either country-hopping or just living and working out of an RV.
  • And some of these people, despite living on the other side of the world, are on the same team!

And yet even with these supposed impediments github.com manage to ship high quality software at an astonishing rate — on average we see 80 changes to the production platform every day! Github employee Alain Hélaïli recently gave a talk at Devoxx 2016 where he went over some of the secrets and techniques that github use to overcome the difficulties of being spread so far and wide.

The art of the conversation

It’s good to talk — but as Alain was quick to point out during his presentation, it’s important to talk in the right way. The golden rule at github is that if you can not provide a link to a conversation, then that conversation did not happen — and the ability to do this is baked right into the github tools. Make a comment on any pull request or issue, and you will be able to get a permalink that will take other users straight to it.

This approach isn’t intended to facilitate some kind of blame culture — where you are suddenly able to prove your manager wrong because you have a link to the conversation he denies having with you — rather it is a means of modernizing conversations and, importantly, conducting them in an asynchronous manner. Not only that, but if you are using an integrated set of tools (such as github) then you will be able to reference not only other users but also issues and even lines within your code base directly within your conversation, thus aiding the discussion by virtue of it happening right next to the relevant resources.

The easiest way to explain this further is to highlight some of the problems with traditional forms of conversation. So let’s say you want to talk about code with one of your fellow developers. How do you engage with them?

Send an Email!

Emails are an awful way of having a conversation. Sure, things start off well — you send your question over to your colleague and he responds but that prompts more questions and before you know it you have dozens of back-and-forth mails between the two of you. Three or four months down the line are you confident of being able to find anything that was said? Also, if you do manage to find the emails how do you know if you’re at the end of the conversation? Come to think of it, after we copied in the various other people who became involved there were like, 20 people involved… and some of those guys actually forwarded the mail to people who weren’t even involved originally and…. Aaaaargh!

Before you know it, your “conversation” is actually hundreds of separate emails spread out over several days if not weeks and the only way you have to piece them together into any kind of flow is to become intimately familiar with the querying tools of your favorite email client. Yuck.

So sure — send an email, but how about if that email is a link to a conversation you are having asynchronously, maybe on a github issue? “Hey Bob, come and join in the discussion” or “Hey Brenda, would you mind hopping into this discussion here and giving us some insight?” that way it is only ever one email and the conversation is much easier to manage and refer to.

Have a Meeting!

Well, sure, gather together 10 people to chat about a common solution you’re trying to come up with. Let’s get the web designers, the developers, the project manager of course, the business analysts, the tester… let’s take them all away from their desks for an hour and have a meeting!

Better yet — let’s have the meeting at 10am — this gives everyone a chance to “settle in” plus I can see that most of these people I want to invite also have a meeting at 13:00 so this will break the day up nicely right? Wrong!

Developers are always scheduling their time around how many interruptions they will be subjected to. If a developer turns up for work at 8am and has a meeting at 10am, they probably aren’t going to schedule in any significant tasks for that 2 hour period. It will instead be filled with a smaller, perhaps less important task. Pepper your day with meetings, and you essentially condition yourself to prioritise small tasks.

So meetings are useful, but it all depends how you approach them. Firstly, remembering earlier the golden rule of no conversation existing without a link, a meeting should be documented. A great place to do this might be on your company issue tracker (which might even be github). That way, people who weren’t in the meeting get a chance to see what was said and comment on it. If we are operating under these parameters, where meetings become issues and issues get shared, this begs the question then as to whether everyone and their dog needs to attend the meeting or even whether the meeting needs to happen at all.

Just tap someone on the shoulder!

Absolutely not! Even the three founders of github accept that when they first started, despite all working in the same office, they very rarely actually spoke to each other. The reason: an interruption is bloody expensive.

Based on research by gamasutra it takes a programmer around 10–15 minutes to get back up to the pace they were at prior to an interruption. The same study suggested that on average a programmer will only get one 2 hour uninterrupted period per day — costing potentially hours of productivity.

If your shop is structured in such a way that programmers need to be interrupted for the cogs to keep turning, then you need to restructure. Instead, use your tools — tag them in an issue, they will see it when they reach a convenient stopping point and reply when they have the time. Prepare in advance and raise your questions and comments at the daily stand up meetings. Just don’t interrupt!

Applying conversations to workflow

Here is a common example of what can go wrong in a development workflow, hopefully this doesn’t sound too familiar;

  • The team have a planning session, everyone is given individual tasks to complete.
  • John goes back to his desk and begins the implementation of his task.
  • A week later, John is finished and creates a PR. He asks his team for a code review.
  • Jane, now seeing John’s code for the first time, immediately spots a flaw in his implementation. She explains that John did not need to reinvent the wheel and points him in the direction of several pieces of technology that would do the same job more efficiently using half the code.
  • The team agrees with Jane, and John slowly begins to weep.

So what went wrong? Well, the problem here was caused once again by a lack of conversation. The first time anyone apart from the developer saw the code was at the point that it was ready for review! This is an incredibly common issue (i’ve seen it in shops I’ve worked in, as I’m sure you have too). So what should have happened?

Github actually have their own workflow (the github workflow) that looks a little like this;

  • Open a feature branch
  • Make a few commits — these could just be comments, or minimal implementation — the main aim is to…
  • Open a PR as soon as possible!
  • Have a conversation on the PR
  • More commits
  • More discussions
  • Repeat until requirements met and everyone is happy
  • Ship it by merging back to master or wherever you ship to.

Much less likely to waste time — everyone in the team is talking about the problem as early as possible. When asked during his presentation, Alain surmised that the average github employee spends an average of 2 hours a day talking about code rather than writing it.

Github take this principle very seriously — they even have a little chatbot called hubot that integrates dev-ops directly into their IM tools (they call it chat-ops). Hubot can kick off builds and things for them directly in the middle of a conversation. Pretty neat!

Have a common philosophy

Something seldom established in teams is a ‘common philosophy’ — a set of principles that you all collectively agree on. At github, they call this their “zen” and I’m just going to paste it here without comment. Some of it is a little wishy-washy (some isn’t) but the idea of having a common philosophy is a solid for a number of reasons. Firstly, as Alain pointed out in his presentation, if you do not agree on a philosophy then clashes will eventually happen and these will in some way manifest themselves in your code. Having a shared philosophy can also prompt and inform discussion within a code review and also serves as the basis of a conversation you can have with yourself when approaching a particular problem.

Here it is then, “The Zen of Github”;

  • Responsive is better than fast
  • It’s not fully shipped until it’s fast
  • Anything added dilutes everything else
  • Practicality beats purity
  • Approachable is better than simple
  • Mind your words, they are important
  • Speak like a human
  • Half measures are as bad as nothing at all
  • Encourage flow
  • Non-blocking is better than blocking
  • Favor focus over features
  • Avoid administrative distraction
  • Design for failure
  • Keep it logically awesome

Conclusion

I know I am preaching to the converted here — chances are that this is exactly how you are already working but if you aren’t, and you are thinking that this is going to be too tough to introduce at work, all I can say is that firstly you have my sympathy, and secondly, just give it a go. Show this article to your team. Convince them to give asynchronous conversations a try. If github can manage it with 600 worldwide employees, there really is no excuse for a team of 5 in the same building!