eBay Tech Talk — How GitHub uses GitHub
A marvelously meta meetup
At our 36th #ebaytechtalk Meetup on October 20, special guest Johannes Nicolai from GitHub took the podium to show us How GitHub uses GitHub to develop and deploy GitHub.
Johannes shared an inside view of how GitHub’s globally distributed teams use GitHub tooling to develop GitHub, collaborate with engineering, sales, marketing, services and legal teams and deploy more than 500 times a week.
This post offers an overview of the evening’s themes and links to more information on each topic for further exploration.
What’s the GitHub secret?
GitHub has revolutionized how developers work together, so a lot of us were curious to hear how GitHub works themselves.
The short answer:
- Common Philosophy — or The Zen of GitHub
- Asynchronous Communication
- ChatOps
- Continuous Delivery
Johannes walked us through each of those points, summarized below.
The Zen of GitHub
GitHub calls its company culture The Zen of GitHub, expressed in a series of tenets[1] that describe the underlying principles and assumptions they aspire to:
Responsive is better than fast
It’s not fully shipped until it’s fast
Anything added dilutes everything else
Practicality beats purity
Approachable is better than simple
Mind your words, they are important
Speak like a human
Half measures are as bad as nothing at all
Encourage flow
Non-blocking is better than blocking
Favor focus over features
Avoid administrative distraction
Design for failure
Keep it logically awesome
These aphorisms seemed to ring true in a room full of developers and DevOps enthusiasts, and Johannes interpreted a few of the more cryptic koans for us — including the last one, which he revealed as a reference to the original company name.
Asynchronous Communication
With a company of around 600 employees — of which over half work remotely from various continents and time zones — synchronous communication in person is usually not an option.
For most of their own work, GitHubbers use issue threads, pull requests and GitHub Flow to manage their daily business in an asynchronous fashion while providing a persistent trail for anyone to follow — even if they weren’t involved in the conversation from the start.
No emails here — everything’s an issue.
Johannes gave us a sneak peak at a few of the issue threads used internally at GitHub to track all sorts of things — not just code.
While he was careful not to give away any precious secrets, we learned that the GitHub legal team uses pull requests to track changes to contracts — which begs the question why more lawyers don’t do this‽
Even the marketing team tracked the whereabouts of the evening’s swag shipment in an issue thread, with links to the parcel tracking site and the local customs office that was holding up the Octocat stickers everyone came for…
Johannes highlighted the importance of ReadMe files as the “virtual doormats” that welcome users to a project, introduce the team and provide context for new hires. The Professional Services team that Johannes works for helps customers get set up with GitHub Enterprise and trains people how to use Git and GitHub.
He pointed us to their blog post on How the Services team uses GitHub, that describes how they use their own team repository as the main interface for interaction:
We have files that document how we work, including everything from checklist-formatted Markdown files for onboarding new team members to guides on how to configure LDAP for our customers on GitHub Enterprise.
The blog post provides an astounding range of examples that show how GitHub as a whole (and this team in particular) really eats their own dog food and believes in the tools they use to support collaboration and transparency:
We try to ensure the decision-making process and our team’s initiatives are as collaborative and transparent as possible. We open an issue to discuss any topic that needs our attention. We try to default to having conversations in the open and with a URL.
Transparent conversation is especially necessary as a geographically-distributed team. […] It also lets people understand the why of a decision even years after the decision occurred.
We heard how heavily @mentions are used to notify teams and bring people into the conversation when their input is needed. It sounded almost unbelievable, but as a testament to the power of mentions in their workflow, Johannes revealed there are actually more teams than people at GitHub, so people can get the right peoples’ attention by mentioning their team handle.
For more information on how other teams use GitHub, Johannes also recommended these blog posts:
ChatOps
In the next portion of his talk, Johannes introduced us to Hubot, GitHub’s hardest-working (robot) employee, who hangs out in the company chat rooms on Slack.
GitHub is big on what they call ChatOps, or “conversation-driven development” as the “cultural glue” they use to manage technical and business operations using a chat bot that helps people do their jobs, reduces friction between development and operations and brings new engineers up to speed quickly.
We watched Johannes use Hubot[2] to retrieve log files, graph various metrics and deploy pull requests directly from Slack to Heroku.
This part of the evening was particularly inspiring, and many of us came away curious on how we might be able to build this kind of ChatOps environment for our own projects.
Continuous Delivery
The common philosophy, asynchronous communication and ChatOps approaches all come together in GitHub’s take on Continuous Delivery.
Anyone that has code ready can ask Hubot where the changes can be deployed. The chat bot responds with an overview of deployment environments and shows the status for each environment so anyone can easily see which are blocked, the branches being tested, and who’s using them.
Multiple environments — A dedicated branch-lab environment is always available for testing, so any branch can be pushed to evaluate the impact that the changes would have if deployed to production.
The deployment API reports on the status of each deployment event in the pull request, which provides context and links to the results so participants can easily verify the outcome of each change.
If the changes pass initial testing, they can be deployed to a “production canary” environment for rollout to a small subset of production machines as a precursor to general availability.
Dashboards and monitoring — A dedicated “confidence dashboard” plots performance and number of exceptions over time so users can easily see whether changes cause any alarming spikes that indicate critical issues.
Once developers are reasonably certain their code is behaving as intended, the changes can be deployed to everyone in the production environment and merged to master.
A “haystack” timeline correlates recent deployment events and exceptions, so developers and operations team members can quickly pinpoint the causes of any anomalies in production.
If any of the Nagios monitoring jobs exceed pre-defined thresholds, Hubot automatically opens an issue to ensure the event is recorded and addressed.
Flipping features — and more metrics — GitHub also uses several dedicated tools to provide fine-grained control over which features are deployed, and evaluate the performance of refactored code:
- Feature toggles are used to change the ways in which individual features are enabled. The Flipper library allows features to be rolled out gradually over time, or to a certain pre-defined group of users.
- The Scientist library assists in large-scale refactoring efforts by comparing the current and refactored behaviors under load.
A recipe for success
It was refreshing to see how thoroughly GitHub has embraced the notions of asynchronous communication, ChatOps and continuous delivery and how well these approaches are integrated into their corporate culture.
Many in the room seemed inspired to implement similar methods in their own teams and projects, and those of us at eBay and other companies on this path came away with a reassuring feeling that we’re headed in the right direction.
Further reading
To learn more about these practices, visit the GitHub Engineering blog, or add the RSS feed to your favorite feedreader.
- Inspired by the guiding principles for Python’s design:
http://legacy.python.org/dev/peps/pep-0020/ ↩ - Note to self — It’s pronounced hew-bot (hyo͞o-bot), not hubbot! ↩
