Cattle, Pets and Pink Eye
You’ve likely heard the analogy of cattle versus pets with respect to running servers. While we used to nurture our precious service instances in production, in the age of Docker and Kubernetes we work dispassionately with many instances.
In the old way of doing things, we treat our servers like pets, for example Bob the mail server. If Bob goes down, it’s all hands on deck. The CEO can’t get his email and it’s the end of the world. In the new way, servers are numbered, like cattle in a herd. For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line.
We’re getting good at this new paradigm. But we’ve left a big gap in our development and delivery. We not only have lots of running services — we have lots of codebases defining them. Those codebases, too, need to be treated as cattle rather than pets. We need the ability to apply changes to them in a consistent manner.
We adopt a cattle versus pets model for running services because there’s no choice at the scale of modern microservices. But we also need the same capabilities for our code bases and delivery flows. The love and nurturing of an IDE is no longer enough. We need an API for software.
Protecting the Herd
One of the first considerations in industrializing cattle production is protecting the herd. The approach we use to look after our pets’ health doesn’t scale.
When I was a boy in Australia I spent time on farms and remember helping to prevent pink eye. Pink eye (veterinary name, “infectious bovine keratoconjunctivitis”) looked and sounded horrible. We drove cows through cattle crushes to protect them by spraying their eyes.
I remember wondering at the efficiency of the process, while being shocked at instances of needless cruelty.
The other piece of technology required was the working dog, like this blue heeler. I learnt from experience that “it was originally bred to herd by biting, and is known to nip running children.”
What’s the equivalent of pink eye for our projects? As with a herd of cattle, there are many preventable health issues in our project herd: Security vulnerabilities, outdated libraries and configuration, code defects, things we’ve learned in one project and need to apply elsewhere… How do we achieve similar efficiency in overcoming them?
One approach is the monorepo: Have many services, but a single repository containing all the code. This solves some problems but creates others. For example, we lose the benefits of the granular access control of source control platforms like GitHub and BitBucket.
What’s the equivalent of a cattle crush and blue heeler for our codebases? It’s the API for software: The ability to make code and configuration changes across many projects, without needing a stethoscope and bed-side manner and chew toys.
Consider the problem of outdated dependencies. We know from the Equifax breach (caused by an old version of Apache Struts) just how costly they can be. We need a scalable way to protect our codebase herd against them, as otherwise we’ll waste much valuable developer time or — more likely — let things slip until we are at risk.
The equivalent of the vet with the stethoscope and chew toys is doing this project by project in our IDE, and manually pushing and raising a PR:
An example of updating a herd using the API for software:
export const tryToUpgradeSpringBootVersion = dryRunEditor(
params => setSpringBootVersionEditor(params.desiredBootVersion),
UpgradeSpringBootParameters,
"boot-upgrade", {
description: "Upgrade Spring Boot version",
intent: "try to upgrade Spring Boot",
},
);The setSpringBootVersionEditor is a simple, testable, function using the Atomist Project API that edits a Maven POM to upgrade the Spring Boot version. It’s straightforward to write such editors to perform custom operations on code or configuration.
For simplicity, I showed it running against one project, but we can run this command across as many projects as we want in a single operation.
Not only is the change automated: it’s verified. Given we’ve used the dryRunEditor wrapper function, if the project builds successfully after the update a PR is raised. If the build fails, an issue is created, as the team should know about the problem, but not be presented with a PR that should never be merged.
Changing Processes
The ability to change code across many projects is important, but not sufficient. We also need to be able to change the processes around that code: in particular, how it gets to production.
The Equifax process was “weak and broken” because it failed to identify the vulnerable systems. This process is supposed to run every day; not only did they not identify the vulnerability immediately, it went unidentified until they found the breach. It was too late.
We need the ability to change not just many codebases, but many delivery flows.
When we have a better preventative treatment schedule for pink eye, we’d like to apply it to our entire herd immediately. We can’t go to every single cow in the field and apply the new treatment to it individually. But that’s what we tend to do with our codebases today. Our hundreds of services have hundreds of build files, meaning we need to make hundreds of changes to change what is essentially one thing. For example, if we want to do a security vulnerability scan, how many places do we need to make that change?
A better approach is to add a “reaction” that performs the necessary scanning on every push. An Atomist software delivery machine makes this easy through domain-specific listeners. The following example checks for a README file (using the same Project API as in the Spring Boot version update function) and complains in any linked Slack channels if one isn’t found:
softwareDeliveryMachine.addPushReactions(async pli => {
const readme = await pli.project.getFile("README.md");
if (!readme)
return pli.addressChannels("Your project has no readme");
})Now every project will immediately benefit from the new check. The same approach works to integrate security scanning or static analysis tools — usually raising an issue or redirecting to a dashboard.
Breeding New Services
Breeding most types of cattle has been industrialized, for efficiency and to ensure consistent qualities in the offspring.
This, too, applies to our software projects. In a world of many services, we need to create them in a consistent manner and ensure that the best genetics are perpetuated.
Many companies invest in building in-house tools to handle this, but Atomist provides a unique and powerful approach. The benefits parallel those in stock management:
- New projects are based on the best genetic material — the correct set of libraries and configuration, referred to as a “seed project” in Atomist parlance. We can have as many seeds as we have distinct types of projects.
- Producing new projects is efficient and predictable — for example, typing
@atomist create springin Slack can result in a shiny new repository, ready to build and deliver to Cloud Foundry or Kubernetes. - When we create a new project, we keep track of it: for example, doing any necessary provisioning, such as updating firewall rules.
We need to apply the kind of learnings we’ve found in managing services to managing our code bases, and how they get to production. Atomist does this via its API for software. Imagine what it could do for your service herd.
