The Pets vs. Cattle Analogy at Whistle

Published in

Whistle Product & Engineering

6 min readMar 11, 2016

My friend says we’re like the dinosaurs
Only we are doing ourselves in
Much faster than they ever did
We’ll make great pets, we’ll make great pets
— Perry Farrell 1993

Here at Whistle, we love our pets and they are the reason we do what we do. Bringing pets to the office is allowed and even encouraged. In fact, it’s not uncommon to have a few dogs under and on the conference table during meetings.

Whistle Software Team Scrum — the author is in the lower right corner

Vitas taking care of business and sporting his Whistle device

In the DevOps world, there is a fairly well-known mantra that you should treat your servers as cattle and not pets. In a nutshell, the idea is that your infrastructure should be as disposable and easy to replace as possible.

*As someone who was born and raised in the Midwest, it is important for me to note that farmers certainly care for their cattle and the analogy should not be taken to extremes. If a cow dies, it is still a significant event for a farmer and if a server dies, an engineer may still wake themselves up in the middle of the night with an automated notification.

Immutable Infrastructure

The analogy is helpful for introducing the concept of immutable infrastructure, which is a philosophy that the DevOps team at Whistle has come to embrace. Previously, the team worked with a variety of environments including managed hosting on physical and virtual hosts, and eventually completely virtual hosting in the cloud. Making the transition to immutable infrastructure was not easy, but well worth it in the long run. It was also a gradual process, some of which I’ll describe here.

Infrastructure Evolution

Prior to joining Whistle, I worked with a backend comprised of dozens of servers. Some were deployed as virtual machines, while others ran directly on physical hardware. Deployments (upgrades, introducing new features, etc) were always a scary, painful event. Often, they involved dozens of manual steps including modifying configurations directly on production machines. Development teams threw their releases over the wall to the operations team and had little collaboration after that. Deployments would work in test and staging environments, but fail in production. In short, the DevOps culture did not exist. I infamously once heard a manager quip that “we don’t do agile”.

At Whistle, the operations team had adopted some of the same practices used by the development team. Configurations were placed under source control (git) and configuration management tools like puppet and chef were introduced. Deployments became more automated and the infrastructure was becoming more codified. Developers became more involved with the operation of the system. They were online during deployments and were also added to the pager rotation. The DevOps culture was being adopted.

This was a huge improvement, however the infrastructure was still fairly mutable. Servers were never completely rebuilt. Additionally, critical non-server infrastructure such as load balancers were still being managed in the traditional fashion. On the positive side, continuous integration was improved and changes were rolled out in a semi-automated fashion. Upgrading applications via a package (RPM) managed by puppet became a trivial task and continuous delivery became a reality.

Once the system was at AWS, we began to experiment with immutable infrastructure as we could now build the entire stack from the operating system to the application. RPM’s were replaced by AMI’s as the new common build artifact for deployment and eventually containers were introduced as we marveled at how truly lightweight they were. All infrastructure, including the troublesome load balancers, was codified and deployed using the wonderful open source tools from HashiCorp such as terraform.

Our clusters became self-healing and even self-scaling. Deployments occurred on a daily basis with little fanfare with the developer of the actual code often deploying it.

Gradually, our servers had morphed from pets into cattle.

The 5 S’s

Our goals for DevOps at Whistle can be summarized in the following order

Stability Without stability, our products can not be trusted by our customers to reliably tell them about their beloved pets.
Scalability Directly tied to stability, we need the ability to grow quickly and easily.
Security Our customer’s data and system reliability can not be compromised.
Simplicity Automate as much as possible with the goal of increasing the “bus factor”.
Saving Finally, we need to continually scrutinize our use of resources without jeopardizing any of the previous goals. Idle CPU, memory and disk is a waste of money.

Our transition from pets to cattle has positively impacted each of these goals. However, the goals are never completely attained. The good news is that the DevOps culture is flexible enough that it allows us to make continual progress.

Lessons Learned

Be patient. Any migration of a complex system is challenging and paradigm shifts do not occur overnight. The benefits of DevOps are not always simple to comprehend and mistakes made along the way, may make lasting bad impressions. Also, set the expectation that DevOps is a journey and is never “done” Milestones can be set and achieved, but the software world evolves at warp speeds and there are always new technologies to explore and adopt.
Culture is Critical. DevOps is truly a culture that must be adopted by all stakeholders in an organization. Sometimes the most difficult challenges will be non-technical in nature as change is difficult for most people. Without cooperation, you are doomed to fail. For example, without true 12 factor apps, immutable infrastructure will be difficult, if not impossible to achieve.
Don’t get too hung up on tooling. In the devops world, there is no shortage of competing tools. Take some time to evaluate and choose the best tool for the job, but prepare to remain flexible when something better comes along. Avoid using different tools for the same basic tasks.
Take advantage of SaaS whenever possible. Focus on your core product/technology, because chances are good that everything else can be done better and more efficiently by someone else. Surely, you’re not managing your email servers anymore, do you really need to manage your Big Data database servers?

Pets Have Priority

Pets come first at Whistle. Fortunately, the DevOps culture has been adopted across the company and our awesome applications have been designed and developed with that in mind. Without solid infrastructure, we lose our focus on making our applications even better. Internally, our applications are named after actual employee pets at Whistle. Some examples are arlo, manolo and hank.