EA Principles Addendum: Build vs. Buy

by Brian Chambers

Build vs. Buy was a popular topic, generating a number of great questions. In this post, we’ll explore the topic at a deeper level.

Photo Credit: https://spectrum.ieee.org/go-language-tops-list-of-indemand-software-skills

Ability to build software as a differentiator

Several commenters called out the capability to “build your own software” as a key differentiator to a business. I had not thought about this in a while, but I agree. Having the capacity to build your own software is indeed a differentiator. There are several reasons why I believe this to be true:

• you don’t have to be entirely at the mercy of public market offerings
• you can build to unique elements of your business model
• you can innovate at your own pace
• you can treat SaaS solutions and OSS as building blocks to compose new kinds of solutions in ways that are unique and differentiating
• you can control your own customer and employee experience of technology

At Chick-fil-A, we made some intentional shifts about eight years ago to build an internal engineering muscle, which was a huge shift from our historical approach of “buy and integrate” with middleware tools. This cultural change was well-timed to accompany our shifts to DevOps delivery, Cloud Computing, “Big Data” Analytics, and modern services architectures and were driven by shifts in our business:

• The shifting to directly consumer-facing technology solutions (Chick-fil-A One) which required a scaling strategy that we did not have the capability to achieve in our previous co-lo data center environment.
• Unique (to our industry) capacity challenges due to growth of the business that required innovative new solutions and platforms (like our Edge platform).
• A growing data-drivenness that outsized our traditional capacities and sent us into the cloud via services like AWS Redshift (which we still use heavily).
• A desire to serve our restaurant Operators and Team Members with solutions that fit their business needs and a desire to enable them to innovate and growth their businesses by applying technology. That required some solutions that were not commercially available and led us to build platforms to enable Restaurant Operator self-service.

Since then, we have built tons of new products and systems and hired lots of software engineers, systems analysts, SREs, quality engineers, solution architects, and platform engineers to build out the things that we see as essential to our business and require different capabilities than what is available on the market.

Given that we have the capacity to buy or build, it was important to create a principle to provide guidance to decision-makers. We still buy a lot but try to build what differentiates us.

Is build vs buy too simple?

Richard shared our last Build vs Buy post and highlighted our “build when differentiating” approach well.

I don’t know Bobby personally, but I love the wording he used here. I think this matches our actual reality much better than the wording in our principle. There is definitely a fine line between terms here, so lets explore them.

• Create — this is akin to our previous definition of “build”
• Consume — “buy”, but acknowledge this could be SaaS, open source software (one we failed to mention in our post but rely heavily on), cloud services, public API’s, etc.
• Compose — mashup of create and consume to develop something new, perhaps by gap-filling. Platform-building would likely qualify here, too.

(links are ones I happened to read the day this was written and they felt appropriate)

I plan to adapt our principle to better reflect these realities.

A call for examples

This would be even more informative if you shared some key examples where you chose to build and where you chose to buy, especially if the decision wasn’t “obvious.”

Ask and you shall receive. Here are a few examples from our world.

Build

• We built our entire Edge infrastructure from hardware and BIOS to OS config to K3s to shared services running on top, a cloud control plane consisting of observability and deployment services, and a number of custom home grown tools to support the entire ecosystem. Why? There wasn’t a market fit that made sense for our desired outcomes.
• We also built our own OAuth-based identity management server for IOT device identities. Why? Solutions were expensive and were usually licensed per-device, and we also wanted some flexibility to implement newer flows for on-boarding like OAuth Device Code flow.
• We built the entire mobile app and API back-end for our Chick-fil-A One application. This is a microservices architecture that was built from scratch.
• We built our own GitOps agent for our Edge deployments called “Vessel”, because 1) we didn’t know GitOps was a thing and 2) felt like it was a good model for our environment. We eventually looked at other OSS tools (Flux, Argo, etc.) but still have Vessel deploying in > 2,500 restaurants.
• We built a mobile application that provides real-time metrics about a restaurant Owner/Operator’s business so that they can make better decisions in real-time. This makes sense because we are the only ones that have the metrics across all elements of our business that Operators want and need.

Buy

• We use Workday for corporate HR — no reason to build that as its not our core business.
• We use Okta for workforce Identity Management — there’s no need for us to build our own solution for that and for managing SAML integrations. Fun fact: we did have our own version pre-Okta… but we were happy to move out of that business.
• Have bought a COTS point-of-sale system and have used it for 15+ years. This one has a catch because we made a lot of changes and customizations to it through a partnership with the vendor over time.
• We pay for GitHub. We’d rather build things that benefit our business than spend time hosting our own git repositories. One exception: we do host our own GitLab or repositories for our Edge GitOps deployments because the licensing model didn’t make sense for the way we wanted to operate (> 2500 repos).
• We license an K8s ingress product called Gloo. Why? It doesn’t make sense for us to create things that already exist, even when they are core to our platform. We also use SaaS Grafana. We use TONS of open source software, from Kubernetes to Gloo to Prometheus to Hashicorp Vault to ArgoCD and so on.

I am not sure if any of these are “not obvious”, but they are some real examples.

Conclusion

I hope this gives some more context to how we are thinking about creating, consuming, and composing systems at Chick-fil-A.