No, you don’t need cloud. Docker? No. Kubernetes? Hell no!
There’s a certain class of problems in our profession that can be solved using simple boring solutions. Not particularly exciting technologically, not worthy of blogging about, likely not get you accepted into conference talks and having that in your CV will make you look like a dinosaur. Then there are a set of technologies that do the exact opposite. Sometimes engineers choose technology for all the wrong reasons. Ultimately everyone looses. Let me explain.
“The buzz” is overrated
When we try to be “in the buzz” of what’s going on in our industry, we flock to sites like reddit, Hacker News etc… There’s a certain class/pattern of things that gets popular in those news aggregators. Exciting new technology, someone rewriting a Java piece in Rust, a shiny new text editor you can use to edit text files — half a gigabyte in size, eat 80% of your RAM (but hey you can use the mouse AND it support tabs!) etcetera.
When we try to adopt technology for simpler problems, we forget the context, scale and budget we have to deal with. I believe that’s a big mistake.
Taking technology decisions based on emotions — Bad idea
Anyone who’s been around long enough will tell you that technology is the easy part. Trickier part is dealing with humans in the equation. Sadly some technology decisions for an entire company are based on,
- Looks good on a resume and likely will help getting hired for someplace else in 2 years
- A new director/VP reevaluating everything because current systems are boring and need to show something different. Measurable impact and help to make the jump to SVP
- Big boss had a lunch date with another big boss from company X and now tells entire engineering team to scrap everything and redo stuff with technology from company X
No engineering text book will prepare you for this.
Technology selection based on buzz words — Bad idea
Some of the powerful cloud, containers, orchestrators are created to handle very large distributed computing problems. Kubernetes solve Google scale problems. It’s good to ask whether you have Google scale problems. If you don’t then it’s going to be a massive waste of time, energy and money.
There are broadly 3 classes of companies.
- Technology driven companies (Netflix, eBay, Uber, Lyft, Facebook)
- Companies who use technology as a key part of their success (Banks, insurance companies, device manufacturers etc…)
- Companies who’ve outsourced their IT (not worth talking about here)
It’s worth noting that there’s a clear differentiation between #1 and #2 type companies. Technology driven company deal with,
- Massive scale problems — large user base (entire world)
- Heavily engineering driven. Engineering team take all technology decisions based on problems at hand
- Solving real hard problems
- Low latency is important to provide an equally fast service for everyone. Edge services, heavy use of CDNs etc…
- Requests coming from all over the world. Need a distributed scalable architecture. Lot’s of microservices, multiple data centers and regions. Complex request routing logic, highly sharded DB layers, heavily cached read ops etc…
Type #2 companies tend to be not so heavily invested in cutting edge technology. However, technology is an absolute crucial part of their success. Tend to have,
- Smaller development teams
- Bound to only a few static data centers
- Very constrained user base — couple of million users at most or less
- Engineering capabilities/skills are comparatively lower as a whole
- Actually need simple architectures to solve relatively small straightforward problems
Primarily looking at this class of companies during this post because that’s where a lot of bad technology decisions are made.
Moving to cloud to cut costs, it’s cheaper — Bad idea
Being on the cloud gives a lot of flexibility to solve problems which was not possible before. My favorite example is from Netflix where Adrian Cockcroft mention that during early days when they were trying to use Cassandra, they were able to stand up a 96 node Cassandra cluster across 6 different data centers, load 18 terabytes of data, hit it like crazy and then shut it down. They were able to understand a lot of characteristics about Cassandra for their workload. They were able to do it because they were on AWS. You cannot do that sort of thing if you’re on a traditional data center.
However, not everyone has that kind of problems. Even though it’s easy to begin with, people often overlook or underestimate the associated costs. It’s easy and painless to get started with AWS lambda for instance. Embracing server-less even for a relatively small enterprise and running things in production adds up to the monthly bill quickly. It’s very important to evaluate the costs much more closely and not drink big-cloud-vendor-kool-aid.
At large scale, numbers are staggering. Lyft for example, in their S1 filing, disclose their AWS investment.
In a January 2019 addendum to such agreement, we committed to spend an aggregate of at least $300 million between January 2019 and December 2021 on AWS services. If we fail to meet the minimum purchase commitment during any year, we may be required to pay the difference. We pay AWS monthly, and we may pay more than the minimum purchase commitment to AWS based on usage.
This amounts to about $8.3 million monthly. Not an ELA (Enterprise License Agreement — usually fixed cost, all you can eat type deal), not annually but monthly. MONTHLY! That’s staggering.
Depending on complexity and type of problems, you might not need cloud at all. A lot of the problems enterprise companies deal with can be solved with technology that’s simple, straightforward and downright boring. If you look at things objectively, that’s perfectly an OK thing to do.
We have to develop a knack of looking past the noise and sales pitches from people who are constantly trying to sell things. Look at the problem we’re trying to solve and choose simple tools.
So what we should do?
Look beyond buzz wordy tech to find simpler solutions. There are high quality open source tools for creating vendor agnostic solutions to pretty much any enterprise problem these days. These open source tools provide similar interfaces and features so it’s very easy to create solutions on a traditional data center. Most recent example I found is Minio. Minio is an open source storage server with an S3 compatible API. That’s very powerful. You can integrate a much more cost effective solution with full API compatibility without cloud vendor lock-in.
Always evaluate economic feasibility and necessity before adopting shiny new tech.
Always choose the simplest tool/technology/architecture for the problem at hand. It’s always recommended to look at the problem first and then figure out the best way to go about solving it. Looking at a solution/technology and trying to adopt it for a problem is not an optimal way to solve the problem. Often time it create more problems than it solves.
Do you have war stories and/or similar problems after having to adopt complex tools? Let me know.
