Is the Cloud Making Us Dumber?
What happens when all of complexity is abstracted away

As is the trend these days, I want to take you back in time. Specifically, about ten years ago, to pre-cloud days.
I had a project back then — deploy and configure a fully working market data distribution environment for a large investment bank. Needless to say, this was a highly non-trivial exercise!
Among many other things, the project required a fairly in-depth understanding of Unix, Cisco multicast networking, DNS, firewalls, physical cabling, databases, etc. Stuff that seems utterly quaint now but was a really big deal back then! Personally, I felt like a wizard schooled in esoteric dark arts, immensely proud of all the arcane knowledge I accumulated over the years.

In fact, the reason why I mention this is not to reminisce about my accomplishments of old but to honestly admit that it was years before I felt confident enough to tackle such a complex project on my own. Many years! In other words, no aspiring sysadmin would ever attempt to run with this solo — the sheer complexity surrounding this project acted as a natural barrier to entry. Junior techies need not apply!
Fast forward to today. Last week, it took me only a few days to deploy a containerized Jenkins server, with ephemeral Docker workers spun up/down on demand, persisting its data in a distributed AWS Elastic File System (EFS).
Somehow, I managed to accomplish all of this, despite
- never having used Jenkins before in my life. No problem — the team behind Jenkins created a working Docker container for me to use. All I had to do was deploy it!
- not knowing how to elastically scale Docker workers on demand. No problem — some kind soul wrote an AWS ECS plug-in that worked on the first try. All I had to do was install it!
- lacking any kind of expertise in design or deployment of a highly available, POSIX-compliant, distributed network file system. This was especially sobering because only a few years ago, implementing such a file system would have been a highly complex exercise tackled only by a team of highly trained, highly experienced, highly paid Sr. System Engineers. But no problem here — the team at Amazon made this service available at a click of a button!
In fact, it almost feels like cheating. Somebody did all the hard work and all I had to do was click a few buttons?

Now, make no mistake about this — I truly think the democratization of these technologies has been a huge success for our country in general and our industry specifically. After all, lowering the barrier to entry to all this amazing tech is what allowed for a tremendous explosion in the startup scene, massive productivity gains and shrinking unemployment numbers.
In fact, if you look at Amazon Web Services’ ever growing catalog of cloud-enabled offerings, you will notice that the vast majority of them are both a) available with a few button clicks, and b) highly complex to replicate in-house. And this is a good thing, we certainly do not want to go back to the days where only a select few, well-financed enterprises could afford all this tech.
However, if all that complexity is abstracted away, what does that mean for engineering talent? Specifically, if I can get instant access to 8 exabytes (!) of AWS EFS data storage, full use of a seemingly limitless computing capacity all over the world, and rich APIs to automate it all, why would I hire and invest into in-house system engineering expertise?
The content of the Library of Congress is commonly estimated to hold 10 terabytes of data in all printed material. Therefore, AWS EFS offers instant access to 800,000 times more storage than all of Library of Congress’ books put together!
Moreover, if I am a new college graduate looking for a career in tech, why would I bother trying to learn any of these highly complex technologies? After all, as soon as a promising new tech makes a splash, all of the big three cloud computing providers make it available for a metered cost — no contracts required!
In other words, are we back to the days of old, where a select few companies scoop up all the engineering expertise, leaving the rest of us to feed off their crumbs?
I confess, I do not have a fully thought out answer to this question. However, it seems clear to me that if I am system engineer hoping to remain relevant for the foreseeable future, I must adopt a software engineering mindset.
Specifically, I need to learn how to treat infrastructure as code, practice Agile Kanban and story pointing. I must learn source code control, pull requests, and how to treat servers as immutable, fully disposable assets.
That said, while all these things are new and fun, are they interesting? Are people really going to get as excited about troubleshooting a new Ansible script as they did about standing up a brand new computing fabric, all on their own? Will there be a similar sense of pride in their accomplishments?
Perhaps not. But maybe none of this matters?
First of all, pundits have been making dire predictions about the future of systems engineering for years and few of those have come to pass. Everyone I know is still fully, happily employed. So, perhaps we will reach some equilibrium between private colos and public clouds.
Second, let’s be honest — few people these days know how to dig a well, build a house or skin a hide — specialization and commoditization of niche expertise means we get to focus on other tasks, ones that are more valuable to our customers, the business owners and shareholders.
Further, we can deliver solutions to our customers much more quickly than ever before — the days of developers waiting for servers are largely over! Once again, this is a good thing. I am certainly not waxing nostalgic over waiting weeks for a new VM.
I do miss naming my servers, though. I don’t know about you but my favorite scheme was the planets in our Solar system. Sometimes I would go the Greek gods route, if I were building a larger farm. :)
