Why does Software Engineering take so long?
Or, why it doesn’t take even longer
Every May I get the pleasure of working with a new cohort of interns as they experience the reality of Software Engineering for the first time. For the most part this is a joyful experience — my company has free food, ping-pong tables, and no homework. Compared with the toils of academia, big-tech Software Engineering jobs are cushy, friendly, and relaxed.
One thing they’re not is fast-paced, despite what the lofty job postings may tell you. Interns are often surprised, maybe even dismayed, at the seemingly tiny scope of the work assigned to them. I’m here for 3 whole months and you want me to create one measly dashboard? They start to plan out how they’re going to blow their team away by over-delivering — how they’ll probably be done with the entire project in a couple weeks, and what side projects they’ll take on in their downtime. Then, 3 months later, they’re rushing to deliver a pared-down minimum viable product.
It’s not their fault. Computer Science courses ask students to implement data structures, puzzle solvers, and even entire web apps in a matter of days. The best students participate in hackathons and leave feeling like programming gods endowed with the ability to create VC-ready prototypes in a single caffeine-fueled night. If they could apply that same pace of engineering to a full-time job they ought to be able to move mountains in no time at all.
So why can’t they? It’s easy to blame corporate bureaucracy and the overhead inherent in large teams: sprint planning, all-hands meetings, etc. Those certainly play a part, but they’re not the whole story. The bigger setback is that starry-eyed young comp-sci students tend to think of code like this:
That’s not entirely fair — Interns are smart. They know coding in the industry is a little different from coding in academia. They’re prepared to take a little bit of extra time to comment their code, test it, and get it reviewed. Still, they don’t realize just how deep this iceberg goes.
Importantly, everything under the surface still involves code. This is not a distinction of engineering versus management, but rather a dolphins-eye view of a project’s codebase. Only a small fraction of a professional team’s code actually comprises the end product. The bulk of it is hidden from end users. But, as in an iceberg, the hidden part is not purposeless. It provides buoyancy. It holds the visible part aloft.
Without tests, features break over time. Without reliable developer tools, a team may lose momentum. Without processes to diagnose and resolve customer issues, customers may take their business elsewhere. Etc, etc.
Not every new product or feature touches every part of the iceberg, but most projects touch most parts.
Even simple things can start to crystalize into major undertakings. For example, are you planning to store user data on a server? Great! But, you’ll need to do a few things before you launch:
- 🗄 Ensure the data is backed up, so as not to lose user trust
- 🗑 Allow users to delete their data at any time, including from backups
- 📲 Allow users to download their own data and metadata, in compliance with GDPR
- 📚 Design your storage system to comply with laws like HIPAA, FISA, and the EU’s Data Locality policies
- 🚂 Migrate existing user data as your schema or storage systems change
- 📊 Monitor backends for outages, quota usage, and security breaches
- 👮Control and audit access to user data by customer support agents
With this example in mind it’s actually surprising, and impressive, that any intern is ever able to complete a full project in only one summer.
How does anything ever get done?
Well, it all started around 2.6 million years ago when early humans discovered they could use pointy rocks to focus all their strength onto a small area. This didn’t actually make them stronger or smarter (at least not directly) but the effect was the same: they were suddenly able to accomplish super-human things. To this day, you and I are not nearly as smart as we think — we just have incredible tools to turn our meager ingenuity into brilliant things.
Your developer tools might not look a whole lot like pointy rocks, but the purpose is the same. Instead of applying your strength to the entire problem of creating a computer program (starting from ‘tricking a rock into thinking’), you can focus it on just the parts necessary to solve the problem at hand.
Indeed, our grand code-icebergs are only possible to write and maintain because of the incredible tools available to modern developers. Tools like compilers, operating systems, databases, libraries, and APIs… and source control, bug trackers, debuggers, and IDES… and documentation generators, linters, flowchart drawing apps, and StackOverflow… and security scanners, static analyzers, automation scripts, and templates… and whiteboards, video calls, emails, and Slack… and keyboards, mice, monitors, and coffee.
So it turns out there’s a whole other iceberg below the surface of our iceberg! We’ve got ourselves a four-dimensional meta-berg!
This new meta-iceberg is still made of code, just not code specific to your project. You may be spinning up a ready-made database server, or implementing your infrastructure on the cloud. You may be writing your UI in React or Angular or Svelte. These decisions mostly won’t affect how your app works, but will definitely affect your team’s development velocity and their ability to respond to problems. Bad tools, or the wrong tools for the job, can cost dearly.
Ultimately, software quality is a function of the code users see, the code user’s don’t see, and the code that helps all that other code get written. High quality apps and services are only possible when all of those pieces come together.
Why it matters
What do Uber, AirBnb, Tesla, WeWork, Netflix, Betterment, Flexport, and Amazon have in common?
Answer: All of them bill themselves as high-tech software companies despite earning their money in age-old industries such as transportation, real-estate, content, and retail.
Partly this is a ploy to attract venture capital money, but it’s deeper than just that. All of these companies do ship software and use it to differentiate themselves from conventional competitors. When those conventional competitors try to ship software of their own they often falter. It’s easier for a software-oriented company to pivot into traditional industries than it is for a traditional company to pivot into software.
Great software takes so much thought, so much time, so much investment, and so much talent that any company not built for it simply can’t support its weight. Indeed, the very concept of having highly-paid interns spend summers being tutored by even-more-highly-paid engineers would make most traditional companies squeamish.
Instead of rebuilding their core businesses around software, most traditional companies attempt to stay focused on their core competencies while spinning up small software organizations on the side. We just need an app, nothing complicated! Throw some nerds in the basement and let’s get this done. That works ok but the results tend to be clunky, and progress slow.
Meanwhile, however, software-oriented companies are investing in bringing more pieces of the code-iceberg in-house. AirBnB maintains a popular animation library; Netflix built its own CDN with custom hardware; Uber open-sourced something called a “Hexagonal Hierarchical Geospatial Indexing System” among many other projects. And we haven’t even mentioned the elephant in the room: Amazon, a retail giant so deep into software that it now sells its infrastructure to competitors. Those competitors, even with the full power of AWS at their disposal, will still struggle to compete in a software-oriented world. Amazon knows it, otherwise they’d be pickier about who gets to use their stuff.
Software is hard on its own, but it’s made harder when leaders fail to understand how deep of an ‘iceberg’ they’ll need to build to support it. They’ll learn as they go, but the process will be slow and they might not have enough time to successfully steer their gigantic ships before disruption comes. After all, it’s no secret what happens when a Titanic encounters an iceberg.
