Could Modern Productivity be Increased a Thousandfold?
In part one we looked at the industrial revolution, when productivity increased by a factor of a thousand by replacing technical generalists with specialists. In part 2 we’ll look at whether the modern software revolution can learn anything from the industrial revolution.
How Did We Get Here?
fungible
replaceable by another identical item; mutually interchangeable
At the start of my career, in the early 90’s, the perfect developer was fungible — an interchangeable component of a project. Engineers were a general purpose technology.
Recently fungibility has returned in form of the full stack developer who can do any technical job.
But why do we want generalists, like the full stack engineer, when we learned 250 years ago that it was inefficient?
Software Projects are Hard
Back in the 90's we were failing at software projects. Overruns or cancellations were common. We only had one method in our toolbox — waterfall — and that was hard to use.
Waterfall gets a lot of negative press but it isn’t a bad technique in certain circumstances. It can deliver large projects fast if the going is incredibly smooth but it’s a high-gear approach that cannot handle uneven terrain. So, we tore our hair out trying to flatten the landscape.
Flattening the Landscape
What made a waterfall project more likely to deliver? A highly controlled environment.
- A fixed specification was key to delivering a waterfall project. If you’ve carefully planned everything in advance, change is bad.
- Known and predictable technology was vital. You really needed to use familiar tech.
- Fungible workers. Even in a controlled environment waterfall projects constantly went astray, which risked one team getting ahead of another then sitting around waiting. If everyone was interchangeable the feudal overlord controlling your project could move developers around to keep them busy.
On big projects, interchangeable developers were required. But they are also important for small companies. A startup may only have a handful of developers, it’s more resilient if every developer can do everything.
So fungible developers gave you some chance of successfully delivering waterfall projects and they kept your startup business from falling over the first time someone left. That’s good.
But was the trade-off for the generalist full-stack developer the loss of that Industrial Revolution thousandfold productivity increase from specialisation?
The End of Control
Eventually, it became impossible to control the environment on every project and waterfall could no longer stagger on. This loss of control was for a good reason — our end users had become much more diverse.
In the early 90's our end user was typically a businessman on a Windows desktop. Even end-user applications looked fairly identical. However, by the late 1990’s that was no longer true.
Cloud applications and e-commerce started to appear. The user-base widened to non-techies at home. Competition intensified and user choice increased. Customer feedback mattered and you couldn’t even attempt to fix specifications in advance. Stack and framework choice got bigger, tech improved and sticking with an old platform could put a company at a big disadvantage.
This process accelerated through the 21st century with new devices like smartphones diversifying the user base even further.
By now, there is literally no demographic that is not a potential user of software.
Diversification Has Been Forced On Us! Maybe.
Acute competition and an expanding user base forced technological diversity and evolving specifications on us. We therefore had to come up with new development methodologies with the torque to handle more unpredictable terrain.
Agile methods shortened projects by concentrating on micro-results that could be delivered in weeks. They allowed constant readjustment of resources and expectations. Agile also encouraged a powerful and scaleable ground-up approach to project management with a focus on better team communication and cooperation.
So what about the specialisation of skill sets? Was that forced on us? Not really. Agile still requires fungible workers.
The Ultimate Project Management Technique?
The power of Agile is its ability to drive delivery from the ground-up. The most successful Agile teams usually collaborate closely and the most effective way to do that is if everyone’s a bit of a generalist. Agile is no less dependent on generalists than waterfall was. Hence the inexorable rise of the full stack developer.
I’m not saying that full stack developers are a bad thing. If we want to deliver constantly changing products in a constantly evolving technology environment they are probably required. I’m merely pointing out that the industrial revolution achieved a 1000-fold increase in productivity by phasing out such generalists and replacing them with specialists.
Maybe there’s no way we can phase out generalists and still get the other benefits of Agile. But I want that productivity increase so I’m not giving up quite yet — the benefits of specialisation in the modern tech industry seem like they should be greater than was achieved in 18th Century pin production. After all, one person can master all parts of putting a pin together, but in the 21st Century one person has trouble just staying on top of new AWS services.
Are We Already at Peak Diversification?
In our teams we already have designers, UX experts, product managers, testers, dev and ops, with maybe front and back end devs. Surely that’s enough specialisation? Probably not. During the Industrial Revolution they broke down producing a pin into 10 separate jobs. We’re talking about a very high degree of specialisation to get those huge benefits from expertise and focus.
What do we do?
So, Agile and specialisation don’t play well together.
There are attempts to work around this. The T-shaped engineer (a generalist with a speciality) is one, but isn’t that still a generalist? I suspect T-shaped engineers don’t give you the super-expertise or the hyper-focus of the Industrial Revolution specialists.
I’m not inclined to throw out Agile quite yet, but I still want those IR productivity gains, so do we have any other options?
Cloud Experts
I suspect there is one source of highly specialised experts that can work with Agile. The cloud.
Cloud services can replace several roles that might otherwise fall to team generalists, for example
- DBA (RDS etc)
- Exchange Admin (Cloud-based Exchange Services)
- Various Ops Roles (IaaS, PaaS)
- SAN & Backup Admin (online storage like S3)
Cloud services have been successful where there is a role that can be wholly outsourced with a clean interface. They are often for commodities with no business value in deviating from a standard implementation (e.g. a database-as-a-service). That’s quite similar to the ultimate result of specialisation in the Industrial Revolution. So maybe we could use services to get those IR productivity increases?
Lock in!
Of course the main fear of using a cloud service is lock-in, but are we overly worried about that? If the alternative is getting our generalists to do more and more, bear in mind the 1000-fold productivity hit we took in the 17th century from doing that. Maybe lock-in isn’t the worst problem to have.
Conclusion
I come to no conclusion here except to point out that we are currently taking a somewhat 17th century generalist-heavy approach in the tech industry. Maybe there’s no way we can do anything else, but if we can find a way to use specialists more (e.g. cloud services) then history suggests there are some pretty big productivity gains to be won.
Note this material (plus part 1) was presented as the opening keynote at the Software Circus Conference in Amsterdam, September 2016.
Please hit the Recommend button below if you found this article interesting or helpful, so that others might be more likely to find it.
Check out MicroBadger to explore image metadata, and follow Microscaling Systems on Twitter.
Image: https://www.flickr.com/photos/britishlibrary/11205393586/in/album-72157639428882575/
