Why so many software engineers are that bad?
First of all, a short exchange that I had on LinkedIn:
Me> There are a lot of clueless engineers writing horrible code and creating terrible designs.
Raja> I am surprised why this happens in almost every organization. And this is in spite of the organization's claims that they have the best team, process, tools, etc.
Man… I love the question, “Why”. So, let’s dive into it.
They are actually not bad
First, I believe many of these crappy results don’t come from engineers being bad per se.
Just take a not-very-senior engineer(s), add some weak management and weak technical leadership, ramp up the pressure, and you will get tons of shit code on the other end.
BTW. What I describe is just one way to get bad results. There are other ways — the culture of not sharing knowledge in the organization, absence of mentorship, unclear direction with a constant drastic change of requirements, etc.
You know, the programming saying “Garbage in, garbage out”?
So, the truth is that you have multiple inputs — people, process, culture and so on and one output (your software).
You may have good (or ok) people, but if the rest of the input is garbage, you are guaranteed to get the garbage on another end.
A side note: You can argue that if people are good, they could/should fix the process, culture, etc. Trying to fix it from the trenches is extremely painful and rarely works out.
Impedance mismatch
This is another topic I wrote about in the “One company’s lousy engineer, is another company’s good architect” article.
Sometimes, people just don’t fit well into the company. The person could be good. The company could be good. However, they may operate on different wavelengths, value different things, and emphasize different areas.
And sometimes, it’s amazing to see how a person who was absolutely terrible in one (reasonably good) environment flourishes in another environment.
Money… Money… Money
Ok… Ok… I was nice and soft in the last two sections. now it’s time to pull the big guns out.
We are rapidly approaching 30M software engineers worldwide. It was around 5M in 2000 and something like 1M in 1980. (BTW. These are very-very approximate numbers that I pieced together from several sources, so please take them with a grain of salt). If you go back to 1960, the number of people doing kind-of software engineering (with a huge mix of hardware engineering) was measured in tens of thousands.
BTW. All of these numbers are purely counting software engineers. If you add to this product management, software engineering management, IT, Ops, etc. The numbers and growth will look even more staggering.
A 60-year span can look like a long stretch for a human being. However, it’s a blink of an eye from a societal perspective. And within these 60 years, the industry went from almost nothing to these 30M engineers. Oh… And 5 out of the top 10 biggest companies by market cap are software companies (with some mix of hardware there).
Getting back to bad software engineers.
The software industry sucked into its orbit everything that moves and can type ‘print “Hello world!”. There are tons and tons of people who got lured by huge amounts of money sloshing in software (BTW. I don’t blame these people. If tomorrow, salaries in astrobiology would become $5M/year, I would seriously consider starting learning more about alien life forms :) ).
The issue is that tons of these people who have/had no interest in software are not interested in learning more or improving (as long as a paycheck keeps coming).
Growth pains
And continuing the subject of explosive growth of the software industry.
Unfortunately, many people who ended up in software weren’t well prepared for it. Some will catch up, but many will be barely treading water and moving from one company to another, leaving a trail of problems behind them.
There is another thing that was caused by this fast growth.
When the company grows really fast, it usually ends up quite detrimental for engineering. If you don’t have enough core people who are good and who can share/upkeep/enforce best practices, you will get quite fast to a place where anything goes. And the quality of everything will drop really fast.
The same ended up happening with the industry as a whole. The growth happened damn too fast, while a lot of best practices were not even well adopted yet. Actually, hell with adopted, many best practices are not even agreed upon.
As a result, everybody (including yours truly) often reinvents the same wheels repeatedly, stumbling and standing up just to uncover things that should have been taught in the first year of university for people entering our industry.
Nonuniformity of industry
How much commonality does writing software for Voyager 1 have vs. some throw-away script to clean up data on your hobby project?
The first one (Voyager) requires surviving decades, working on extremely limited hardware, and needs to be documented, reviewed, re-reviewed, and re-reviewed again.
Another could be a piece of crap, put together in 15 minutes, executed once, and thrown away. And it can actually run on a really beefy setup and won’t be documented, reviewed, and so on.
The software looks the same from a 30k view. However, the truth is that tech is not a uniform industry by any stretch of the imagination.
Precious lessons learned in one area of the software market may be detrimental in another. As a result, engineers moving even to adjacent industries may bring wrong lessons.
Summary
First of all, I think the biggest contributors to bad outcomes are not bad engineers but rather bad companies.
However, right on the heels of bad companies, you have people who have zero interest in any improvements (and putting the honest work in) and who joined software just for a big(er) paycheck.
P.S. Got a good additional point from Vivek that there is a question of motivation. I think a lot of people who came into software just for money may happily enjoy even the lower side of the software market salaries. And it could be too big a hassle to invest a lot of time and energy in learning more (to get a raise big enough to make it interesting).
