On the immaturity of Software Engineering

A short while ago, an excellent article titled ‘You Are Not Google’ was making the rounds. In it, Ozan Onay made the case that Software Engineers act irrationally around new technologies, and jump on whatever new technology is hip right now whenever they get the chance. With as result over-engineered solutions and the introduction of ridiculous levels of complexity. Instead, Onay argues, we should select our technologies more rationally: based on the problem, not on all possible solutions.

Onay is of course completely right. But he’s far from the first one to realize this, and to describe the concept eloquently. In fact, I’ve read similar arguments nearly a decade ago, and I consider myself relatively new to the field. Furthermore, solving problems and selecting the appropriate solutions is fundamental to our profession. I’ve read books that were written before my professional career that touch upon the issues of over-engineering and choosing the right tools for the job. I would expect every professional Software Engineer with at least a few years of experience to know these concepts and live by them.

But I know from my own experience that this is not true. Worse even, I’ve seen this with many concepts that I feel are basic to Software Engineering as a profession, such as the need to communicate with a client to develop good software, the fact that requirements cannot be “gathered” but must be engineered, the existence and importance of different levels of abstractions, and so on. I’ve often become disappointed when I discover a complete absence of such a concept in a fellow engineer.

And given how articles like this become popular so fast, there’s apparently still a big need for these insights. How can it be that the fundamentals of our profession are still one of the most important things that people have to be taught about?

Obviously I don’t have an answer for this problem, but to me it’s one more sign that indicates the immaturity of our field. So I’ll attempt to at least try and explain the phenomenon of what I’d like to call tech-driven development instead of good engineering practices by exploring a number of potential root causes and having a quick look at other professions. Because as my good friend Sun Tzu said, “If you know the enemy and know yourself, you need not fear the result of a hundred battles.”.

Education

I’ve studied Computer Science, and while I learned a lot of valuable things, I don’t believe that the specific concept of selecting technology for solving a problem was a part of it. Worse even, I’ve found that educators struggle with new technology themselves. It’s not uncommon to encounter classes filled with students that know much more than their teachers, or teachers that desperately introduce new technologies that they barely understand just to try and stay current.

The role of education in the immaturity of the Software Engineering world is very significant. We still haven’t properly figured out how to reliably develop good software, and tooling is far from standardized. So it’s no wonder that educators struggle with finding the right concepts to teach; it’s much easier to teach something concrete such as the syntax of Java, rather than how to approach a complex real-world request and to solve real-world problems. And no, just picking a real use case and letting students have at it is not educating, it’s brute forcing.

Curriculum are often organized around what is basically applied mathematics, and software design classes are filled with practical work. Both are very important, but fail to teach the basics that a Software Engineer really needs to be aware of when designing and developing systems. This is very visible to anyone who has ever hired someone directly from University; they are usually woefully under-qualified, and require a lot of on the job training.

Other, more mature professions have much more rounded programs. Chemical Engineering for example also teaches about the financial aspects of your decisions, as well as logistics, safety, some financials, and a number of other concepts. They have developed tooling and standards for these things, and while these might not match exactly what you’ll use in practice, they succeed in teaching you the concepts that you need to do an actual job.

So I argue for more well-rounded curriculum in Software Engineering education. Unfortunately, to exacerbate the problem, educators are highly under-appreciated and undervalued which makes it even harder for highly skilled engineers to get into a position to teach future generations, and as such to improve the level of the Software Engineer.

Resume Driven Development

An explanation that a commenter on Hackernews brought up is Resume Driven Development. In short, developers have their companies adopt new and shiny technologies, and because it is so hard to hire people, companies start hiring developers for those specific technologies. With as result that you have developers for specific technologies, in which it is relatively easy to specialize, rather than well-rounded Software Engineers that pick the right tool for the job.

General experience is heavily undervalued in the job market, as recruiters always try to boil down the extremely complex question of whether someone is suited for a position to simple metrics. Which is understandable, because hiring people is really hard, especially if you’re not a technical person. So they ask for an arbitrary number of years with a specific technology, and as such the company ends up with someone who can only do a specific thing. And because these people get hired, and compensation is largely the same, there is apparently no need to learn anything else.

The “one trick pony developer” is actually a perfectly fine and a valuable asset to have around in many situations. But don’t confuse someone like that with a well-rounded Software Engineer, who is able to choose the right tool for the job and keep an architecture clean. In fact, I would argue that such a person is not a Software Engineer, and should not be called that either. Because calling both of them Software Engineer is demotivating for the latter, and calling the latter Software Architect is only confusing the issue as well as the people involved.

In other professions, there also exist specialists in certain areas. Where we have a Redis or Docker specialist, they’ll have operators, planners and calculators. Sometimes specialists are given higher regard, such as in the medical profession, and in some professions well-rounded individuals are the ones pulling the cart.

A similar scheme can be adopted in Software Engineering. We need to stimulate engineers to become well-rounded and to adopt engineering practices and reward them accordingly. If someone wants to choose the easier path of specialization in a relatively common field then that’s fine, but the rewards should not be the same for such a person. Perhaps the title “Software Engineer” should even become a protected title, that can only be used by those with the right qualifications. Although that is not exactly without its challenges.

Lack of an experienced workforce

Where the hell are all the engineers with more than 25 years of experience?

While this argument writes itself, I actually want to make the case that I don’t feel that this is a huge contributor. I still learn a ton, but a lot of what I know is cemented by fundamentals. I haven’t often seen engineers with more than 10 years of experience add to their fundamentals. And I’ve seen a lot of them missing those fundamentals.

The older you get the more conservative you become however, which in other industries does seem to help keep the youngsters from doing crazy things. So perhaps it would help to have more engineers of 50+. But it’s impossible to even test this hypothesis because once again, where the hell are they?

New technology is “free”

Compared with other industries, it’s ridiculously easy to use a fundamentally different technology for a new project. In a factory (you know, with tubes and valves and stuff), you are not going to make friends if you propose to purchase brand new equipment for every project; the finance department will eat you alive for the proposed purchasing costs, and the operators will laugh in your face if you make them maintain new technology every time.

However, in Software Engineering, it’s the most normal thing in the world.

As most of our technology is free and open source, no investments are needed to adopt something new. This removes one of the most important safeguards of conservative business practices of traditional companies, finance, from the equation. In addition, the “operators” are like-minded developers who can’t wait to ignore instruction manuals and operate dangerous new technology, unhindered by a lack of knowledge. Training for new technology is not budgeted or planned for, and those who struggle to pick it up quickly are looked down upon, as they should “keep up with the newest developments”. And to be fair, training is no fun, so let’s just start typing instead.

Don’t get me wrong, free technology and easy adoption is awesome. Cutting finance out of the equation is also great. But it does require for a team of Software Engineers to self-govern regarding costs of adoption and training, which is not something that happens consistently enough. There are no resources available to help teams handle this, and no real willingness to do it either.

A good first step to professional self-governance in this area is to realize that there are costs of adoption. Then, because in general we don’t have the skills to estimate them, it’s a good idea to pull in knowledge from other departments, such as finance, and ask them for advice. Try to make a good guess, and actually use it in your decision making process.

Software Engineering is immature because Software Engineers are immature

That sounds a bit harsh. But consider someone who:

• Is relatively unexperienced (≤ 15 years);
• Plays with a new shiny thing every opportunity he gets;
• Has pretty much nobody to answer to (except for his peers);
• Operates in a world that only he and his peers understand;
• And complains about the ignorance of those who “just don’t get it”,

then that sounds awfully a lot like a child, doesn’t it? Playing with a new toy whenever he can, complaining about the stupid adults that don’t let them ring the doorbell of the neighbor, and so on.

Software Engineers can afford to act like this because companies cater heavily to all their needs due to the enormous shortages of Software Engineering talent. They spoil them with great compensation and perks that make things more “fun”, like air-hockey tables and such, often regardless of actual performance. And they insulate them from the rest of the company because their work is “specialized production”, so the time they spend producing code has to be optimized. To optimize the amount of code written, the company proceeds to create an in-between department with “analysts” and “architects” or “product managers” whose sole purpose is to do those things the Software Engineers don’t feel like doing. These people mediate between the Software Engineers and the rest of the company, and end up taking the responsibility of the work of the Software Engineers as well.

The end result? A completely disconnected team of Software Engineers that has to be convinced, herded and baited to do the things that the company would like them to do. Many Software Engineers can afford to see their job as a game. They can play with whatever they deem the most fun and don’t often need to answer for it. So why bother with boring good engineering practices? Oh look, a shiny ball!

My opinion is that a Software Engineer, to do his work properly, cannot be insulated from the rest of the company. In fact, he has to be integrated in the company processes, he has to understand the client needs and the user needs, as well as the technical consequences. It is a very broad role, and while this is not something that necessarily should be demanded from all Software Engineers, it is ridiculous to consider our work to merely be “writing code”, because it is so much more than that.

Wrapping up

To end on a positive note, I’m very happy that there are also plenty of really good Software Engineers that do their best to educate others. There are a ton of good resources out there, in the form of books, articles or stackoverflow answers. So a lot tools, concepts and ideas are out there; the question is, how can we get them into the DNA of Software Engineering as a profession?

It’s a long way to go, but I’m sure we’ll get there. In the meantime, I suppose that writing and sharing articles such as Onay’s is a good way to go. But we should keep in mind that our end goal is that these concepts are fundamental to our field, and should not be something one learns for the first time while reading a random article somewhere a number of years into their profession.