Most Claims in Software Development Are Unjustified
Severely test not just software but also ideas
Here’s something that bothered me quite a bit in my last years at the university. I taught various aspects of computer science and programming skills, and a little bit of software engineering, and very rarely did I find any claim I could with a straight face say to be true, not just my opinion.
In the software field we have a tendency to follow fads. Some really smart people come up a way to do something in a way that works for them. They give it s catchy name — object-oriented programming, Scrum, behavior-driven development, and there are many many more — and get lucky enough to build a significant following. There will be lots of people promoting that idea saying it will make software development better. Yet, why should we believe it?
It is well documented in the research literature — I too contributed — that there is hardly any research evidence backing up claims of benefit in software development. They say object oriented programming improves reuse but what is that statement based on? Nothing much. They say similar things about functional programming, but the evidence for that is nowhere to be found. They say you must follow Scrum to the letter to get benefits but who demonstrated that this is a reliable effect? Nobody.
Now, it’s mostly just the academics who care about research evidence. In the industry, there is little patience for academic ivory tower nonsense. Yet, when a company chooses tools and methods, what do they base the decision on? Usually someone in the company believed in that tool or technique and argued successfully for its adoption. Often there are spikes or pilot projects that are used to evaluate a new idea before widespread adoption, but even these provide only a weak argument in support: we tried it and it seemed to work.
The best rule for figuring out what actually works that I know of is this: try your damnest to show that the idea is without merit. If it survives this severe testing, it is probably worth something. I do not see this in the software field much — usually people try their damnest to show their idea in the best possible light and ignore all the bad bits.
One of the most damaging tactics enthusiasts and peddlers do is a bait and switch: if their preferred technique does not deliver what is promised, they will claim that it was not done correctly. No reuse benefits? Well you must not be doing object oriented programming correctly. Trouble in your Scrum? Well you must be not following Scrum but some adulterated version.
There is an interesting protocol they do in medical research: analysis by intent to treat. The idea is, once it has been decided (by randomization) what drug this experiment participant will get, everything that happens to them will be counted for or against that drug. If the participant has to stop the experiment for whatever reason, this is counted in the medicine’s results. If the participant switches to another drug midway, it all still counts for the original drug. The reason? These changes are assumed to reflect a bad aspect of the drug that would be missed if that participant were to be eliminated from the analysis.
So say 50 companies decide to adopt a new development method called Tortoise and prepare for it properly, following relevant best practice protocols, and things go badly wrong in 40 of them. In post mortem review, it is discovered that the implementation was not Tortoise. In the 10 that did Tortoise correctly, things went very well. Conventionally, we would then say that this was not a fair test of Tortoise, or that Tortoise actually works. In an intent to treat analysis, we say that this absolutely counts against Tortoise. It shows that current best practice makes Tortoise difficult to follow, therefore it is not ready for wide deployment.
Accordingly, I believe software developers — and particularly those of us in the business of telling other people how to develop software — need a lot of humility, much mote than is seen out there today. By all means, share your ideas and successes, but stop claiming universal benefits and applicability unless you have strong evidence.
As a developer, you can help by bringing your inner skeptic to the table every time you run into a new idea. Try to break it and make it fail. If it survives, you have much better understanding of where and how it might be desirable to use.
As a leader in software development, you can help by demanding severe testing of new ideas. By all means, look for the potential benefits but do not buy into them before you have seen people trying to break it.
If you are an educator, you have a severe responsibility to teach useful things. By all means, teach the stuff that is hot and new, but do not parrot unjustified claims of benefit. Instead, teach your students some skepticism.
We are all better off following Socrates: know that you do not know.
Post scriptum
Based on feedback, I should point out this:
All the things I name in this story (including object oriented programming and Scrum) I find to be useful. I have nothing bad to say about them. My critique is aimed at overblown claims I have seen and heard about them and many other things.
Generally, these claims do not come from the experts — they know their stuff and its limits — but from enthusiasts who do not have sufficient experience to temper their claims.
