Why Most Software is Broken
At the airport, the computers are not able to change your flight. In the office, the ERP System will not print a report. Try to return a defective purchase and it takes forever due to the clerks computer being slow. Hopefully not all at once, but it happens. In fact, it is almost daily that some of the software we use is not working.
While technology has made things better we cannot help but think about all of the time lost and money wasted due to bugs in the software we use daily. However, this is minuscule when it comes to financial applications. Here, a single defect can represent millions of dollars lost. Miscalculated prices, improper orders, and failure to execute transactions are just a few the issues broken software causes.
The problem comes down to the processes used to improve quality. Since modern development techniques eliminate the obvious errors, the problems left are well hidden. They only surface after that system has been in production for weeks, months, or even years. These are the issues almost impossible to test for. However, it does not have to be this way.
Dave, the Director of a Hedge Fund, needed an automated means of finding arbitrage opportunities using Artificial Intelligence. After a few months of development, Dave’s Arbitrage System was a massive success. Then, almost overnight, the opportunities stopped. Actually, it was just the software was no longer able to find them.
Distraught, Dave asked me for help. Turns out, it had a miscalculation that only appeared with specific market conditions. Those states that did not exist at the time the strategy was being built and could not be tested for. In fact, the 30-year IBM veteran leading the team tested for every conceivable scenario during development.
This is why software breaks. All of the possible conditions are too numerous to test for. Even backtesting of automated financial strategies has serious flaws as the market is never guaranteed to act like it did yesterday, last week, or even last year.
However, software does not have to be broken. For Dave’s Arbitrage System, the solution was to apply formal verification to the system, find the issue, and improve the algorithms. This formal verification ensures the methods used actually work regardless of state in a manner similar to a mathematical proof.
Once the issue was corrected, Dave was able to realize near risk free returns again. Using the results to grow his fund from just a few million to ten times that over the following few months. Today, he has expanded his automated strategies and continues to prosper while other funds are closing with tearful YouTube videos.
How Slow Development Creates Defects
There is a correlation between speed of development and number of defects. While not intuitive, the truth is, that generally, the longer software takes to build, the more defects there are. This is not to say to rush through development. It is to illustrate that taking too long is often due to introducing bugs into the system.
The relationship between quality and time in software development follows a bell curve. Have the team finish near the top of the curve and the code will be stellar. Complete on the outside of the curve as either too short of time or too long, and the code quality suffers. While easy in theory, the trick is discovering the top of the curve.
One case in particular involved Brad, the VP of Software for a Hedge Fund who was tasked with created a new Options Trading Platform. However, after 2 years of work and over one-million dollars in expense, the system was far from ready for release.
When asked, Brad explained that most of that time spent was on an endless cycle of correcting bugs while apparently creating new ones. This phenomenon results from the hurried approach of wanting to finish — a vicious cycle that actually increases defects and skyrockets development cost. To correct the cycle, one must remember that the fastest way to finish is to first slow down.
First, I had everyone stop what they were doing. Then took a few days to audit the entire code base, marking areas of concern and making recommendations for changes. Next, the focus was changed from developer responsibility to team effort. As the project stress caused the team to go off into survival mode with every person for themselves.
Ignoring all of the existing bug reports, I made tasks that corresponded to those code quality concerns in the audit. In so doing, forcing the team to operate at the top of the curve. Not taking too long to introduce defects or too short to create careless error. As a result, the team was able to complete the entire project two weeks later with nearly zero defects. On last report, they sold it to another company for a large profit.
How Software Teams Create Bugs
I have heard Software Teams proclaim that no software is bug free when presented with defect reports. While true, this statement is often made to justify mistakes made during development. The issue is that the people creating bugs do not realize they are doing it. Instead, the believe they are adding features, correcting issues, or building code.
It is not the fault of the Engineering Team, they are just following supposed best practices. Instead, it is a problem that has been oversimplified by the industry and as a result all of the past means to solve it have failed.
Quality Assurance reviews, Unit Tests, and Automated testing have gotten us closer to defect free software, but still leave much to be desired. Peer Reviews of code give us good results but often fall short of catching more than 70%. Today, formal verification processes hold much promise for reducing mistakes in logic, allowing teams to catch issues a test cannot. However, even the latest methods only focus on catching a small subset of potential problems.
When teams combine multiple processes to monitor, analyze, and test code quality, the number of defects are reduced by as much as 90%. Case in point is Kevin, the CEO of a company that rates credit worthiness of Government Contractors through automated software. During a scheduled feature update, I assisted his team with a complete process.
In the past, Kevin’s team missed deadlines due to back and forth with Quality Assurance or QA. However, after implementing and training the team on peer-to-peer code reviews, automated testing, and formal verification of calculations, the project passed QA the first iteration. Reducing the projected development time by one week for a savings of $12,000 in cost.
A few weeks into production and the usual issues never occurred. Instead of spending frantic days correcting problems, the team was able to focus on applying these processes to future projects. Today, the team delivers nearly defect code in 20 to 40 hours less time than just a few months ago. As a result, the company is able to grow without worrying the software will fail.
Conclusion
Most software is broken but yours does not have to be. You can have rapid software development with nearly zero defects just by incorporating a few processes. These may include any combination of specialized peer-to-peer code reviews, automated testing procedures, memory profiling, mathematical proofs and/or many other strategies to ensure the hidden defects do not cost you like the ones you just read about.
I have worked for over 22 years on more than 100 software projects. As Lead Engineer for a Hedge Fund, where millions could be lost on a single defect, I created a system that eliminates 50–90% of these problems during development. Later, adjusting it to other projects to reduce those hard to discover bugs that seem to surface months after going into production. Today, I assist companies by creating custom processes to do the same.
As you were reading this, you probably thought of a friend or network who could benefit from it. Please share it with them as I would really appreciate it. Want to learn more about making better financial software, then visit https://toddmoses.com.
