My Internship Days: When I Underestimated the Crucial Role of Software Testing.

Lessons Learned the Hard Way in the ’90s Tech World.

In 1994, I was doing a job internship at a small company that had recently lost most of its IT staff. The company previously relied on a group of technical experts. Still, when they decided to part ways and split the company, the owners were left severely understaffed in the IT department.

Suddenly, I was the company’s only IT resource, shouldering various tasks from maintenance and IT support to development and various other IT-related duties. Those were when a single developer could grasp the entire scope of a software solution; software was quite basic compared to even the most straightforward solutions we have today.

Given the relatively small scale of the applications in our portfolio, we could answer any questions and address any issues promptly. However, time was always scarce; with bills to pay, bank errands to run (thank goodness for the advent of online banking!), support calls from clients unfamiliar with computer operation, and in-person upgrades that had to be executed due to the limited access of the internet. We were stretched thin, and software development became a precarious balancing act.

These conditions were ripe for catastrophe, and not long after starting my job, I had a firsthand experience of the drawbacks of not conducting Software Testing.

The era when a single developer could comprehend an entire software solution is long gone. Applications were much smaller in the early ’90s and even smaller in the ’80s. For context, Lotus 123, the first commercial spreadsheet application, was created and maintained by a single person. Given the small scale of applications, there was a strong belief among us developers that we could design, code, and test every conceivable scenario without formal testing. We used to make ad hoc software modifications and distribute them to customers immediately.

Though our applications were small, they were complex, dealing with intricate business rules for importing/exporting goods, calculating taxes, and classifying goods. Despite these complexities, we were confident in fixing any issue promptly without causing side effects. After all, we had created the software to control it.

That was until disaster struck.

In those days, the standard method of data storage, in case of an application crash (which happened frequently), was to save temporary files on the hard drive using rudimentary techniques. Our software would scan the current folder for these temporary files to recover any work in progress. One week, we received reports from several clients that temporary files were being deleted mistakenly. Each time they completed a form, old data filled the fields, impeding their work. They had to delete all the form fields to start anew manually. We termed this a P2 incident:

“Priority 2 (P2) — A major part of the client’s operation is affected. Some aspects of the business can continue, but it’s a serious problem.”

Such incidents required swift resolution as they disrupted our clients.’
operations, causing delays and a high risk of not completing work during the business day. This delay could affect their clients’ import/export operations. The urgency was evident.

Have I emphasized enough the risks of not performing Software Testing? If not, I’ll continue.

In response, we dove into the application code, identified the problem, and it took just one line of code to “resolve” the issue. We compiled the application, made several copies to send to our clients, and personally delivered and reinstalled the software for each client affected manually. We skipped testing; it was just one line of code.

We didn’t hear from any clients that day, which was a positive sign. However, a desperate client left numerous voice messages on our answering machine the following day. They were seriously panicking, and we couldn’t understand what was happening; we had to pack our single laptop and head to their offices.

A historical note: disk drives were small back then, and software installation was often manual. Applications were not as sophisticated as they are today. Like many others, our client had installed all software applications in the same folder, including their accounting system. When we ran our newly patched application, it deleted our temporary files and, unfortunately, several files from the accounting system. They lost all accounting entries for that year.

You might think we could have recovered the files, right? Well, it wasn’t that straightforward. First, you needed specialized software to recover files; concepts like the Windows Recycle Bin didn’t exist. Secondly, and most importantly, due to the sensitivity of the data we were dealing with, we designed our applications to ensure efficient and non-recoverable data erasure.

Just one single line of code without testing.

The fallout wasn’t pleasant for us. We had to go back, fix, test (yes, we finally learned our lesson), compile, and deliver a new version to all our clients. Moreover, we had to spend an entire week, four to six hours a day, at the client’s office, manually entering every entry from the bookkeeping records. The cost of one line of code that wasn’t tested can be summarized in dozens of hours lost on our part reentering data, the client’s inability to use the accounting system for a week, relying solely on physical books for everything, and the extra hours we had to spend fixing the application and installing it in each client’s office. Not to mention the reputation damage to our firm.

Software Testing isn’t a mere option; it forms the backbone of quality assurance. Errors can inflict substantial damage, with repercussions far beyond the immediate issue.

In today’s world, where software solutions often entail hundreds of thousands of lines of code and development teams dispersed across
the globe, the possibility for failure has multiplied manifold.

Embracing Software Testing at the earliest and maintaining its consistent application throughout development isn’t just strategic — it’s indispensable. By prioritizing thorough testing, we can mitigate risks, enhance the reliability of our solutions, and ultimately safeguard the trust that forms the cornerstone of our relationships with clients.