Regression testing, or: why the fastest developers might not be the best ones

So the feature works for your first three releases…

Fish Percolator is a small software development house that often helps startups get their technology off the ground by turning someone’s ideas into a reality that they can click on and play with.

There are lots of other people out there doing what I do, but many of them avoid this important subject, perhaps because they know business owners don’t realise quite how important it is.

The problem with the “Fuck it, ship it” mentality

Image thanks: Opbeat

A lot of software development houses will take the idea you have and turn it around really quickly, creating a usable prototype that you can put in front of your customers in the space of a week or so and maybe only for a few hundred quid.

This is often called the “Fuck it, ship it” mentality, and is sometimes summed up by the soundbite “if you’re not embarrassed by the first version of your product, you’ve released too late”.

The emphasis on getting things in front of customers as early as possible is important, and it pays to get out of the mindset of trying to make something perfect before you do; you’ll never notice all the things your users do so you need to get it in front of them quickly.

But lurking to the side of this mentality is a hidden danger: the first version of your product is not the last version of your product. Good software is never done so it pays to do some planning for the future on every release to avoid something called technical debt.

What is regression testing?

When a software house puts out the first version of a product, it’s usually really easy to show how all the features work. The developers probably play with every feature every day, and they’ll be confident enough to show these off in a demo and everything will be fine.

Your first release is usually a proof of the product concept: a way to get your users excited. But it might well not even have some of the core features you need to make this a successful business. First releases probably don’t (and shouldn’t!) even include a way for customers to give you cash.

So you’re going to need a second release. And a third release. If your product is successful you might have a new version with some new feature — big or small — every two weeks or so.

So at the end of the second phase, your developer will give you a demo of the new features and they all work. But how do you know whether the features from the first release are still working? How does the developer? Any change to the codebase could introduce bugs into the existing features.

The only way to be sure is to make sure all features from previous releases are tested at every release. This is called regression testing, because when an old feature breaks as a result of introducing a new feature, it’s called a regression.

Regression testing can be done manually (someone has to personally try out the feature and make sure it is still working as expected) or it can be automated (where someone writes code that performs the same actions as a human would, and lets you know if it passes or fails).

Technical debt

Image credit: StockUnlimited

Not automating your tests creates a kind of technical debt. Technical debt is where you compromise on technical quality in order to get something ready on time, or because you don’t have the relevant skills to achieve it, or because you don’t think the client will notice and they’re going to pay you anyway.

If your regression tests are not automated, it’s easy to see what happens a few releases down the line. After 3 or 4 releases, it starts to take longer to test all the features than it does to develop the new ones. That’s either going to come to the product owner as a heavily inflated cost or, more likely, the developer is going to have to cut corners in order to keep their prices down. So then you’ve got untested code going out to customers, and that’s when nasty bugs start to show up. HTTP 500 errors. Broken links. JavaScript controls that once worked and now they don’t. That kind of thing.

You can think of technical debt as being a bit like a financial debt. If you “pay it off” every release by automating that release’s features, then your costs each release are higher, but fixed. If you let it build up, then not only is it going to be a big task when you want to pay it all off and automate all your earlier releases, each release the developer has to pay the “interest” of running all the existing regression tests manually.

Forgotten designs

Image credit: StockUnlimited

Another kind of debt is the designs themselves.

When you’re putting a release together, you and the developer are likely really excited about a feature and you’ll work together on the design. The release comes out while it’s all fresh in your mind and you’re super-happy.

Six months later, you think you’ve found an issue. But you swear you said you wanted the user to have to click through two pages here and the developer swears that it was three.

Without good documentation of your features, it’s hard to remember what was agreed at the time, and this can lead to disagreements about whose responsibility something is.

Leaving a long-lasting legacy

Image credit: StockUnlimited

When you hire a software developer, it’s easy for them to drop the code in your lap and run away fast. And if they are believers in the “Fuck it, ship it” mentality, that’s exactly what they’re likely to do.

On the other hand, with a little bit of care and attention, developers can leave a lasting legacy that not only gives you confidence in your product, it’ll encourage you to develop it further, keep it up to date, and maybe even hire developers of your own one day.

At Fish Percolator, we use a tool called Spinach, which is a variant of the extremely popular Cucumber system. Spinach uses the Gherkin format of executable documentation.

Effectively, developers and the business can work together to write documentation of the features in a language that can be understood both by humans and machines, and then those descriptions can be turned into automated regression tests. The documentation can be picked up by new developers who want to know how the system is meant to behave, and the tests ensure that no one changes the behaviour accidentally.

Here’s an example of some Gherkin from one of our current projects:

Hopefully you and I can both understand that, and a continuous integration system (we use Codeship) will also understand it and test it forever, whenever anyone makes a change.

Writing code quickly is important. But writing code that will last and continue to make you smile for the lifetime of the product is just as important. When you want to make something cool on the internet, it pays to make sure it stays cool!

If you want to talk to us about the ideas in this post, or anything else, consultation is always free!