Automated testing: What you should know

On an essential tool for delivering high-quality software

Navigating a code base is like swimming in a swamp at night

Do you care if the features of your product work as advertised? Does it matter to you if your product crashes when your customer’s using it? Do you worry that buggy code may help attackers take down your system, or worse steal your user’s data?

If you answer yes to any of these then what you need to employ are automated tests. They are one of the most important tools to give you confidence in the quality of your product. Done right they can:

• Ensure new releases deliver the features and bug fixes as promised in your release notes
• Ensure older features don’t break with new releases (i.e. minimize regressions)
• Increase developer productivity

You might wonder “why would it make developers more productive?” — let me explain. Most developers work in silos, they know a certain, small area of the code base and they can make changes to that. They can add new features, fix a bug or do the occasional refactor (i.e. code changes for the sake of future development). One thing they typically can’t do is understand the entire impact their changes might have on the rest of the product. This is because most code repositories are sort of like enchanted swamps that you have to navigate at night.

Now, I am sure you guessed it: Here’s where automated tests come in. With them in place a developer can now make a change and be rest assured that nothing else in the product was affected by their change if all automated tests are passing. This, of course, assumes that these tests cover all (or nearly all) features of the product.

Without automated tests, the developer or their best friend, the test engineer, would have to exercise each feature manually which can take a very long time (think days or even weeks) making the feedback loop back to the developer extremely lengthy.

OK, but what are they?

Automated tests are code that typically runs either on every new change to the production code or on a schedule like once a night or similar. Think of an automated test as a building inspector making sure your house is in order, except they come by without asking. Once the test suite (collection of tests) is finished you’ll get a test report letting you know which tests have failed if any.

There are different types of automated tests. The most common categories are:

Unit tests

• Small and fast tests written in the production language
• They test a unit of code, i.e. a function or class
• They don’t do I/O, i.e. no networking or file system access
• They are very reliable and can run in isolation, not requiring any significant setup
• Code coverage (how much production code was exercised) can be meaningfully measured and reported

Integration tests

• Are typically written in the production language
• They test the integration of larger components of code
• They are allowed to do I/O
• They typically require significant setup
• Measuring code coverage is not meaningful for understanding how well the production code is tested

System tests

• Are written in a language convenient for testing (e.g. Python)
• They test the product as a whole, often from a user perspective
• They require maximum setup, i.e. anything needed to run the product
• Are less reliable due to complexities in setup and tests themselves; false positives are not uncommon
• The test suite should output the list of features tested

In an ideal world, you’d write tests in all of these categories, and lots of them. For a green field project that’s great! However, developers often find themselves in a situation where an established product has no automated tests to speak of.

There are no tests.. help!!

Don’t worry, nothing’s lost. With a bit of love and time almost any product can be brought back to the modern world of software engineering. If your code doesn’t have tests at all I would recommend to start writing system tests and keep adding new ones as you find time and features are developed. Eventually you’ll have so many tests that releasing a new version becomes a breeze. Just wait for your test suite to report green, verify untested features manually and off you go!

The next most important category are unit tests because of how reliable and easy to write they are. You’ve got a function? Just unit test it, ensuring it’s always delivering on its promise as time goes by. Unit tests can be written in isolation without knowing the rest of the code base. Beautiful!

Now say you’ve been writing unit tests for a while and have lots of correct functions. Putting all these functions together into a component will now most likely result in something working correctly as well. To make sure just write an integration test!

But what are good tests?

When you’re writing tests keep a few guiding principles in mind:

• Tests should be reliable (no one wants to debug your tests)
• Tests should be small and readable
• Test the happy path but also edge cases. Both are important
• Automate your tests using Jenkins or similar
• Add new tests for each feature and bug fix

Now what’s the most important property of tests you might ask. I’d say it’s the fact that they exist at all! Thanks for reading!