Learning To Love Automated Testing

3 Stops Along My Journey to Automated Testing Enlightenment

Published in

Capital One Tech

6 min readFeb 13, 2019

There are lots of opinion pieces online which explain the benefits of automated testing: better quality, faster releases, and eventually faster development too.

Unfortunately, like any technology, automated testing has a ramp-up cost — you need to learn the tooling and skills. Early on in my career I was put off by those costs, and it took me too long to really feel the benefits.

As I learnt more about automated testing, I found specific examples much more helpful and inspiring than more general, abstract benefits. So, I am going to share some real world examples of automated tests that have saved me lots of time and embarrassment. :-)

1) Finding Fundamental Design Issues

A few years ago I was implementing some code to decrypt some data from the server. The server side team gave me some example encrypted data and I started writing foundational unit tests to check that my decrypt utility was working correctly.

@Test 
public void testDecrypt() {      
  String decryptedData = decrypt(EXAMPLE_ENCRYPTED_DATA);     
  assertThat("...43XabcGYvTwII=").isEqualTo(decryptedText); 
}

Almost immediately I hit a weird problem where the test was failing, even though the expected and actual results appeared to be identical.

Expected: ...43XabcGYvTwII=
Actual:   ...43XabcGYvTwII=

Copying the output from Android Studio and pasting it into a plain text editor, I was able to see some unexpected characters.

Expected: ...43XabcGYvTwII=
Actual:   ...43XabcGYvTwII=����

I went and sat with the server side team and looked at their code with them. We discovered that they were using Zero Padding to pad the blocks. This was a big mistake in the design, as our payload was allowed to contain zero values. This meant if our payload ever ended with a zero, that zero would get stripped off, which would corrupt our data.

To fix this we changed to one of the standard padding schemes supported by Java (you can see a list of them in the Cipher section of the Java™ Cryptography Architecture).

The root cause of this problem was a miscommunication over whether some code was an example prototype, or whether it was an exact specification of what should be shipped to production.

The iOS integration with this server side component appeared to be working fine because the Objective C utilities (this was back in 2015) were automatically stripping the zeros in the zero padding.

The automated test I wrote helped us find this bug before this feature was shipped to production. The issue would only have happened once in every 256 times (any time the final byte of the payload was a zero), so I suspect we might not have noticed it…

Takeaway: Make sure you build foundational unit tests for all the components in your system. If you see something odd, don’t ignore it — there could be a big design issue hiding behind it!

2) A Dependency is Broken

When building features, we usually start by writing integration tests against the HTTP APIs that we depend on. The benefits are:

Flushes out any major issues as early as possible during development. This is particularly important if changes need to be scheduled with other teams.
Gives us a quick way to check that the services are up and running.
Provides a regression test suite that we can use to verify that API changes work.

Back in 2015 I was trying to jump from Retrofit 1.6.0 to 1.9.0. I bumped the version number and then ran my tests. To my surprise, I had three failures — here’s my original Slack message where I asked for help:

The authors were very quick to confirm that this was an accident (https://github.com/square/retrofit/issues/854) and that the fix would be in version 2.0.0. They subsequently removed the 1.9.0 release from GitHub, and 1.6.0 is still marked as the “Latest Release”. Retrofit is an awesome library, and the maintainers handled everything well. :-)

The key takeaway for me was that I was able to avoid opening a pull request containing buggy code. I avoided having bad code reviewed, merged to master, tested, or even shipped to production. I dread to think how much time it would have taken to diagnose some edge case crashes due to behavior of a third party dependency.

Similar API integration tests have saved us twice recently: a newer version of an HTTP API was released and our automated tests immediately revealed multiple edge cases that the newer API had broken! 🎉

Takeaway: Write HTTP API integration tests — they’ll help you catch API issues early and will speed up debugging issues when an API is down or misbehaving.

3) Preventing Functional Problems

Many years ago, I was working on an Interactive Voice Response system system (IVR). If you’ve called a customer support line and gotten a long, complicated menu then you know what an IVR is. :-)

The main engineers had a huge suite of automated tests, including some tests which checked that the different schedules kicked in at the right times. In this example, we expected the IVR messages to change when the office opened at 9am.

testClosedOneMinuteBefore() {
  assertEqual(convertTime("8:59am"), "sorry, we are closed")
}testOpenExactStartTime() {
  assertEqual(convertTime("9:00am"), "welcome to ABC corp...")
}testOpenOneMinuteAfter() {
  assertEqual(convertTime("9:01am"), "welcome to ABC corp...")
}

I did a giant refactor, and along the way I took some code like this

if (startTime >= currentTime) {
  // Offices are open
} else {
  // Offices are closed
}

and I inverted the if test to this

if (startTime <= currentTime) {
  // Offices are closed
} else {
  // Offices are open
}

Can you see what I did wrong?

The problem was because I forgot my basic computer science rules on how to invert a boolean condition. Inverting <= should give >, not >=.

Fortunately, the testOpenExactStartTime() test failed as soon as I made my change, and it was super easy to diagnose and fix. I dread to think if I would have spotted the schedules starting 1 minute too late in my manual testing…

Takeaway: Use automated tests to help make sure nobody breaks your code! Make sure your edge cases are well tested, and anything that would be slow to test manually. :-)

I ♥ Automated Testing

Hopefully these examples help demonstrate the real world benefits of automated testing. Once you’ve learned the tooling for automated testing on your platform, your development with speed up and your quality will improve. It’s incredibly liberating to rely on your tests to keep your codebase stable.

When I first started working on mobile apps I would have a multi-week code freeze leading up to a major release, frantically fix bugs found during manual regression testing, and then nervously watch the crash rates and analytics for 2 days after every production release. Now I don’t need to. :-)

My team are hiring Android developers. If you know anybody who might be interested please send them the link and introduce me to them!

DISCLOSURE STATEMENT: © 2019 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.