5 Minute DevOps: Effective Testing

Bryan Finster
Jun 18, 2019 · 8 min read
Image for post
Image for post

Protecting the Business

Developer driven software testing is not new. Studies show that high performing organizations depend on tests written by developers instead of handing off to external teams of “Test Automators”. Many vendors sell The One Test Tool to Rule Them All and promise to take the thought out of testing. I’ve seen repeated demos of tools with slick UIs designed to dazzle those with budget control that promise to make testing so easy that even external contractors with no context can do it. I’ve been told that only unit tests are the responsibility of the development team. I have even heard that for developer productivity testing should be given to testing teams because, “developers have better things to do and aren’t qualified to test anyway”. Wow!

CI/CD == Test

For anyone wishing to master CI/CD, there are few areas of study more important than the effective use of metrics and testing.

  1. Keep the pipeline green. If there is a broken build, drop everything and fix it. If you cannot ship, nothing else you are working on matters anyway. Make sure that change can be delivered on demand to protect the ability to fix production without heroics.
  2. When (not if) a defect penetrates, harden the pipeline further by designing additional efficient tests for that defect.
  3. Write features and ship them daily to continuously test our ability to fix production with minimal MTTR.
  4. Get feedback about the status of the changes as fast as possible.
  • There is only a single path to production for any code change, no matter how critical. We always use our emergency process and we always test the same way.
  • Tests version with the code.
  • Test may not be deactivated for an emergency.
  • While there may be a button to push to deliver to production, there are no other manual stage gates, human intervention, or handoffs to external teams once code has passed code review.

Building a Resilient Test Plan

Your test suite needs to be hardened against the realities of the chaotic world. In a fantasy testing world, networks never have latency, external dependencies are always up and always stable, and the teams who own those dependencies never deploy breaking API changes without versioning. It’s fine to assume everything will work perfectly, until it’s an emergency and your assumptions are sadly mistaken. You MUST trust your tests. You must not accept random failures in your tests. Your critical path pipeline tests must work even when everything around you is falling apart.

The Table Stakes: Solid Unit Tests

Unit tests are absolutely the foundation. They are also where most developers stop. While unit tests are crucial, they need to be used for the correct purpose; black box testing of complicated, isolated units of code. Typically, we are talking about classes, methods, or functions at this level. For architecting a Continuous Delivery test suite, we want it to be fast and efficient. We want to target testing at risky behaviors. 100% code coverage is neither reasonable nor efficient. There’s no purpose in testing things like getter and setter methods unless they are doing something interesting. Testing Java’s ability to assign data to a variable isn’t useful unless we are developing the JVM. There are libraries full of good information on patterns and anti-patterns for unit testing. A favorite is xUnit Test Patterns. It’s a massive tome on proper testing and common anti-patterns that I highly recommend.

Meeting the Business Goals: Functional Tests

Unit testing will give a good view of how individual units will perform, but most applications are implementing some business flow. Testing that core business domain logic is critical and unit tests are ineffective at that. Many teams will jump immediately to end to end testing to solve this, but those are unstable (flakey test anti-pattern) and cannot effectively test all of the logic branches.

Being a Good API Citizen: API Contract Tests

Communication interfaces are where most defects occur. It’s obvious then that we should prioritize testing those interfaces even before implementing the behavior behind them. This is where contract testing and contact driven development become important.

Playing Well With Others: Integration Tests

Contract tests give you a level of confidence during CI builds that you’ve not broken a contract or broken how you consume one but since they are mocked, they require another layer of test to validate the mocks.

Verifying User Journeys: End to End Test

End to End testing tests for a flow of information and behavior across multiple components. Beware of vendors selling record and replay testing tools that purport to take the load off of the developer by simply testing the entire system this way. However…

What Didn’t We Test?

Exploratory testing is needed to discover the things that we didn’t think of to test. We’ll never think of everything, so it’s important to have people on the team who are skilled at breaking things continuously trying to break things so that tests for those breaks can be added. Yes, this is manual exploration but it’s not acceptance testing. If you use a checklist, you’re doing it wrong.

Will it Really Operate?

Load testing and performance testing shouldn’t be things left to the end. You should be standing those up and executing them continuously. There’s nothing worse than believing everything is fine and then failing when someone tries to use it. Operational stability is your first feature, not an afterthought.

Entropy Happens

  • What if that critical dependency isn’t available?
  • What happens if it sends you garbage?
  • Cloud provider reboots for an upgrade or has an outage?
  • Excessive latency?
  • Coronal Mass Ejection?

Testing Ain’t Easy!

Proper testing takes the right mindset and at least as much design as production code. It takes a curious, sometimes evil, mind and the ability to ponder “what if?” Proper test engineers don’t test code for you; they help you test better. Tests should be fast, efficient, and should fully document the behavior of your application because tests are the only reliable documentation. If you aren’t confident in your ability to test every layer, study it. It’s not an added extra that delays delivery. It’s the job.

Walmart Global Tech Blog

We’re powering the next great retail disruption.

Bryan Finster

Written by

Value Stream Architect and DevOps insurgent who optimizes for sleep. All opinions are my own. https://www.linkedin.com/in/bryan-finster/

Walmart Global Tech Blog

We’re powering the next great retail disruption. Learn more about us — https://www.linkedin.com/company/walmartglobaltech/

Bryan Finster

Written by

Value Stream Architect and DevOps insurgent who optimizes for sleep. All opinions are my own. https://www.linkedin.com/in/bryan-finster/

Walmart Global Tech Blog

We’re powering the next great retail disruption. Learn more about us — https://www.linkedin.com/company/walmartglobaltech/

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store