Reliable Systems Series: Model-Based Testing

The Temptation of Saint Anthony

Model-Based Testing

It’s OK to Feel Skeptical

What Is QuickCheck?

Why is this useful?

  1. We are essentially building a fuzzer, dramatically increasing the coverage that is possible with a relatively small amount of test code.
  2. QuickCheck will generate test cases that you may never have thought to check in a more traditional unit test. But once QuickCheck has found a failing case, you can copy and paste the input into the body of a new regression test. It writes your regression tests for you, and the shrinker keeps them simple! Your new machine-generated regression tests are deterministic to the extent that your implementation is deterministic.
  3. By randomly generating more complex inputs, our tests are less vulnerable to the pesticide paradox, and longer test runs can keep turning up bugs over a long period of time, keeping your engineers busy with deterministically replayable (far easier to fix) bugs.
  4. The generated QuickCheck test cases are not necessarily deterministic (look at how the random number generator is seeded for the implementation available to your languages of choice), so we can basically get the best of deterministic and non-deterministic testing by turning failing inputs into regression tests. This is particularly nice for inputs that fail less frequently, as QuickCheck may only generate a few dozen to a few thousand inputs by default. This is almost always configurable, which is particularly nice for running longer burn-in tests when desired.
  5. Because we, the implementors, are not thinking up specific test cases, our biases do not impact the test quality as much. The machine is only impacted by the biases of our generators, which is usually a significant improvement. QuickCheck frequently generates test cases that amaze me, both in terms of the subtlety of the bugs that they uncover, and the seemingly-magical ability to shrink the failing case down to something I can reason about easily to fix the bug.

QuickCheck + Models = ❤

Enough Blabbering, here’s the recipe! DO IT NOW!

  1. Decide to implement a thing.
  2. Create a toy model of how it should work.
  3. Write a test asserting identical behavior between your implementation and your toy model, given a randomly-generated sequence of operations that will be applied to both.
  4. Build the thing. when the model and implementation diverge, your assumptions have been invalidated. Your implementation may be incorrect, your model may be incorrect, or both are incorrect.

Building a Tree

Writing the Model

enum Op {
// our K and V are both of type u8
Insert(u8, u8),
Get(u8),
}
[package]           
name = "btree"
version = "0.1.0"

[dev-dependencies]
quickcheck = "0.6"
rand = "0.4"
λ tree                    
.
├── Cargo.toml
├── src
│ └── lib.rs
└── tests
└── quickcheck.rs
2 directories, 3 files
λ cargo test                                                                                                     
Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running target/debug/deps/btree-9ef8f49ee1976a3b

running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered outRunning target/debug/deps/quickcheck-dc970f46510fa339

running 1 test
test implementation_matches_model ... FAILED

failures:
---- implementation_matches_model stdout ----
thread 'implementation_matches_model' panicked at '[quickcheck] TEST FAILED. Arguments: ([Insert(0, 192), Get(0)])', /home/t/.cargo/registry/src/github.com-1ecc6299db9ec823/quickcheck-0.6.0/src/tester.rs:171:27
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
implementation_matches_model
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered outerror: test failed, to rerun pass '--test quickcheck'
[quickcheck] TEST FAILED. Arguments: ([Insert(0, 192), Get(0)])

All Watched Over By Machines Of Loving Grace

You Have Been Challenged!!!!

Dealing with New Abstractions

If it’s hard to model or check specific invariants of a system, it could be a symptom that the architecture is unsound, leading to unreliable implementations and high human costs for debugging over time. If you can’t describe it in a model or set of invariants, good luck getting humans to understand what should be happening.

That’s it For Now!

--

--

--

webmaster http://tylerneely.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

JBOSS Wildfly and JMX Prometheus exporter

How to install ROracle on Windows 10

Bandwidth Throttling in Web Browser

Patterns for infinitely scaling & cost effective serverless microservices

CS 373 Fall 2021 Blog 7

An elegant way to dump MongoDB using Docker

Tmux Tutorial : Part II

lift default shortcuts

Best moments of coding in 2019–20

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tyler Neely

Tyler Neely

webmaster http://tylerneely.com

More from Medium

Get Set Go(lang).. as a Java Pro

Functions, Arguments, and Pointers

Hexagonal architecture: catch ‘em all

Unleashing the Power of Bio-inspired Parallelism With Membrane Computing