Cynefin and Software Testing — The Obvious

Cynefin is a sensemaking framework comprised of five distinct domains — Obvious, Complicated, Complex, Chaos and Disorder. The term framework is deliberately chosen and is important. It is not a categorisation model, though I have used it as one to good effect on multiple occasions. Rather, each domain provides you a distinct lens through which to look at your current situation, to make sense of what is going on and to decide how to act.

It is important to note that just as you would not expect to become intimately familiar with a piece of software by reading documentation, the same goes applies with Cynefin. What I’m presenting here is a brief introduction to concepts in the hope that you will see enough value to explore and experiment with it further.

In the next few posts, I’m going to take a look at each of these domains and how they might be useful to software testers and software development in general. In this post, I’m going to talk about the Obvious domain.

The Obvious describes a space in which the system has rigid constraints. That is to say, the rules are known, are easily understood and can be accurately predicted. Constraints in the Obvious domain either hold or they fail — often catastrophically.

You may well be thinking ‘what is obvious and easily understood to me is not the same as for someone else’ and you’d be right. When I say ‘easily understood’, I mean specifically by the individual or group who is being considered in relation to the domain. For example, a sysadmin team in relation to a system they have been taking care of for years. For them, this is an ontology that holds few surprises.

In this space, one acts by sensing what is going on, interpreting the information at hand and then responding accordingly.

To extend the sysadmin analogy, an admin might receive an alert about memory usage on a particular machine (sense). Given their intimate knowledge of the system, they have a rote troubleshooting response. Look at the process(es) that are taking up memory, look at the logs and so on (categorise), then determine whether the relevant application needs to be managed/restarted (respond). Based on what occurs, they can say ‘ah, yeah it’s sort of problem.’ and apply a rote solution.

This reaction is sufficient in this team’s context for 99% of situations that arise. They could safely call this a best practice. (Here be dragons — read on)

On Best Practices

Best Practice is a word to which we as testers, particularly in the Context Driven community tend to have an allergic reaction. I have been no exception. I’ve very slightly relaxed my view on best practices and my view is now this: It is possible for best practices to exist only in a very specific context and these contexts are temporal. Put another way, you cannot have context-free best practices or even particularly context loose best practices. The ones you do have will be tightly constrained within to a very specific knowledge space at a particular time. These contexts may repeat, but once the context no longer holds, neither do the best practices and indeed, between repetitions the previous ‘best practice’ may no longer be ‘best’. In short — it is useless to generalise best practices, though you might codify them for a very specific context that regularly occurs.

The example that Dave Snowden likes to use is the counting of swabs and medical instruments at the beginning and end of a medical procedure, so none are left inside the patient. What about from a software testing perspective? Are there practices you can take from project to project and apply as best practice? Not realistically, no. You might have some best practices within a given project that exist for that project alone. You might have some go-to approaches to investigation, troubleshooting and experimentation, but they are either applied with due consideration or by rote neither of these qualify as ‘best practice’. Indeed, indiscriminate application of best practices is an example of a rigid constraint that could fail spectacularly.

The Cliff of Complacency (or The Problem with Best Practices)

Imagine you’re in the business of transporting goods between two villages, one at the base of a mountain, the other at the top. Your couriers are the fastest and the best and they have the shortest routes memorised. In fact they know them so well they can run blindfolded along the narrow tracks and perilous cliffs — which they do, to prove how awesome they (and you) are. Other couriers try and compete, but you own the lion’s share of the market and it’s no real hardship to undercut them until they fail.

One day, you notice your sales have tanked hard. Well, you know how this business works. You’ve been at the top of it for years. No one knows more about it than you, so you double down on your best practices. You look at your people and lay off the slowest runners, then train some younger, faster couriers. Strangely, sales continue to decline. You talk to your people and they tell you the routes have changed — it just takes longer to get up the mountain.

You talk to the village elders to see what’s going on. They tell you there’s a new player in town and they can ferry goods in a fraction of the time at half the cost. What? Outrageous. Impossible. You go out to check the routes your people have always run. Along the most treacherous part of the route where the path is narrowest and the drop is fatal, the path has been carved away and in its place is an ingenious system of ropes and pulleys leading from the base of the mountain straight to the top. If that wasn’t infuriating enough, the cargo that your people usually carry is being hauled up and down by these ropes at a rate you couldn’t possibly hope to match. It’s not fair. Not only have they changed the game, they’ve destroyed your ability to compete.

You petition the village elders. What this new company is doing is unethical. It’s going to put hundreds of couriers out of work. It must be stopped! To your dismay, they are unsympathetic. One of the elders smiles at you and says ‘We appreciate your business. It’s been our lifeline for years, but we can’t ignore this new technology when it saves us so much time and money. Remember what you said to the other couriers when they complained about your monopoly? — It’s not personal. It’s just business. You either find a way to be competitive or you starve.’

This courier business is not alone in having been an apex predator who was unceremoniously toppled. Uber has done this with taxi companies worldwide. iTunes and Amazon were the death knell for record and music stores. Netflix was the death of the video rental industry and now with Amazon they are changing the face of movies and television. Hollywood production companies are still in their denial phase and they’ve doubled down on remakes and reboots of existing successful franchises. Meanwhile the vast majority of the writing talent has gone to the new players.

When you have a system that works better than anything else, then you codify it and you get very good at doing that thing. They are your ‘best practices’. You make enhancements, you polish your process until it’s as good as it can be. This is not innovation. This is refinement and evolution to suit a specific environment. You might be the best of the best for a number of years. That is until the environment changes and it suddenly doesn’t work as well (or at all) anymore. In Cynefin, this is called the ‘cliff of complacency and it is why there’s a small curl at the bottom of the image. It depicts a drop off from the Obvious space into Chaos.

It happens when you have an ignorant application of best practices over a prolonged time in an environment that can no longer sustain them. By ignorant, I don’t mean stupid. If an apex predator has over-evolved specifically to take advantage of the environment and the environment has irrevocably shifted, then they are acting from a place of ignorance despite not having started there. If the apex predator cannot recognise this change of environment and sufficiently adapt, they are doomed.

When you experience a situation like this, typically the following things happen. You double-down on your best practices — if something isn’t working well, do more of it, or do it ‘better’. This makes sense if you don’t realise the state of play has irrevocably changed. Once you do realise, you might alter your behaviour and adapt, or you enforce tighter constraints to try and reshape the environment — as an example, taxi companies and hotels lobbying governments to outlaw Uber and Airbnb respectively. You can make the same argument with Fossil fuel companies lobbying to suppress the rise of renewable technologies. You don’t have to look too far to find examples of large incumbent structures fighting to turn back the inevitable tide of change. Trying to enforce more rigid constraints may work for a while, but at some point that will stop working too and so you’ll shift back to chaos with increasingly violent oscillation between Obvious and Chaos until you achieve catastrophic failure.

What does this have to do with Software Testing?

I’m glad you asked.

The real value for me in understanding the obvious space is its proximity to chaos. It is a reminder against complacency. Even if you have dotted your i’s and crossed your t’s, on a procedure that has worked for you for a considerable amount of time, success is not guaranteed.

For any non-trivial software development project, your value as a tester will be in uncovering the things that reside outside of the Obvious space at any given point in time. For the things inside the Obvious space, you can further add value by continuing to pay attention to the things that others tend to take as gospel; maintaining your sense of uncertainty when everyone else is losing theirs.

For example:

Automation

• Non-deterministic failure If something fails intermittently, don’t take ‘oh it’s green on re-run’ as sufficiently good. Quarantine non-deterministic failure until you get to the root cause and can correct it.
• Audits Are you positive that your automation really is checking the things you think it’s checking? Has the project (or even the industry) your software serves changed such that more attention is warranted in other places? Are any of your checks obsolete?

Test Environment and Deployment Pipeline

Anchoring to design or development strategy

In software testing, the parts of a project that fit into the Obvious space is where test cases and traceability matrices might be useful and where you might choose to automate checks that have deterministic outcomes. Of course, whether you should use any of these is still a decision based around the value to be gained from doing so. It is important to also remember in the set of all things that a software tester might find potentially interesting, what fits into the Obvious space is still incredibly small. If you’re feeling confident about your ability to cover the things in your requirements and the problems you can foresee, you might like to keep in mind the vast array of things that are not obvious. I’ll cover the Complicated domain in the next post.

Further Reading Liz Keogh — Cynefin for Developers Greg Brougham — Cynefin 101

Originally published at http://testjutsu.com on April 18, 2017.