Reflecting on the design of test tasks

I’ve been reading in “Seeing Like a State” about the polarity between deep/thick and shallow/thin understanding. About how tunnel vision provides sharp focus on certain aspects, while creating many blind spots and oversimplifying assumptions as well.

I’m thinking about the relationship between these concepts and how I design my tests in order to cover both in depth and in width. Inspired by a few rules of thumb that the book presents for making development planning less prone to disaster, here are my thoughts on a few rules of thumb for making test tasks or sessions less prone to disaster (in other words boring people while missing important points).

Rules of thumb for designing test sessions:

  1. focused, yet loose;
  2. small and meaningful;
  3. design in flexibility to accommodate surprises;
  4. make room for other perspectives.

focused, yet loose

Identify a few important topics to search for and look at. These can start from risks, flows, functions, or variables, depending on the model of the app I use to design the test session.

During testing, I want people to explore the application from one perspective at a time. That’s what I want to focus testers on: a perspective from which you’re testing. I don’t want to give people a step-by-step guide to do what I thought of but didn’t have time to do. Instead I want to provide a guide that is helpful for sparking ideas around a theme, for whoever will be performing the testing. A focus point to develop on.

small and meaningful

I choose one of those models or a few related for a session/task. I want to avoid making a session big, looking dreadful. Instead, each session is a small step that looks accessible and doable.

I want the description to be clear enough so that I or another tester will know what questions to ask the application.Also motivating enough so that I have hope that there are interesting cases to think of, ones that stakeholders will care about.

design in flexibility to accommodate surprises

I don’t know in advance all the bugs I’m looking for. I want to be able to act on information I find during the session, instead of being pressured to stick to ideas I thought about before I started testing. So I will list ideas I am not sure how I will explore. I don’t need to figure out all the details, although I don’t avoid looking in detail in some directions.

I formulate ideas as questions I want to find answers to. How should this behave? I want to know in advance enough to create useful models of it, but I don’t want to be left with no more questions about it. It’s important to remain in a questioning state of mind while I design the test task, and throughout performing it, until the very last test.

Suggesting to try to get into seemingly impossible states can stimulate the people who perform the sessions to look beyond the obvious and familiar to them.

room for multiple perspectives

I add seeming redundancy where variety is precious. Sometimes it’s worth to cover different platforms or insert other independent variables that increase coverage to a test session theme. So I will end up with the same session, but ran in different contexts. If it will be ran by different people, it increases the variety of ideas.

This doesn’t pay off for every task, and doing it for too many can quickly increase the effort, so I keep in mind to choose wisely.