Practical Guidance for Design Studies

How to conduct effective design studies for problem-driven visualization research.

Photo by William Iven on Unsplash.

Visualization researchers deal with real-world problems for domain experts by analyzing the problem and designing & validating visualization systems. Sedlmair et al. (2012) define this kind of project as “design studies”. Design studies have become increasingly popular in recent times. However, while patterns how to do this have emerged, no systematic guidance has been provided yet. This means that researchers often repeat the same mistakes. Therefore, why not propose a holistic methodology that can guide visualization researchers through a smoother design study? This is what Sedlmair et al. were trying to do in their 2012 paper “Design Study Methodology: Reflections from the Trenches and the Stacks.”

Why use a design study?

Design studies can lead to three main contributions:

1. Problem characterization and abstraction

Design studies can help to establish a shared understanding between visualization researchers and domain experts by clarifying fuzzy tasks and the uncertain scope of the project. Also, requirements for solving the problem can be reused in future studies as well. Moreover, by articulating and externalizing the domain knowledge, design studies can help to automate the problem-solving progress.

2. Validated visualization design

Design studies usually involve validations of proposed designs, which can testify to its effectiveness in solving the domain problem for the experts.

3. Reflection

Retrospective analysis is a unique part of design studies. It allows researchers to learn the lesson from the project thereby further improving design/evaluation/process guidelines.

Fig.1. The task clarity and information location axes (Source: Sedlmair et al. (2012))

When not to use design studies?

There is a large field to adopt design studies with different characteristics (see the white space in Figure 1). However, note that it may not be suitable under two situations:

• When researchers don’t have access to enough available data, they are possibly not able to create an effective visualization design to solve the problem, or
• When the required task is clear and concise with enough information that can be done using an automatic approach. It can be an ideal way and a final goal of design studies, but if it is already achieved, it is not necessary to conduct design studies anymore.

Fig.2. Nine-stage design study methodology framework (Source: Sedlmair et al. (2012))

The nine-stage framework and pitfalls

Based on their own research experiences and an extensive literature review, Sedlmair et al. (2012) proposed a nine-stage framework for design studies split into three phases: a precondition phase, a core phase, and an analysis phase. Overall, it is a linear process, but it’s also flexible for different situations. The previous stages do not need to be fully finished to move on to the next one, and if need, jumping back to earlier stages can be helpful as well.

1. Precondition phase: learn, winnow, and cast

During this phase, researchers need to get enough knowledge by reviewing the relevant literature. This will ensure a solid foundation for their future designs. Then, during the initial meetings, researchers should talk to a broad group of people and then select several suitable collaborators by asking: if they can provide the real data in time; estimated project time; the problem they are looking to solve and also by confirming that there is a real need and it is meaningful to conduct the project. After that, collaborator roles should be identified.

Common pitfalls: skip stages; start with insufficient knowledge; collaborating with the wrong people (with no available data, limited project time, meaningless project focus); not identifying/misunderstanding collaborators’ roles).

2. Core phase: discover, design, implement, and deploy

Here comes the main part of the framework. At this stage, researchers are going to first define the research focus and goals. Then they will discover how visualization can help solve the problems and fulfill needs for the domain experts. To do so, they should not only ask the users what they want but also do interviews or observations to identify real needs. Then they can start to design a visualization solution by considering a set of possible proposals. During this stage, they may also implement the design by prototyping. Finally, researchers should validate the design by releasing it and gathering feedback from users.

Common pitfalls: poor task clarification and abstraction; limited to one or few design solutions; taking too long to prototype; focusing too much or too little on usability; no-enough deploy time; unreal/ineffective user test.

3. Analysis phase: reflect and write

Finally, in the analysis phase, researchers are going to critically reflect on the design study research they have done, which helps to improve current design guidelines and will further benefit other researchers as well. Meanwhile, they can also begin to write about the design study.

Common pitfalls: no reflection; insufficient writing time; inappropriate content/structure arrangement.

Conclusion

The nine-stage design study framework can be an efficient way of conducting good design studies. Many aspects of the framework align with other research methodologies from the HCI field. Sedlmair et al (2012) believe that there is still room for the framework to expand and its adoption to grow. Applications and further studies of this methodology are expected.

This blog post is inspired by the following paper:

• Michael Sedlmair, Miriah Meyer, and Tamara Munzner. (2012). Design Study Methodology: Reflections from the Trenches and the Stacks. IEEE Transactions on Visualization and Computer Graphics, 18. 2431–2440. 10.1109/TVCG.2012.213.