Making the invisible, visible
Design is regularly being implemented in diverse settings, but as a community, we can’t just do design without reflecting on the impact we have when we do the design we do and make the decisions we make. Measuring impact is possible when the process and results are quantifiable. But design, in its very nature, deals with the complexities of qualitative factors such as human behavior and experiences. This often means that our pursuit of design involves understanding intangible factors of society such as relationships, motivation and emotions — factors that cannot be seen but are certainly felt. These are more problematic to understand and map, because they often get oversimplified or ignored from decision making processes. Without being able to appropriately understand the impact of intangible factors, we cannot fully assess the intentional and unintentional consequences of our designs. Given the qualitative nature of intangible factors, this is always going to be challenging to do. However, can we really be responsible designers without a thorough understanding of the impacts of our designs?
Why do we value measurability?
Society values the ability to measure the world around us for three core reasons.
- Metrics create a common language and understanding about the values that are important to an organization. This common language helps to set clear expectations for performance, progress and ultimately, helps to define success.
- The perception of certainty that is produced by measurement makes it much easier to set goals and define the path to achieve them. This makes it easier to get buy-in from people and enact change.
- We value measurement because of its ability to produce feedback against the metrics and goals that we set. This gives us the opportunity to ensure that the change being created is having the intended impact.
An example of the way that organizations introduce metrics is through key performance indicators (KPIs) that are tailored to each role and team. They have been used extensively to understand the state of the organization according to the tangible measurements set and expectations for growth. As corporations have grown in scale and profit, the ability to measure through KPIs has provided clear structure for growth efforts. The resulting boom in growth has enabled organizations to create significant impact on our society today (for better or worse, you can make your own assessment).
Currently, there is no such clarity and structure in place for intangible factors. As a result, they are not being taken into account or are being misrepresented by proxy variables that are measurable. For example, organizations use repeat buyers as a measure of trust which does not reveal anything about the actual impact on the buyer and oversimplifies the complex reasons why people trust organizations. As designers, we have a responsibility to get below the surface, to understand these relationships and illuminate the gaps that are created when only taking into consideration tangible, quantifiable data.
Facing the biases in fact
While finding a way to accurately measure intangibles would help solve some of these issues, bias in the collection of data can create some significant distortions in interpretation. First, a fundamental issue with measurement is that the data it creates is viewed as being neutral and a reflection of reality (Brain, 2018). However, how we measure the world around us is a highly political issue as only a few have the privilege to decide what type of data is important for us to collect or not (Brain, 2018; Onuoha, 2016). As a result, our perception of reality is shaped through the lens of the priorities of the privileged, potentially imposing a singular world view.
Second, measurement boils information down to factors that are able to be analyzed, abstracting the world and removing some of the messiness in the process (Brain, 2018; Onuoha, 2016). This can, once again, distort our understanding of reality through the story that data tells. An illustrative example is the Allegheny Family Screening Tool (AFST) in Pittsburgh. The AFST is a predictive-risk modeling tool that is aimed at improving initial child welfare screening decisions, ultimately increasing objectivity in the system. However, only public databases are taken into consideration (Hurley, 2018). Given that users of the public system are often in the lower socio-economic brackets, it unfairly biases screenings against this group due to the setup of the data available. As a result, the reality of this data is that children in low socio-economic families are at a higher risk of being mistreated. While data gaps mean that this is not the full picture, design and policy decisions are made on the basis of such findings, potentially having a large, unintended impact on those most vulnerable. This depicts how important it is to ensure that we understand the qualitative story and question the reality of fact.
Now what?
Given the clear need to understand the intangible and that the current mechanisms of measurement are flawed, how do we measure the impact? As intangible factors are distinct from the tangible, we should look into alternative methods of measurement rather than trying to fit them into existing measurement systems. In her explorations of the fishing industry, Cheryl Dahle, founder of non-profit Future of Fish, suggests a framework that maps activities to Professor F.W. Geels’ theory of how society embraces multi-level socio-technical change.
The ultimate premise of her framework is that many small interventions can begin to create significant change. Here, she maps how we might start to recognize that small, guerilla efforts are beginning to have impact by identifying the level of adoption of the desired change on the left.
Of course, Dahle’s framework doesn’t have the same level of depth that has been developed in systems like KPIs for decades. Given the intangible factors and complex systems that are being assessed, it is likely that this level of clarity is still far off. However, to complement this framework, we can leverage existing qualitative methods that are already present in design to gain insights. Tega Brain, professor at NYU, identifies ethnography as one potential method that doesn’t abstract that information we see around us or bias research with assumptions. She argues that it is through careful observation of the intangible factors around us that we may begin to understand the patterns and relationships in new ways, creating greater insights and understanding.
The potential adoption challenge that Terry Irwin, Head of Carnegie Mellon School of Design, has observed for observation-based methods such as ethnography, is that it is a much slower method of measurement in comparison to what organizations are using today. This could potentially negatively impact interest in adoption as it does not fit into existing paradigms of working. Building on this, using slower methods also means that organizations have less of an opportunity to catch unintended consequences and course correct. Qualitative methods often require a significant time investment to gather enough observation points to begin to create connections in our environment and behavior. As Irwin observed, the technology driven world that we live in today moves much faster than ever before, meaning that there is a risk that qualitative assessments may always be playing catch up rather than leading the discussion.
However, qualitative methods do not necessarily have to be used in isolation to measure intangible factors. In fact, there is probably no single method that could be used to help us understand the complexities of intangible factors. Instead, using multiple qualitative and quantitative methods together, can begin to create a much more accurate, holistic picture of both the tangible and intangible. It is important that intangible factors play a role with the tangible in creating a shared definition of success and common starting point from which to measure change. Working together, the invisible can slowly become visible. We can begin to see patterns in the insights generated and track some of the probable consequences of our designs. While this is nebulous, subjective and difficult to do, we are responsible, as designers, to attempt to complete the picture in the world of information that we live in.
References
Brain, T. (2018). The Environment is not a System. A Peer Review Journal About. Retrieved from http://www.aprja.net/the-environment-is-not-a-system/.
Dahle, C.L. (2018). Designing for Transitions: Addressing the Problem of Global Overfishing. Cuaderno.
Hurley, D. (2018, January 02). Can an Algorithm Tell When Kids Are in Danger? The New York Times Magazine. Retrieved from https://www.nytimes.com/2018/01/02/magazine/can-an-algorithm-tell-when-kids-are-in-danger.html
Irwin, T. (2018). The Emerging Transition Design Approach. Design Research Society.
Onuoha, M. (2016, February 10). The Point of Collection. Medium. Retrieved from https://points.datasociety.net/the-point-of-collection-8ee44ad7c2fa.
