Designing visual features for appropriate reliance in human-machine interactions

From the MA thesis of Giulia Zerbini

The aim of the project is to investigate the concept of reliance applied to human-machine interactions, and to propose a principled method for measuring as well as optimizing reliance through a combination of machine learning and visualization design. The idea is to create a closed-loop process of data collection (applying principles of experimental design and game-theory), analysis and modeling (machine learning) and interactive data visualizations (design). The objective of the research is to use this process to create visualizations that optimize the level of reliance of the users in human-algorithm pairings by allowing the users to understand the algorithm through investigation. The goal of this research is to apply reliance optimized visualization designs in real use-cases.

The main challenge of this research is to combine design and machine learning and work fruitfully in the intersection between them. Giulia Zerbini’s experience of working as a designer with the Institute of Applied Computer Science at Harvard has been deeply challenging and motivating. The intersection between disciplines is represented in both the research interests and in the backgrounds of the team members. The working team was composed by researchers from different background. In particular, it consisted of Giulia Zerbini, a designer from Politecnico di Milano, and Weiwei Pan, the machine learning expert, who is a postdoctoral fellow at Harvard University. Also part of the team are Pavlos Protopapas from the Institute for Applied Computational Science at Harvard University and Paolo Ciuccarelli from Politecnico di Milano. The research was conducted at Harvard for six months, from September 2017 to March 2018. During the first phase of the project Giulia Zerbini, as the designer, worked to gain basic knowledge about data science and machine learning, from both the theoretical and practical point of view. For this purpose, she followed lectures on Data Science and gained basic fluency in coding languages like Python. This initial step was necessary in order to discuss the research topics and understand how to integrate design in the machine learning processes. The work has been driven by both the design and technical perspectives, with multiple iterations through the process of literature search, research hypothesis formation and hypothesis evaluation, until we agreed on a satisfactory formulation of a specific problem statement. The research, as well as the writing, has been structured in four sections, corresponding to different steps of the project development:

- Research
- Experimental design
- Modeling reliance
- Tool prototyping

All the phases are explained in detail in the full text document. We note that the phases should not be taken to be in perfect chronological succession, but as a structured way to understand the research process.

Link to the full text document: https://www.politesi.polimi.it/handle/10589/140859

Credits:
● Giulia Zerbini, Creative Technologist at The Visual Agency / MA graduate at Politecnico di Milano in communication design and previously visiting researcher in data visualization at Harvard University
● Weiwei Pan, IACS Postdoctoral Researcher
● Principle investigator: Pavlos Protopapas, IACS Scientific Program Director and Lecturer
● Thesis advisor: Paolo Ciuccarelli, Associate Professor at Politecnico di Milano

● Communication Design Master of Science, Politecnico di Milano
● Institute for Applied Computational Science, Harvard University

Visit us: https://thevisualagency.com/