Human-AI Interaction at ACM DIS 2024
by Daniel Buschek (University of Bayreuth, Germany)
The ACM conference on “Designing Interactive System” (DIS) took place at the IT University in Copenhagen from July 1 to 5, 2024. With a theme of “Why Design?”, the program attracted a rich and diverse mix of contributions, including papers and workshops that focused on speculating, building, evaluating and critically questioning the design of interactive technology.
In this article, I highlight just a few of these papers, with a focus on the theme of human-AI interaction. This is not a comprehensive list and is biased by my choice of tracks at the conference; DIS had five parallel sessions this year. I encourage you to check out the full technical program. This overview is structured into three parts:
- Interaction & impact: Papers that advance human-AI interaction concepts and/or assess their impact on people and interaction outcomes.
- Human-AI workflows in research and practice: Papers that engage with the broader workflows in which human-AI interaction is — or needs to be — integrated.
- The future is present: Papers that envision the future of human-AI interaction and/or draw inspiration from past and present sources, including cultural ones.
Interaction & impact
Several papers at the conference addressed interaction with AI and its impact on users and outcomes.
Object-based iterative prompting
Peng et al. presented their work on “DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI.” DesignPrompt is a multimodal tool for design exploration with generative AI. It is based on the concept of a moodboard, but allows users to extract semantic information from images, such as colours and objects, which can then be used to prompt a text-to-image model for additional inspiring images. In this way, images become a way to prompt the model:
Users selectively combine aspects from multiple images via direct manipulation to compose new prompts.
DesignPrompt demonstrates a new way of combining intent-based interaction with direct manipulation interfaces.
Co-creating learning material with AI
Bhat et al. posed the question, “Do LLMs Meet the Needs of Software Tutorial Writers? Opportunities and Design Implications” Their paper highlights another direction for using generative AI in co-creative software development tasks, focused on teaching and learning.
Based on qualitative research with software tutorial writers, the authors identified three key requirements for LLM-based software tutorial writing tools:
- They should set realistic expectations that LLM-generated outputs will need to be verified and edited.
- They should provide clear prompting mechanisms that help writers precisely control tutorial-specific content generation and editing.
- They should make it easy to verify the reliability of generated tutorial content.
Supporting human collaboration
In “Text-to-Image AI as a Catalyst for Semantic Convergence in Creative Collaborations,” Lin et al. make a similar argument as Bhat et al. that AI systems should be used to support and augment human effort, rather than replace it. They examined human-AI collaboration in the context of creative ideation with a text-to-image model and found that the text prompts used to generate images helped teams of design students converge on ideas. In this way, the benefit of generative AI lay in its impact on the brainstorming process:
“Text-to-image AI can be beneficial […] as a catalyst for brainstorming rather than a tool for generating design imagery and presentations.”
Authorship & ownership
Draxler et al. raised important questions about the authorship of AI-written texts in “The AI Ghostwriter Effect: When Users do not Perceive Ownership of AI-Generated Text but Self-Declare as Authors.” Across two controlled experiments, they found that people are less willing to give credit to generative AI ghostwriters compared to human ghostwriters. In addition, as a human author made more edits and modifications to an AI-generated text, their sense of ownership in that text increased. This work raises important design considerations for both AI-infused writing tools and the readership experience: At what level of granularity do readers need to know what was written by a human or AI, and how can AI-infused writing tools capture this meta information?
Liddell et al. also addressed the issue of content ownership in their poster, “ORAgen: Exploring the Design of Attribution through Media Tokenisation.” They propose a framework called ORA in which the ownership of digital assets is established via possession of non-fungible tokens (NFTs), rights are defined by licenses defined and issued by a creator, and attribution is tracked via immutable transaction records contained within a distributed ledger. This decentralised approach may be useful to establish licencing schemes driven by and for creative peers.They demonstrate their concept in a prototype that allows users to “create, tokenise, licence, and remix simple colour collages.”
Finally, in “AI Art for Self-Interest or Common Good? Uncovering Value Tensions in Artists’ Imaginaries of AI Technologies,” Jääskeläinen et al. investigated values that artists in the Global North expressed for the design of creative AI technologies. They observed that artists felt matters of design justice were important — in conversations, artists emphasized the importance of common-good values of “universalism” and “benevolence.” However, when asked to envision and sketch AI tools to support their creative work, artists’ sketches focused on values of self-interest such as “self-direction,” “stimulation,” and “achievement”. The paper concludes that:
“[There is an] emergent need to find approaches for critically examining the design political and value oriented notions of Creative AI.”
Agents representing people
Some uses of AI can also raise questions about authenticity beyond authorship. What happens when an AI agent is used to represent a person? Hwang et al. tackle this question in “In Whose Voice?: Examining AI Agent Representation of People in Social Interaction through Generative Speech.” They report on people’s concerns with the use of AI agents to represent individuals in interpersonal communication using generative speech technologies. They used a mixed-methods approach of interviews with developers and members of the general public, and a co-design workshop. Their results highlight concerns about the use of agent representations:
“Both technologists and potential users worry adopting agent representations might harm the quality, trust, and autonomy of human communication.”
The authors argue that users should be empowered to define “red lines” for being represented by AI agents in social interaction — situations in which this should be avoided. The figure below shows people’s views on when it might (or might not) be appropriate to use an agent representation.
Human-AI workflows in research and practice
Another important topic addressed at DIS focused on the configuration of work when AI was involved: What is the AI’s role within a workflow and how can designers successfully incorporate AI within existing workflows?
Studying interaction with AI beyond the novelty phase
Long et al. reported on the use of AI that is “Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow.” Their three-week study investigated how users became familiar with generative AI tools within the domain of science communication. Given the fact that so many HCI studies on interactive AI systems only focus on usage within a specific moment in time, this study stood out to me because of its longitudinal approach. This approach allowed the authors to examine the perceived utility of an AI system after its novelty wore off with users:
“After familiarization the perceived utility of the system was rated higher than before, indicating that the perceived utility of AI is not just a novelty effect.”
Comparing AI workflow integration, instead of UI features
Shin et al. investigated the use of LLMs in user research in their paper, “Understanding Human-AI Workflows for Generating Personas.” They identified and studied three workflows with varied extents of AI involvement to address the question of understanding which subtasks should be delegated to human experts versus which should be conducted by AI. They found that, for the case of constructing user personas, “experts’ efforts are better spent defining important user groups and personas’ qualities while LLMs summarize grouped user data accordingly.”
Interestingly, this work highlights a different approach to empirical studies of human-AI interaction. Often, studies compare prototypes that have different UI features — for example, many studies have been conducted in which one group of participants works on a task alone, and another works on a task with AI assistance. In this work, participants experienced completely different workflows which varied with respect to when AI was used. The figure below shows how they varied the persona-creation workflow with different forms of AI support.
Hussain et al. also studied different human-AI workflow configurations in “Development and Translation of Human-AI Interaction Models into Working Prototypes for Clinical decision-making.” They used a co-design approach with various relevant stakeholders to translate different clinical decision-making workflow configurations into functional prototypes that could be tested with users. Their paper visualises the different workflow configurations, which involve multiple parties: patients, clinicians, and AI.
Facilitating co-evolution of prompts and UIs
Petridis et al. presented “PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers’ Workflows.” PromptInfuser is a Figma plugin that enables users to create semi-functional mockups that connect UI elements with LLMs to help designers iterate on their UIs and LLM prompts at the same time. The authors conducted a study with 14 designers and found that PromptInfuser helped them more readily identify technical constraints. This work motivates a much closer relationship between front-end and back-end design:
“PromptInfuser encouraged iteration over prompt and UI together, which helped designers identify UI and prompt incompatibilities and reflect upon their total solution.”
Generative AI in practice
Several papers at the conference reported on case studies or investigations of incorporating generative AI into practical contexts.
In “Clay to Play With”: Generative AI Tools in UX and Industrial Design Practice,” Uusitalo et al. investigated the adoption and appropriation of generative AI tools by 10 UX professionals. The authors discussed their findings through a lens of metacognitive skills — designers’ ability to monitor their own design processes critically and constructively, which reminded me of work presented by Tankelevitch et al. at CHI’24.
Related, Takaffoli et al. studied “Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?” They interviewed 24 UX practitioners from different companies and countries and identified a significant lack of policies, practices, and training when it comes to generative AI.
Providing such training might include tools presented by Bhat and Long. They addressed the challenges of learning about AI for non-technical audiences in “Designing Interactive Explainable AI Tools for Algorithmic Literacy and Transparency.” They developed three tools for adult novices to learn about AI: The Art of Edge Detection, Confidence Calibration Explorer, and Toggling Sensitivity in Machine Learning. In a user study, they identified four different kinds of personas of AI learners:
- Tinkerers, who drive their learning through hands-on experimentation
- Ethical Observers who focuses on the societal and ethical implications of AI technology
- Realists that seek to understand practical applications and real-world relevance of AI tools
- Visionaries that are interested in the broader implications and future applications
Finally, Kim et al. addressed the fact that AI systems and workflows often involve multiple stakeholders, such as AI experts, designers, and clients. Their paper, “AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients,” introduced a workbook that serves as a guide for collaboration and project planning across these different roles. The figure below shows a sample exercise from the workbook in which an application builder articulates their intent for building an AI application.
The future is present
A final theme of the conference program focused on envisioning the future of human-computer interaction. These visions drew inspiration from both the past and the present. As the Design Museum of Denmark put it during our visit with the “Human-AI Assemblages” workshop: “The future is present.”
Capturing AI affordances with design fiction
Vaithilingam et al. posed a question about the future of design in “Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation”:
What new value can natural language AI models provide to design processes that was difficult or impossible to achieve with classical methods?
The authors address this question by identifying three unique affordances of LLMs compared to past technologies:
- Dynamic grounding, in which the user grounds communication with the AI in ways that are relevant during an interaction (such as natural language or sketches).
- Constructive negotiation, in which AI nudges the user to consider design aspects that they had not yet anticipated.
- Sustainable motivation, in which AI provides the user with long-term support to complete their projects and accomplish their goals.
Next, the authors provide a design fiction that showcases these affordances and how they manifest within a human-AI interaction through a narrative about developing a video game.
This work highlights a bottom-up, iterative approach to design fiction in which both the narrative itself and the resulting AI affordances it showcases evolved hand-in-hand. These affordances provide useful terminology and concepts to talk about the value of AI models in interactive systems.
Full steam ahead?
Cremaschi, Dorfmann, and De Angeli presented “A Steampunk Critique of Machine Learning Acceleration.” True to the referenced pop-culture, the authors built an unwieldy, newfangled piece of tangible technology by extending a historic typewriter with a screen and an LLM for writing tweets.
This work questions the commonly-assumed focus on efficiency and speed of input of our technological devices. It encourages us to slow down our thinking, especially when interacting with AI.
We might not always want to design for speeding up input with AI, in particular when interaction serves thinking.
Learning from human writing and storytelling
The paper by Feng et al., “Smiles Summon the Warmth of Spring: A Design Framework for Thermal-Affective Interaction based in Chinese Cí Poetry,” brought the conversation on written texts back to human authors, rather than AI. Their work focused on the expressive value of carefully-composed human words. The authors analysed historic poems to build a vocabulary for articulating the nuances of users’ experiences of thermal-affective interaction. That a 1,000 year-old poem inspires HCI research today is a powerful reminder that great writing moves through space and time, and inspiration can be found anywhere.
My own paper, “Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools,” also takes inspiration from literary work. It proposes the concept of “Collage” to analyse and critique the UI design of AI writing tools, and it too motivates the development of future writing tools by drawing inspiration from past writing innovations. I have written more about it in a separate post.
Finally, Halperin et al. refer to cinematic surrealism to study “Artificial Dreams: Surreal Visual Storytelling as Inquiry Into AI ‘Hallucination’.” Based on their analysis of 100 visual stories, they conclude:
“While AI “hallucination” is in many ways a problematic phenomenon and term itself, we can understand and re-describe it in computational visual storytelling as ungroundedness: narrative elements (e.g., characters, events, objects, etc.) that are not grounded in the inputs themselves.”
That’s it for this overview! I’d like to give a huge thank you to the DIS 2024 organisers, student volunteers, ACs, reviewers, attendees, and everyone involved!
