Identifying Challenges and Opportunities in Human-AI Collaboration in Healthcare:
CSCW 2019 Workshop Highlights
This blog post is written by Sun Young Park, Lauren Wilcox, Junhan Kim, Pei-Yi (Patricia) Kuo, Andrea Barbarin, and Astrid Chow. The Human-AI Collaboration in Healthcare Workshop was held at the 22nd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) in Austin, TX, in November, 2019.
As research in artificial intelligence continues to pursue applications to health and well-being, the urgency to identify both its positive and negative effects increases. Emerging solutions promise to ease the burden of care delivery by assisting clinicians with diagnostic tasks, automatically generating summaries of patient data for clinicians, and tailoring interventions for high-risk patients to improve health outcomes and minimize costs. While the future sounds promising, the “side effects” effects of AI — whether amplifying human biases and inequality or the potential pitfalls of over-automation — cannot be ignored since they have a direct influence on people’s health and wellbeing.
We brought 33 researchers together from both academia and industry for the CSCW 2019 workshop on Human–AI Collaboration in Healthcare. Participants came from diverse backgrounds (physicians, computer scientists, social scientists, and designers) and from different countries (India, China, Denmark, UK, and the US). We discussed the opportunities and technical, social, and ethical challenges related to incorporating AI into healthcare.
The Urgency for Research in Human–AI Collaboration in Healthcare
Recent advancements in computational techniques, networking, and computing power result in partial or full automation of tasks that humans historically have conducted manually or couldn’t perform. In healthcare settings, AI and algorithmic decision-making systems have been developed to conduct a variety of tasks, from supporting clinical processes to transforming access to medical benefits. However, these rapid developments lead to complex sociotechnical challenges to the medical domain and its stakeholders; e.g., the potential for algorithms that are biased against underserved populations (“New York Regulator Probes United Health Algorithm”), lack of thorough consideration of data used when developing algorithms — and how it is collected (Mind The Gap: AI & Healthcare), the future role of clinicians and caregivers in medicine (“Skin Comes in Many Shades and So Do Rashes”), as well as patient–clinician communication, are all sociotechnical challenges that need to be addressed as this technology develops. The workshop gathered scholars and practitioners across different fields to discuss these challenges in the age of AI.
We set the following two overarching themes for the workshop: 1) new roles/labor relations created by AI, and 2) trust in AI. By the end of the workshop, there was discussion on a variety of topics related to those two themes. Among these topics, trust was the top priority, and many participants raised issues such as shifted workflows, bias, privacy, and ethics as well.
The workshop comprised panels with group discussions in the morning and hands-on activities in the afternoon. In the morning, participants were divided into four panel groups on four different topics based on the position papers they submitted — 1) AI systems in clinical settings, 2) trends in patient-focused health applications, 3) ethnographic and organizational studies, and 4) understanding bias and unintended consequences of AI. Each panel consisted of four to five panelists working on similar topics or application areas, and was moderated by one to two workshop organizers. Each panelist presented their own works related to AI, and shared lessons learned and challenges in designing healthcare technology in the age of AI and automation, followed by a Q&A session open to all workshop participants led by the panel moderators. The second part of the workshop sought to create ecosystem maps for different scenario settings related to Healthcare and AI. Participants were organized into different groups focusing on 4 different healthcare scenarios. Using the context from the healthcare scenarios, each group worked together to “map” the space of human-AI collaboration to identify stakeholders, research gaps, boundaries, tensions, and ethical issues by considering sociotechnical aspects of various health system use. Below, we summarize key highlights from both parts of the workshop.
Trust
Trust is a huge factor in successful adoption of any new technology, let alone the healthcare industry’s adoption of AI technology into care teams’ workflow as it is central to the amount of reliance that patients and physicians have in the system. Workshop participants focused on the key stakeholders surrounding AI and trust. They identified different perspectives based on different individuals and their roles, such as primary end users, secondary users (e.g., caregivers), peripheral users (neighbors, etc.), or healthcare providers.
Trust is very nuanced and contingent on individuals who interact with the system and the environmental context. For example, the appropriate amount of trust may differ between patients and healthcare professionals. Healthcare professionals may require a larger amount of trust considering the level of their expertise while patients could have a lower threshold for trusting and using AI-based applications. Many participants were concerned with the detrimental effect of over-trust in AI-based health systems. Although trust is important for adherence, willingness to use, and positive outcomes or benefits, over-trust in the system can have deleterious results (e.g., increased risk for patients). Additionally, participants discussed the interplay of different existing layers of trust (e.g., technology, clinician-patient relationship, and healthcare data) within the healthcare environment. For instance, trust toward data could be influenced by algorithm transparency and interpretability. Familiarity with technology could also influence people’s perceptions of trust. Therefore, building appropriate amounts of trust is a design direction to be further studied in the field of HCI.
Workflow
How AI could be better incorporated into existing workflows was another crucial discussion topic. AI should not be forcibly integrated into existing healthcare work environments, but rather be carefully evaluated based on its potential benefits and intended consequences for existing tools, artifacts, and work practices; we should not simply incorporate or adopt AI technology without taking into account existing systems and work practices, and its accompanying tensions with the existing workflows. This will mean that prospective studies of AI systems in the clinic will need to focus on issues of human acceptance and workflow in addition to model evaluation alone. Moreover, concerning how to better integrate AI technology into existing workflows, participants pointed out the importance of interdisciplinary collaboration, particularly between health providers and AI experts. Successful system implementation requires both clinical knowledge and machine learning expertise to ensure the accuracy of data trained and delivered by the AI system. This interdisciplinary collaboration effort may require some role shifts in clinical settings.
Ethical issues
As expected, participants raised issues related to ethics. The digital divide was mentioned as a relevant ethical issue. AI-powered healthcare systems may deliver different results and predictions depending on users’ status, such as living location or socioeconomic status. Participants discussed the importance of being aware of unintended or unexpected consequences such as bias in data collection, algorithm design, and outcomes. Additionally, participants posed questions about what we should or can do to address concerns about inherited bias. Furthermore, the issue of privacy was raised in terms of how it is important to obtain informed consent from users if their data are collected and used to train the data sets of such AI-based systems. Further research into how “informed consent” is defined in the health and medical industry is needed as we increasingly adopt more AI technology.
Ecosystem Maps
In the second part of the workshop, we led hands-on activities in small groups to create ecosystem maps. Considering health systems as a complex sociotechnical challenge, this activity drew on interdisciplinary knowledge, inspiration, and insights across fields to “map” the issues and challenges of human–AI collaboration. This activity helped us visualize and articulate research gaps, boundaries, activities, sociotechnical relations, and ethical concerns involved in human–AI collaboration, focusing on trust and labor in particular.
The workshop participants were divided into four groups, and each group worked on a different human-AI collaboration health scenario given by the organizers. The given scenarios included the four following contexts: clinical consultation in a hospital setting, elderly care in a community health system, children with mental health issues in a pre-school, and healthcare resource distribution for refugee communities. After completing the ecosystem mapping activity, we had each group present their maps and ended the session with a large group discussion and reflection.
Group 1: AI system in a hospital setting
The hospital group considered using conversational AI to focus on improving repeated tasks in hospitals (e.g., triage process). The main goal of the AI was to save time spent on the routine work of information gathering and delivering, and to summarize the data for healthcare providers. This was ultimately intended to improve the quality of interaction between patients and their care providers and increase the efficiency of care.
As also discussed in the panel session, this group described the type of Human-AI interaction that would affect trust and labor. First, the comfort of patients’ data sharing needs to be closely examined. Also important is ensuring the reliability of a patient’s statements as this may directly relate to the patient’s own experience. Finally, ensuring a clear explicit feedback loop is necessary to continuously modify, train and improve the conversational system. Through explanation of the data and feedback, the AI will be able to learn and develop. Regarding the reason for choosing a conversational form of AI (e.g., “chatbot”), the group mentioned that they wanted something that could summarize information for the clinician while also including validation with patients to help close the feedback loop. Yet, questions abound about cases for which a conversational approach is inappropriate (e.g., the modality is not reasonable for the particular situation, the AI is not suitably trained to handle urgent situations verbalized in different ways, or a human is needed to interpret, explain and contextualize in ways that will always elude a conversational system).
Group 2: AI system for community elderly residents
Group 2 started by defining their goals in creating an AI system for community health. Their AI system aim was to provide coaching for elderly residents who are feeling lonely. The group tried to consider what elements make up a “good” companion and how the use of AI technology can help make positive changes in the elderly community. The group came up with two ideas to design the AI algorithm. An AI script could be developed to gather tips from community members, or it could automatically generate the tips.
the core values that must be considered in the development of the AI technology should emphasize trust and empowerment, empowering elders to connect with each other and clinicians more seamlessly through technology
The group also provided and shared thoughts regarding different aspects and roles of AI, including the data source for the AI algorithm, the set of core values that must be considered in the development of the AI technology, and how this AI solution could help elderly community members connect with each other. First, the group mentioned that the data needed to be derived from a diverse set of stakeholders such as family members, caregivers, friends, neighbors, nurses, doctors, and community health workers. Second, the core values that must be considered in the development of the AI technology should emphasize trust and empowerment, empowering elders to connect with each other and clinicians more seamlessly through technology, while considering the social aspect of the whole system. Finally, the AI solution should connect community members by acting as a companion, creating social bonds among elders, facilitating different degrees of communication among members based on the multi-layered personal data gathered.
Group 3: AI system for children with mental health issues
In mapping an AI-enabled ecosystem that helps identify and provide resources for children with mental health issues in primary schools, Group 3 participants mentioned the three following main stakeholders for their envisioned AI system: children, parents, and teachers. They discussed what data could be produced from each stakeholder, what actions they can take, and how the system can be incorporated into their daily lives. For instance, the child can provide information about their daily emotions and an account of their own experience and behaviors. The integration of the AI system into their daily life needs to be ongoing since stakeholders, teachers in particular, should interact with the system continuously to identify and be aware of children’s emotional and behavioral changes. In addition, the parents should be able to input and share relevant information about the child’s behavior at home. If the parents see a certain change in their child’s behavior, they can record or flag it in the system. Finally, the role of the teacher, as a crucial stakeholder, was highlighted for their ability contribute to and draw connections from different sets of relevant data that would be needed for the AI system. For instance, one of the teacher’s core responsibilities would be to consolidate different types of academic data such as test scores, attendance status, and classroom behavior. Teachers should not be limited to consolidating academic, individual and parent data, to inform a report at the school level, but they should also be able to highlight changes in the data, and flag these changes as potential signals in the AI system. The AI system could then have some potential to guide stakeholders in determining the future mental status of a child and suggesting appropriate resources for the child.
When these types of systems are being scoped and built for the child and family members, it is crucial to consider whether such system might produce or amplify any existing bias.
During the large group discussion, there were significant privacy concerns with such an AI system; participants recommended that infrastructure and data protection be considered when collecting sensitive and personally identifiable information from users. Despite of the benefits of helping to identify students with mental health needs and suggesting resources more efficiently, such a system could create privacy concerns for the child and family members and could potentially produce bias against those children. This could contribute to potential prejudice based on the child’s mental health status. For example, when a child moves to a new community and a new school, there could be the potential that the child’s mental health status reaches their teachers even before the teacher meets the child. When these types of systems are being scoped and built, it is crucial to consider whether such system might amplify any existing bias.
Group 4: AI system for refugee communities
Group 4 participants began their discussion by focusing on the current challenges that refugees face in the United States. Initially the group gravitated towards an AI decision-based conversational system (e.g., “chatbot”). Similar to the Group 3, this group also identified various data types to be collected for the AI system (e.g., age, gender, population size, location, immigration status, language, and country of origin). The group then identified relevant stakeholders including governmental agencies, NGOs, community organizers, religious communities, healthcare providers, and teachers.
Literacies regarding technology use, healthcare knowledge, law and policy guidelines were considered to be factors significantly influencing the success of such an AI system, as well as literacy related to data elicitation and how to make judgments about what data to provide as a refuge.
Participants discussed issues related to refugees including social measures such as political stability and safety (e.g., vaccines and insurance). Participants also noted the importance of transparency when providing information resources to refugees, transparency about the type of information resources, as well as how the information would be delivered to ensure a transparent AI system. Interestingly, Group 4 saw literacy as crucial. Literacies regarding technology use, healthcare knowledge, law and policy guidelines were considered to be significant factors influencing the success of such an AI system, as well as literacy related to data elicitation and how to make judgments about what data to provide as a refuge. How an AI system can gauge and support such literacy is an open challenge, and research that makes progress toward overcoming this challenge should coincide with other research and development efforts.
Photos from our workshop group dinner at a great Mexican restaurant!
Citation: Sun Young Park, Pei-Yi Kuo, Andrea Barbarin, Elizabeth Kaziunas, Astrid Chow, Karandeep Singh, Lauren Wilcox, and Walter S. Lasecki. 2019. Identifying Challenges and Opportunities in Human-AI Collaboration in Healthcare. In Conference Companion PublicatioS. of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW ’19). Association for Computing Machinery, New York, NY, USA, 506–510. DOI:https://doi.org/10.1145/3311957.3359433
