Sepsis Watch in Practice
The labor of disruption and repair in healthcare
By Madeleine Clare Elish and Elizabeth Anne Watkins
Portions of this post are drawn from the Data & Society report by Madeleine Clare Elish and Elizabeth Anne Watkins, Repairing Innovation: A Study of Integrating AI in Clinical Care.
Sitting at her workstation on the 10th floor of Duke University Hospital, Jennifer*, a nurse with expertise in intensive care, provides timely and effective treatment for patients at risk of developing sepsis in the Emergency Department (ED). One of her shift responsibilities is to monitor an iPad application called Sepsis Watch that uses a type of AI known as deep learning to display a patient’s risk for developing sepsis. If the risk reaches a certain threshold, Jennifer calls the ED physician, who uses this information to determine a diagnosis. For positive diagnoses, Jennifer monitors the patient and course of treatment through the app.
However, Jennifer’s work is not that simple. When the Sepsis Watch app was first integrated into the hospital, it disrupted existing systems — such as power dynamics and information flows — that needed to be repaired. Jennifer and the other Rapid Response Team (RRT) nurses worked to mend these breakages — not physical breakages, but ones of social norms, expectations, and contexts. They applied their knowledge of the team’s schedules, inquired into the well-being of the physicians to understand their moods, and leveraged their clinical expertise (among other activities) to effectively integrate Sepsis Watch into the hospital’s clinical system. This work performed by the RRT nurses, which we refer to as “repair work,” was crucial to integrating Sepsis Watch, demonstrating how essential it is to recognize and value repair work in the integration of future AI systems.
…Sepsis Watch is constituted by a complex combination of human labor and expertise, as well as technical and institutional infrastructures…
Sepsis Watch incorporates deep learning to improve and support the diagnosis and care of sepsis, a widespread and deadly complication from infection that is the leading cause of death in hospitals in the U.S. and around the world. The app helps physicians in the Duke University ED rapidly diagnose the treatable condition. Today, Sepsis Watch is working smoothly and, anecdotally, has been felt to be improving patient care for sepsis. A clinical trial will eventually report results. But as a successful application of AI that is actually working in the world (most AI-driven interventions remain in the research phase), the deep learning model tends to eclipse the other parts of the system; for instance, in news headlines about the project that announce, “Hospital to roll out AI system for sepsis.”
In reality, AI is not the only system at play in Sepsis Watch. Research we conducted, in collaboration with the Duke Institute for Health Innovation, demonstrates that Sepsis Watch is constituted by a complex combination of human labor and expertise, as well as technical and institutional infrastructures, with nurses playing an integral role in allowing the system to work as it was intended.
Disruption and Repair
If the introduction of new technologies, such as AI, are beneficial because they are disruptive — in that they create new pathways to achieve a goal — this disruption also causes forms of breakage, upsetting existing power hierarchies or rerouting information flows that must be repaired in order for the intervention to work effectively in a particular context. Repair work is not about recovering a status quo, but rather about creating a new set of practices and possibilities. This form of repair work, while necessary, is consistently undervalued and often rendered invisible.
Repair work is not about recovering a status quo, but rather about creating a new set of practices and possibilities.
For example, Sepsis Watch was developed to refocus how and when septic patients are cared for; it purposely did not follow an old way of doing things — it disrupted it in order to create the conditions for better care. At the same time, Sepsis Watch disrupted existing workflows and social relationships both within and beyond the context of sepsis care. These disruptions created gaps, breakdowns, and miscommunications that needed to be addressed in order for the intervention to work effectively. While everyone involved with developing or integrating Sepsis Watch into effective clinical care carried out essential repair work, the specialized RRT nurses who monitored the Sepsis Watch app developed key forms of repair work that allowed the system to succeed.
What did this look like in practice?
In order for Sepsis Watch to be effective, the app’s outputs and the diagnostic practices of the ED had to be woven together. This work of stitching together fell on the RRT nurses. For instance, the nurses performed repair work by carefully timing their calls to work with, rather than disrupt, the schedules and “rhythmscapes” of the ED. Nurses also began contextualizing the Sepsis Watch scores with other patient health data, as a means to more effectively engage in a diagnostic back and forth with the physicians. This kind of relationship between physician and nurse was unusual, and in many ways challenged the existing power hierarchies within the hospital.
In addition to managing call timing and supplementary information, the nurses developed techniques to mitigate the annoyance that many doctors felt when first receiving the Sepsis Watch call, such as constantly negotiating their tone and approach based on the affective reactions of the doctors. These techniques involved a great deal of what sociologist Arlie Hochschild calls, “emotional labor.” “Emotional labor” refers to the planning, management, and display of feelings and emotional expressions at work to facilitate organizational goals and norms, often supplementing other forms of physical and cognitive labor. While emotional labor is a critical component of many jobs, it is stereotypically gendered as female work, and is associated with lower wages. In order to integrate Sepsis Watch into the clinical workflow of ED clinicians, RRT nurses not only managed their own displays of emotion, but also found ways to inspire feelings in others. One nurse explained how she had developed a way to effectively probe what she might need to do in the conversation with an ED doctor:
In a typical interaction I always start the same way. ‘Hi, this is [person’s name]. I’m the RRT nurse from Sepsis Watch. How are you?’ I ask how are you so that I can immediately get a feel of whether or not they’re busy and want to talk. … And even though I know they don’t want to waste time with ‘how are you,’ it gives me an indication of how this call’s gonna go.
Throughout the six month pilot, RRT nurses carried out various forms of repair work that were both unanticipated and essential, drawing on their existing knowledge of clinical practice and trying to produce ad hoc solutions. “Workarounds” following implementation of new processes or technologies are a common and widely-studied phenomenon in healthcare. However, as scholars Kathleen Pine and Melissa Mazmanian argue, it is important to understand such practices not merely as temporary or casual forms of work but instead as skilled and essential forms of coordination.
Indeed, the repair work performed by the RRTs revealed their implicit and explicit forms of expertise. The RRT nurses applied their knowledge of organizational rhythms, performed emotional labor in order to facilitate interactions with harried and skeptical doctors, and leveraged their clinical expertise by engaging in diagnostic conversations to effectively integrate Sepsis Watch. The RRT nurses also became experts in the care of sepsis, itself.
Valuing Repair Work
To be clear, not every RRT nurse found the new role and responsibilities enjoyable or satisfying. Some we spoke with felt that it took away from their primary duties of bedside care. In fact, all the nurses we spoke with were very careful to differentiate what they were doing with Sepsis Watch as being distinct from providing clinical care, which they understood to necessarily involve physically seeing patients. Nonetheless, many embraced the new work, finding it an interesting challenge.
Focusing on the role of repair expands our understanding of what constitutes innovation — who does it, what it looks like, and where it happens.
It’s important to note that much of the repair work we describe was possible because the autonomy of the RRT nurses was respected. The RRTs were allowed professional discretion and, in turn, the flexibility to improvise and create the conditions and tactics of effectively communicating the risk scores produced by Sepsis Watch. Moreover, their perspectives and expertise were incorporated into the design of the system both before and after the launch. The repair work performed by the RRT nurses could be effective and for many, empowering because their professional discretion and expertise was supported — not undermined—by the team developing Sepsis Watch.
Repair work is essential and valuing it helps us reframe what’s at stake in innovation. Focusing on the role of repair expands our understanding of what constitutes innovation — who does it, what it looks like, and where it happens. The work of repair also requires creativity, skill, and ingenuity — and it should be valued as such. If only the work of initiation and theoretical construction, typically elite and masculine forms of work, are valued when it comes to the future of AI and society, then much of the day-to-day work that is required for AI to function in the world is rendered invisible and undervalued, further contributing to conditions of social inequality. Recognizing repair work shifts our focus from those who initiate a project to those whose work and skill are required to make the project work out in the world.
*All names are pseudonyms to ensure the anonymity of research participants.
Madeleine Clare Elish directs the AI on the Ground initiative at Data & Society.
Elizabeth Anne Watkins is a research analyst on the AI on the Ground initiative.
