Civic competitions aimed at building knowledge around a problem
This post is a synopsis of an article about how participants in a civic design competition talk about civic design problems. The paper was recently presented at the Computer Supported Cooperative Work (CSCW) Conference. Here’s the full reference and a link to read more about our work:
McInnis, Brian, Xiaotong (Tone) Xu, and Steven P. Dow. “How Features of a Civic Design Competition Influences the Collective Understanding of a Problem.” Proceedings of the ACM on Human-Computer Interaction 2. CSCW (2018): 120.
How do cities discover and address their most pressing challenges? For example, think about the variety of challenges with a city’s transportation system. How do we assure public transit is accessible? How can we improve walking and biking safety? As transportation systems evolve, they can have an impact on how people work, where they work, or whether they can work.
In a recent paper on Civic Design Competitions presented at the 2018 Computer Supported Cooperative Work (CSCW) conference, we argue that the process of addressing challenges, like transportation, typically only involves experts and decision makers and a relatively small fraction of the public. Civic engagement technology and participatory design practices aim to improve civics, but we have much to learn about how such processes affect the collective understanding of complex challenges.
A form of public engagement, which we motivate through the paper is civic design competition. Civic design competitions offer one promising approach to engage a wide-range of residents and for structuring activities that attempt to generate insight about a civic problem, such as transportation.
Within the domain of transportation, almost everyone has personal experience with transportation issues and insights into potential solutions, regardless of background. In this paper, we focus less on specific solutions that emerge from a civic design challenge and more at how online conversations surrounding a competition reflect the breadth and depth of knowledge generated about a civic problem. We were particularly interested in the factors affecting the breadth (or range of topics) and the depth of discussion (called “topic coherence”) into the transportation problem-space.
To study this, we could have partnered with organizers who were running a civic design competition, but instead we conducted a form of action research, where we actually designed our own competition and set up the infrastructure necessary to study it.
We created Design for San Diego (D4SD). We structured the timeline of activities to include both online and in-person events over the course of a 62-day period, so that participants could meet and share ideas with each other, before settling into a design rhythm with their own teams.
The competition officially launched on August 25th, with several targeted email blasts and newspaper articles about the competition. A few weeks after our initial soft-launch, we organized a large event at a downtown makerspace with the Mayor of San Diego. Following this big event dozens of people participated in a two-day hackathon (13 teams). During the final thirty days of the competition, teams could participate in a weekly design studio that offered training to people less familiar with human-centered design.
People formed teams and developed their ideas into prototype solutions. The civic design challenge culminated at a summit with professional designers, where the top 8 of 23 teams pitched their ideas. The team that won, was a group of high school students focused on equipping bicycles with sensors to increase their visibility to vehicle systems. Their winning concept won $5,000 in cash, with support and mentor-ship from the San Diego startup community. But, our research isn’t about the transportation solutions, rather how the community came to understand the problem-space through discussion.
We chose to use Slack as the official online discussion platform for the competition. We created three types of channels. Four public channels for general discussion about the mobility challenge (i.e., accessibility, autonomous vehicles, commuter experience, walking and biking safety). Channels so that participants could learn about the process (e.g., news, events, and a channel to help them form teams), and we encouraged teams to create Private channels within Slack. This analysis focuses on 476 messages posted by 92 participants to these four public channels.
We considered the possible influence of three key factors on the depth and breadth of discussion: The competition structure, discussion system, and participant interactions. Competition can help to sustain a regular rhythm of activity among an online community, though too much can be detrimental to collaboration. Online discussion organizers often plan a discussion so major topics are separated into specific channels or time periods. Lastly, the content of a message influences whether other members are likely to engage with the message. For example, posting on-topic messages, introducing oneself to others’, asking questions, and being succinct, all tend to elicit a reply.
We developed a coding scheme to capture how broadly participants covered the different design topics and how deeply they negotiated design constraints, questions, and proposals. The process involved an open coding of the design topics, an analysis of how participants talk about the design problem, and how participants agree and disagree, as well as how they greet and appreciate each other (with social talk). To examine how each type of message content correspond to discussion behaviors, we fit a series of mixed effects logistic regressions and supported our statistical analysis with participant interviews, follow up surveys, and qualitative analysis of the content.
The figure above captures the volume of discussion by day of the competition, with key events marked along the x-axis, and color coding to show the volume of messages to the four general channels, the process channels, and to the private team channels. The online discussions went live about 30 days before the kick-off with the mayor, but the discussions didn’t pick up until September 21st. A typical participant joined and then posted their first message within a few days and not long after was on a team. This made for a brief window of opportunity for collaborative discussion before setting into their team work. Discussion activity decreased following the team registration in the general channels, but simultaneously picked up in the private team channels. In their post-survey comments, participants expressed that they felt the pressure to compete made some teams less willing to share.
As I mentioned, we focused the general discussion around four transportation topics and structured them in separate public channels. Some channels generated more messages than others, similarly some structured discussions were more topically focused than others. We found that 359 (75%) of the messages to the general channels added insight about the transportation design topic, and 149 (31%) added directly to the structured design topics. For example, 80 of 84 the messages posted to the autonomous vehicles channel were about autonomous vehicles, but only 10 of 48 messages posted to the accessibility channel were about accessibility.
In fact, many of the messages added new transportation design topics. By analyzing these design topics that emerged through participant interaction, we found that several design topics, such as driving culture and ride sharing, appeared in multiple of the public channels, yet these parallel discussions on similar topics, were never connected. For example, a common problem is how pedestrians and drivers communicate with each other. In the accessibility channel this problem sparked ideas about how the steering wheel or stereo might call a drivers attention to a pedestrian, in the walking & biking channel there were ideas about insurance incentives for courteous driving, and in the autonomous vehicles channel the discussion was about how a car might signal to a pedestrian that they have been seen or that it is safe to cross.
“Slack was effective in showing the problem space in each topic area and helping guide further research into problems to be solved. However, due to the competitive nature of the challenge, people stopped sharing.”
In the post competition survey, participants conveyed mixed feelings about the channel structure, some described the Slack channels as a shared space to collect relevant information, such as news clippings. Others felt that the discussions were too free-flow to follow.
To investigate the role played by participant interaction, we fit a set of mixed-effects logistic regressions to model specific online discussion behaviors, such as initiating a reply-thread from a main channel post and replying with a topically coherent message.
The figure above presents an example of a topically coherent response, where an initial main channel message about commuting with a bike elicits several responses to a reply-thread that offer design constraints based on personal experiences (“the bike racks on the bus are sometimes full”), some intuition into the problem (“San Diego is built for cars”), and a short synopsis of the key rules for commuting with a bike.
We found that messages posted during a period of high message activity, were more likely to elicit a response. Similarly, messages that added a “design constraint” or included a link were also more likely to elicit a response than those that did not. In short, participants responded to substance.
By comparing the content of messages posted to the main channel or to a reply-thread, we also found that messages within a reply-thread were more social, but to our surprise, were less likely to raise questions about the design topics. Echoing our other findings, messages that added substance (i.e., design constraints, proposals), were more likely to elicit a coherent response. However, this was less likely for main channel messages that add social-talk.
To address the drop in general channel activity following the team registration, researchers should look for ways to curb the cost (or boost the benefit) to sharing, perhaps by introducing different team-to-team and individual incentives to entice participants to continue interacting. To connect similar conversations in parallel channels, we echo other calls to investigate ways to visualize the topics under discussion or use crowdsourcing methods to synthesize the information about key topics into short summaries.
Our work connects to a long history of CSCW research about how message content can influence whether and how others’ respond. In the future, research could try automated methods to encourage more design questioning in real-time and within a reply-thread. By leading and studying D4SD, we learned about how the competition timeline, discussion system, and participant interactions influence the breadth and depth of discussion about a problem-space. In the future, researchers could pursue questions about how to access and update the breadth and depth of knowledge captured by such digital structures. During the competition, teams also coordinated outside of Slack, both online and in-person at formal events and team meetings. Future research could examine, not just how to generate information about a problem-space, but also how to create the connective tissue between the different agents who are exploring the problem. How might we uncover the thinking behind the design that each team offers?
Steven Dow and Tone Xu at UC San Diego co-authored the paper. We owe a deep debt of gratitude to our collaborators at the UC San Diego Design Lab, ScaleSD and the Design Forward Alliance, as well as to our regional partners at SANDAG, the Port of San Diego, and at the City of San Diego. Our research was supported by the National Science Foundation.
To read more about how I develop and evaluate public engagement technology, visit https://bmcinnis.github.io/ or follow me on Twitter.
