CrowdIA: Solving Mysteries with Crowdsourced Sensemaking
This blog post is a summary of our CSCW 2018 paper “CrowdIA: Solving Mysteries with Crowdsourced Sensemaking” by Tianyi Li, Dr. Kurt Luther, and Dr. Chris North.
Events such as the September 11th, 2001 terrorist attacks and the miscall on weapons of mass destruction in Iraq — “the two major U.S. intelligence failures of this century” — illustrate the difficulties that even experienced professionals face in analyzing this data and the high-stakes consequences of failure [Clark].
Making Sense of Large Text Data
Assuming we have a mystery:
Mr. Boddy was found dead in the kitchen. His throat had been cut with a knife. Who killed Mr. Boddy?
The police have collected a pile of documents, but they don’t have enough experts to solve the mystery. Can we use a group of non-experts to conduct such complicated analysis?
Intelligence analysis to identify crimes suspects or develop hypotheses has always been experts’ job, who manage the entire process and datasets [Pirolli et al.]. However, given the magnitude of data we are dealing with in today’s world, the analytics demand far exceeds the cognitive capacity of individuals or small groups.
Fortunately, new technologies such as artificial intelligence and crowdsourcing are here to help. Artificial intelligence has played an important role to help extract and manage knowledge from raw data. But making sense of the information to develop hypotheses and make actionable decisions still requires human intelligence. We need larger scale collaboration in sensemaking, but how?
Crowdsourcing is a powerful new paradigm that augments individual human intelligence at a large scale [André et al., Chilton et al.], showing potential to bridge the gaps between the information overload and the limited cognitive capacity of individual experts. However, a larger pool of intelligence does not necessarily lead to a wiser crowd, as shown by failed crowd investigations of the Boston Marathon Bombing and Unite the Right rally.
CrowdIA: the Pipeline and Context Slices
Our paper proposes a sensemaking pipeline and the concept of context slices to modularize both the process and the data in complex sensemaking. The pipeline is developed based on the expert sensemaking loop and transform the information from raw external documents to a final conclusion with five major steps. Each step specializes a clear goal of analysis, passing the outcome to the next step. At each step of the analysis, each individual will analyze a small slice of the input information that is contextually cohesive, thus the name “context slices”.
Example of Crowdsourced Mystery Solving with CrowdIA Pipeline
Step 1: Search and Filter Relevant Documents
We first hire crowd workers to evaluate which documents are relevant to solving the mysteries. For each document, we hired three people to give a rating from 0 (completely irrelevant) to 100 (completely relevant). By taking a threshold 50, we take the majority vote and pass documents that received more than two rating higher than 50 to the next step.
Below is an example task interface of step 1 crowd workers. This crowd worker will read the document, and associate the knife owned by Prof. Plum to the murder context, then give a rating of document relevance.
Step 2: Read and Extract Important Information Pieces
Now we can focus our crowd intelligence on the most relevant documents, and further extract the most important information from each document. For each document, we sequentially hire crowd workers to extract and review important information. After the first worker extracts information pieces, a second worker will be invited to review and refine incrementally. This way we guarantee the quality and avoid redundancy in the information pieces.
In the example interface below, the crowd worker extracts the important information that Prof. Plum’s car was seen outside of Mr. Boddy’s house the night of his death.
Step 3: Schematize Information Pieces with Tags
After Step 2 is finished, we will have a list of information pieces. How do we organize them into meaningful schemas that lead to more in-depth insights? In this murder case example, we tag the information pieces by the suspects’ means, motive and opportunity to kill Mr. Boddy. By inviting three people to tag each information piece and taking the majority vote, we can assign tags to each information piece.
In the example below, the information piece is about Prof. Plum being seen at the murder scene. Thus the crowd worker tags it as the opportunity of Prof. Plum to kill Mr. Boddy.
Step 4: Hypothesize the Likelihood of Each Suspect
With these tags, we can organize the information pieces into profiles of each suspect, including their means, motive, and opportunity to kill Mr. Boddy. We implemented a simple example that compares suspects by single elimination tournaments. We rank the suspects by the completeness of profile and the number of information pieces, then compare two profiles at a time until we reach a final winner. Each pair of profiles is compared by three people, and we take the majority vote to decide which one is more likely to be the murderer, or if the two profiles are actually aliases.
In the example below, the crowd worker is assigned to compare Scarlett and Plum. Although both of them have all three tags (means, motive, opportunity), the motive of Prof. Plum is not as strong as Scarlett, since he doesn’t know about the affair of his wife and Mr. Boddy. Thus s/he pick Scarlett as the more likely suspect.
Step 5: Tell a Story to Connect the Final Answer to the Original Problem
We consider the final winner as the most likely murderer. The final step is to generate a presentation that explains and justifies why and how this final answer connects to the original problem.
In the example below, the crowd worker put the means, motive and opportunity together and explains how each evidence of Scarlett fits the murder case.
Evaluation: Solving Three Mysteries
We implemented CrowdIA, a software platform to enable unsupervised crowd sensemaking using our pipeline. We evaluate our pipeline with easy, moderate and difficult datasets. We consider datasets with more documents, more elements (who, what, where, when) and more complicated relationships among elements to be more difficult. With CrowdIA, crowds on Amazon Mechanical Turk successfully solved two mysteries and were one step away from solving a third.
The example above is our second dataset. The easy dataset contains three documents about three girls who might have ruined Mr. Potter’s flowerbed. The difficult dataset is part of the Sign of Crescent dataset used as training material for professional intelligence analysts. There are 13 documents, four terrorists, and 12 locations mentioned in the documents.
The crowd workers successfully solved two mysteries and were one step away from the actual target location New York Stock Exchange in the third mystery. However, they found the weapon storage location North Bergen, New Jersey and ranked New York Stock Exchange as the second possible target. Below, we examine the crowds’ performance in detail.
Overall, the crowd’s submitted work revealed their reasoning process and provided evidence that justifies their conclusions. The modularization of the process and the data effectively support large-scale collaboration among novice, transient crowd workers and produce high-quality analyses laying out the intermediate outcome of each slice of information in each step.
Reference
Robert M Clark. 2013. Intelligence analysis: a target-centric approach (4th ed.). CQ Press, Thousand Oaks, Calif.
William Wright, David Schroh, Pascale Proulx, Alex Skaburskis, and Brian Cort. 2006. The sandbox for analysis. In Proceedings of the SIGCHI conference on Human Factors in computing systems — CHI ’06. ACM Press, New York, New York, USA, 801. https://doi.org/10.1145/1124772.1124890
Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of International Conference on Intelligence Analysis 2005, 2–4. https: //doi.org/10.1007/s13398–014–0173–7.2 arXiv:gr-qc/9809069v1
Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing Taxonomy Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems — CHI ’13. ACM Press, New York, New York, USA, 1999. https://doi.org/10.1145/2470654.2466265
Paul André, Aniket Kittur, and Steven P. Dow. 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’14). ACM, New York, NY, USA, 989–998. https://doi.org/10.1145/2531602.2531653
