Unpacking the Black Box of Scholarly Synthesis
How do scientists synthesize existing knowledge into new?
Effective scholarly synthesis is a crucial foundation for conceptual progress in research. However, it is rare, in part due to a lack of adequate tools. To design the next generation of synthesis tools, we unpack its black box by analyzing the intermediate products, processes, and tools that scholars use to synthesize prior knowledge into new concepts.
In 2019, the economist Esther Duflo shared the Nobel Prize in Economics for her work on experimental approaches to alleviating global poverty. She traced the genesis of this fruitful research agenda to an inspiring “masterful survey of the literature on health, education, labor markets, and household behavior in development economics…” she encountered during her Ph.D. studies. This review paper enabled her to identify important questions and apply appropriate methods.
This example illustrates how synthesis — the creation of a new conceptual whole from independent intellectual sources — is a foundational process in scholarly work. An effective synthesis is generative: it enables innovation in scholarly and scientific work.
Despite their importance, effective synthesis products are rare. While most published scholarship includes some review of prior literature, a minority truly synthesize prior knowledge. Instead, literature reviews are frequently barely more than lists of citations, which has been described by psychologist Daryl Bem as “resembles a phone book — impressive case, lots of numbers, but not much plot.”
Software and information systems could help scholars overcome the difficulties of synthesis and make effective synthesis less rare. But we don’t have a good picture of what scholars are actually doing during synthesis. To develop the next generation of tools for synthesis work, we need a detailed, situated understanding of the work practices behind synthesis products. We need to open up the “black box” between original sources and resultant synthesis: what are the fundamental “data structures” and “operations” in the processes of synthesis? How do scholars use tools to do these operations and sequence them to create an effective synthesis, and where are the most significant pain points?
We translate this need into the following research questions, and seek answers through a set of contextual interviews with scholars.
- What intermediate products do scholars create in the course of synthesis, and for what purposes?
- What processes operate on and between these intermediate products and to what ends?
- What are points of friction or challenges do scholars confront in creating and using intermediate products?
Intermediate products, processes, and tools for synthesis
We identified three distinctive intermediate products of synthesis work: (1) in-source annotations, markings and marginal notes within a single source; (2) per-source summaries, written distillations of the key conceptual building blocks of a single source that the scholar might want to use as part of a synthesis; and (3) cross-source syntheses, representations of an overall understanding of the research problem that is emerging from the papers being synthesized.
We also observed three themes regarding the processes that operate on and between intermediate synthesis products.
First, there is upward progression of the intermediate products in increasing levels of structure and formality. This is the transition from an earlier, casual, low-level phase, to a later, formal, high-level phase.
Second, researchers deliberately construct intermediate products with “higher level” products in mind. This involved selectively adding, retaining, or dropping key details from sources that might prove useful for the construction of the next intermediate product. For example, during the transition from in-source annotations to per-source summaries, one participant reads a paper with two questions in mind: what this paper does (the “problem”) and how they do it (the “solution”). The answers to these questions, in the form of a schematized problem-solution statements, are then used to construct a structured per-source summary.
Finally, researchers need backward engagements with earlier intermediate products, referring back to confirm details. The backward engagements from the cross-source synthesis product all the way down to in-source annotations were most common when there was a substantial time lapse between these intermediate products.
The following figure illustrates our initial model of the process of synthesis that covers the intermediate products and processes we identified in this study. It resembles the sensemaking model in intelligence analysis, which also describes iterative processes producing progressively more structured and formalized intermediate products, but with frequent back-referencing.
Diverse and fragmented tooling practices for synthesis work
Regarding the tooling practices for intermediate products and processes, there is a considerable diversity and fragmentation amongst the tools selected, as shown in the following figure.
To cope with fragmentation, researchers developed creative mechanisms such as simultaneous display of the first two intermediate synthesis products, and customized naming of sources to memorize and later retrieve ephemeral knowledge.
Implications and next steps
Our findings are a first step towards a deeper understanding of scholarly synthesis that can enable design of effective tools for synthesis. One immediate implication is that tools for synthesis need to account for transformations between quite distinct intermediate products of synthesis. Another is that, while frictions exist in these transformations, at least some of those frictions — such as effortful selection of key details, and backward engagements — may be productive for synthesis and should not be designed away.
Following up on this, we want to examine how our findings generalize to expert scholars. We also want to study this problem more with “in-the-moment” observational methods, to deal with the current limitations of participants recalling their experiences.
