Escaping a black hole
The struggle and blessing of peer review
When a scientific paper is submitted for peer review, the hardest part is already done. Or is it? Here is my story of how a paper of ours evolved under peer review over the course of more than a year. After reading the story, you may be left wondering: is this about the difficulties of pushing a paper through peer review? Or about the challenges of scientific writing? Or about the maturation of one’s thinking during peer review? It is about all these.
A scientific paper is considered important if it changes how scientists think. Our paper falls in this category — it surely had changed our way of thinking about an important area in biology. The paper is about using evolution to coerce microbial communities into performing a useful function. Microbial communities often do wonders that member species cannot do. Wine and cheese fermentation, sewage treatment, and your own health all involve microbial communities. But what if community function is low?
Community function can be improved via an evolutionary procedure called “artificial selection”. Artificial selection has typically been done for animals and plants: Horse breeders breed fast-running horses to improve speed; crop breeders breed high-yield crops to improve yield. To artificially select or evolve microbial communities, one could start with many “baby” communities and allow cells to grow and mutate. “Adult” communities that perform the desired function best are selected to reproduce in the hopes of improving the function further. These communities are split into their own “baby” communities and cycle is repeated.
Superficially, this sounds straightforward — “you get what you select for”. However, few researchers have tried this, and for good reasons. Although a baby horse grows up to be an adult horse, a baby microbial community may grow up to be a very different community. For example, noncontributors — cells that do not contribute to community function — could grow faster than contributors and eventually destroy community function. Instead of brewing better beer, suddenly you’ve created something undrinkable!
For any community selection experiment, one can turn many knobs: the number of communities under selection, the number of total cells in a baby community, the time one waits for baby communities to mature, the mutation rate… It is not clear how one could best do such experiment.
We decided to simulate community selection instead of turning experimental knobs, since experiments are slow and labor-intensive. From simulations, we identified multiple obstacles to successful community selection, and proposed ways to overcome them. In a most dramatic demonstration, we predicted that a basic lab practice — using a pipette to split a community into baby communities — could potentially mess up an experiment, and that a more precise method may be required.
In Feb. 2018, we submitted our paper “Reducing compositional fluctuations facilitates artificial selection of microbial community function” to eLife, a top-tier journal. We had very positive experiences there as authors. eLife is also supportive of theoretical work. In fact, when contributing to an eLife editorial on theoretical biology (Shou et al., 2015), my own perspectives on theory and biology deepened.
I fully expected a smooth ride, like my other papers. I had already shown the paper to several people, including Bill Hazelton (a physicist at the Hutch) and Doug Jackson (an ecologist), and got rave reviews. And since our work was comprehensive, I expected that the reviewers would only ask for minor revisions, and paper would be accepted within two months. Thus, I was shocked when two weeks later, our paper was rejected by the editorial board without being sent to reviewers.
“Specifically, the editors agree that the papers tackle a very important topic…. This is indeed a frontier in microbial ecology and evolution. On the other hand, the presentation of the work is not convincing. The main article starts out with a fairly long ‘philosophical’ discussion.”
I did design a full-page, multi-panel figure to compare different types of artificial selection: selecting individuals, selecting groups of individuals from one species, and selecting communities of different species. I wanted to properly set the intellectual stage and to pay due respect to relevant work. However, the editor had a great point. A long “run-way” will fatigue the readers, and this section could be easily moved to a supplementary file at the end of the paper.
“Then the actual model sections are fairly short, and technical details are relegated to the Methods section. But the technical details are very important, because the devil lies in the details.”
Choosing the right level of details has never been easy, because it depends on the audience. Too few would irritate readers who want precise details, while too many would bore readers who want the bottom-line. What is the right balance?
“In fact, the modelling approach is a bit cumbersome, and should be compared to previous frameworks (e.g. reference X).”
I looked up the reference, and the model was of a different style. It is “black-and-white” wherein an individual contributes either 0 or 1 to community function. Our model is “grey-scale”: a mutation could eliminate contribution, or slightly increase or slightly decrease contribution. Thus, an individual’s contribution is between 0 and 1, a common scenario in biology. So, I added two sentences to the introduction, stating that our model is more biologically realistic.
“Nevertheless, the paper is very interesting from a theoretical point of view, but there is also a need for the models to be less tied to a specific kind of interaction.”
We performed additional simulations on different types of communities, and got exactly the same results. Thus, our results are not tied to a specific kind of interaction.
State of emergency
In March, our revised paper went back to eLife. Ten days later, it was returned without review.
“I am familiar with the work and just on its merits of creativity and importance to our understanding of microbial communities, I could certainly imagine seeing this paper in eLife. This being said, the manuscript is very hard to follow.”
I became alarmed. Clearly, this new editor had heard about our work in a conference, and was even fond of it. It was also painfully clear that my writing was flawed —although it was fine for Bill Hazelton and Doug Jackson, it failed to make sense to this new editor. In other words, the paper only captured a fraction of the interested audience.
“I mean the abstract alone is dry and confusing. For example, it is completely unclear whether the first half of the abstract describes what can be done (and has been hard to do) in the field in general or what the authors do in this paper.”
I looked back at my abstract. “To improve community function, we can artificially select communities.” I see! The “we can” could be interpreted as “we the authors” can or “we the field can”. I replaced it with “one can”. Even one wrong word could derail a reader’s perception!
“It’s impossible to figure out whether this is a purely theoretical study, a purely experimental one, or a combination.”
We did state early on that our work was purely theoretical. However, a lone sentence in the midst of a long paper could be easily missed. In contrast, “pipetting” — an experimental term — did pop up numerous times since our simulation tracked chance effects caused by pipetting. A whimsical idea struck: why not say that we simulated an experimentalist called Selexa? “Selexa” sounded like “selection” and “Alexa” (Amazon’s digital assistant). If we wrote “Selexa pipetted cultures”, then no one would mistake our work as real experiments.
“The rest of the manuscript seems rather dense as well… I would suggest that the authors do a serious overhaul of the manuscript.”
The bell of apocalypse tolled. I was sure that the editor was correct — if a reader finds anything hard to read, it is always the fault of the writer. But at the same time, I had no idea what the editor meant.
I looked hard at our paper. Yeah — it was kind of dense. Unlike a traditional article with one and only one central point, it had quite a few points. Each time I gave a talk on this work, the audience would ask interesting questions. My colleagues Harmit Malik and Jim Roberts wanted to know why we could not first optimize each species in isolation, and then combine them to form a blockbuster community. So, we wrote a section to demonstrate the difficulty of this approach. Lin Chao (UCSD) and Jeff Gore (MIT) questioned certain assumptions we made. So, we added a section on that. Alvaro Sanchez (Yale) asked why we did not model noises associated with community function measurements. So, another section was born. And, of course, the “generality” section requested by the eLife editor from the first round. I was unwilling to cut any section, because these were all excellent points.
Perhaps, I got the order wrong? I often lament my sub-optimal memory. However during re-writing, I constantly wished that I had a memory eraser. That way, I could read my paper as if someone else had written it, and then I would have no trouble spotting problems. Without any directions, I experimented with re-writing. Months later, I sent the revised paper to Kirill Korolev (a theoretician in Boston University). Kirill concluded that it was straightforward to read.
I submitted the re-written paper to eLife the third time in June 2018. I asked Diethard Tautz, the senior editor in charge, whether he could have one of his postdocs take a look first. Diethard had been incredibly supportive of my work since before the birth of my lab, and this time was no exception. Three days later, we got feedback:
“It really is very hard to read, even the abstract… Choosing SELEXA as a name for an imagined experimenter is not optimal. People are familiar with SELEX experiments (RNA evolution experiments). It is too easy to mix this up. Also the term ‘pipetting’ within an entirely theoretical paper is very confusing — why not stick to well understood terms, such as magnitude of stochastic error.”
I deleted “Selexa”. However, I could not bring myself to delete “pipetting”, because I wanted to connect with experimentalists. I sent the paper to Alex Sigal, an HIV researcher from Africa Health Research Institute. Alex echoed a similar sentiment, “The first time, I just did not get it. The second time, I got it — and it is really cool, but very heavy!”
Diethard must have felt sorry for us. He wrote,
“We have had recently a visit by Nick Barton and he told me that he has also tried to get what he considers a major landmark paper into a general journal (I attach it) but was told he should publish it first elsewhere and then write a better understandable version for a review journal. Well, he did only the first part so far…”
I’m perhaps not alone after all… But this thought pushed me into an even deeper despair: am I doomed? A delicate balance must be struck between understandability and precision, between readability and thoroughness, and between connecting with theorists and connecting with experimentalists. What is that balance? It seemed that there was no right answer. I felt being stuck in a black hole that sucked away my energy and happiness.
While struggling with my writing, several colleagues commented, “We don’t waste time fighting with editors. We just get papers out in next-tier journals and move on.” This made sense, but I had other considerations. The first author Li Xie was waiting for this paper to get into a high-profile journal for her green card application. Giving up would not be fair to her. Moreover, it is critical for me: I must find a way to reach a broad audience so that I can secure future grants!
Who might rescue us? The first name came to mind was Jim Bull from UT Austin. Jim has thought a lot about many questions in evolution biology. Years ago, he reviewed my conceptual paper (Shou, 2015) and provided constructive suggestions on writing. Jim is also well-known in the field for his wisdom and generosity. Jim once visited me in Seattle. I offered to pick him up from the airport. He said, “I do not want to commit the crime of wasting your time.” Mind you, he is far more senior than I am!
I hate to waste Jim Bull’s time. But I really had no other options. I sent the paper to Jim. Jim commented, “Yes, this is difficult to follow.” Strangely, I was overcome with a sense of relief. I finally saw a ray of hope — Jim detected the same problem as other readers, and he could surely pull us out of this black hole.
For a month and a half, Jim gave us feedback one bit at a time. A life line had been thrown at us. It felt like I went straight back to graduate school for a second PhD since Jim’s style was very different from the molecular genetics writing style I learned in graduate school. Jim’s contribution was so critical that I offered him co-authorship. He declined, “No. Not appropriate. I do this all the time.”
“Hello darkness, my old friend”
In October 2018, we submitted the revised paper “Artificial selection of microbial communities can become effective after using evolution-informed strategies” to eLife. This was our fourth submission. It was again rejected without review.
“The first, who has not seen previous versions remarks: it is often difficult to disentangle a genuine insight from rather trivial observations. Statements like ‘non-heritable variations -> no successful selection’ are pretty obvious.”
Ironically, doesn’t Darwin’s “the survival of the fittest” sound equally obvious? After all, being “fit” is being able to survive!
“What I miss most are quantitative predictions and relationships… I am sure there is more one could do.”
Essentially, this editor wanted a second paper on quantitative predictions of community selection efficacy, which we were (and still are) working on. There is always more we could do — in fact, a great paper should open a door for many future investigations.
“I still feel the manuscript is confusing”
I looked hard at my writing, and realized that perhaps, I had pushed “one central point” too hard. The success of artificial community selection was more complex than one central point.
The second editor was the same editor who believed that a black-and-white model is superior to a grey-scale model. I respectfully disagree, because to me, they are complementary.
In any case, philosophical debates are rarely productive. Time to move on.
A new home?
Which top-tier journal should I try next?
Years ago, Lauren Richardson, a senior editor of PLOS Biology, contacted me to find out what interesting stories were brewing in our lab. I asked her about theoretical manuscripts. Lauren gave a positive response, although she quickly added that most papers would also have experiments. Sure — theory fortified by experiments is always superior to pure theory. Regardless, I decided to give PLOS Biology a try.
I did not want to ruin our chance with PLOS Biology. Thus, I organized a paper-staring group meeting where group members stared at the title, the abstract, and each of the main figures. Alex Yuan, a new graduate student in our lab and a coauthor of the paper, suggested, “If you don’t want any confusion between simulations and experiments, why not add the word ‘simulation’ to the title? No one could possibly miss that!” That was such an obvious solution, yet I did not think of that! Alex continued, “We could add ‘insidious’ to the title since some of the challenges were really not obvious.” Brilliant! The meeting went on for hours. I actually had fun.
In December 2018, we submitted our paper “Simulations reveal insidious challenges to artificial community selection and possible strategies for success” to PLOS Biology. I addressed cover letter to Lauren Richardson. In the cover letter I argued, somewhat defensively, why pure theory papers can be important. Two days later, we were informed that the paper would be sent out for peer review. The speed was astonishing. For the first time in over 10 months, I started to feel that the load on my shoulders became bearable. I was even tempted to celebrate — finally, the paper would be in the hands of reviewers!
In Jan 2019, I was invited by Wallace Marshall (UCSF) to give a talk at the “Stochastic Physics in Biology” Gordon Conference. I decided to talk about Li’s work. Because talks were constrained to 30 minutes, I was forced to focus on a small number of points. It then became clear to me that an entire section of our paper should have been moved to supplementary files.
Afterwards, my session chair James Boedicker (USC) informed me that he was one of the reviewers of our paper, that he liked our work, and that he had submitted a two-page review to the editor. I could not believe how small the world was. I invited James to have lunch with me, trying to fish out his thoughts. While talking with him, I realized that in my relentless pursuit of simplicity, I had gone too far. For example, James wanted to know not only the conclusions of a figure, but also the exact steps that had led to the figure.
Two weeks later, we received four reviews, all positive. All four reviewers offered great suggestions on various aspects of our writing.
Reviewer 4 inspired us to write a “Future Directions” section in Discussions.
Reviewer 3 stated,
“I couldn’t help but feel toward the conclusion that this modeling work amounts to a death knoll for the prospect of community selection.”
Yes indeed, our paper had a gloomy undertone. But reality is reality…
Reviewer 2 (James Boedicker, USC) wanted the central problem to be presented in an ultra-explicit fashion in Introduction. Thus, I added a Figure to Introduction so that readers could quickly board the ship instead of swimming in the ocean of information.
Reviewer 1 (Sara Mitri, University of Lausanne, Switzerland) asked,
“The way in which you select only the best or the top two communities for the next generation results in a very tight bottleneck (1–2%). What happens if you relax it? Or can you motivate this choice?”
I was tempted to answer these questions with a couple of sentences. This choice was intuitive — after all, top communities were enriched for high contributors. However, we decided to run simulations just to make our argument bullet-proof. Simulations are fast.
Twist of fate
The high-performance computer cluster chimed away. A day later, we received results. We were ASTONISHED. Choosing the top 5% or even top 50% communities to generate baby communities outperformed our original strategy of choosing the top 2 communities. The result was so unbelievable that we double-checked it using an identical simulation that had been written independently. The result was indeed correct.
The next month was spent trying to understand this new result. Once we understood it, it became intuitive: top communities could be top either because they harbored the best contributor mutants or because they were lucky (e.g. receiving a more-than-average number of cells due to inevitable fluctuations associated with pipetting). Thus, when we allowed more communities to reproduce, we gave “unlucky but good” communities a chance. This turned out to qualitatively change the gloomy tone of our paper. “Insidious” in our title, which received complaints from two reviewers, was justifiably deleted.
The survival of the unlucky
Another month passed as Li and I incorporated this new thinking throughout the paper. As I revised under the guidance of the four great reviewers, I felt incredibly fortunate. After another month of re-review, our paper was accepted for publication (Xie et al.2019).
Weeks later, I was invited to review a PLOS Biology submission “If we can breed dogs, why not microbiomes?” To my great surprise and delight, it was a “primer” written by Sara Mitri and her students. You may ask what is a primer? A primer “provides concise and accessible introduction to an important area in biology… alongside a research article that would benefit from additional context and explanation.”
It then struck me that by sending our paper out for review, senior editor Lauren Richardson and academic editor Mark Siegal (NYU) gave our “unlucky but good” paper a chance to survive.
Without Li Xie, this story would not have been born. Alex Yuan’s unfailing sense of creativity and insight brought much light in dark moments.
We thank editors of PLOS Biology for giving us a chance. We thank colleagues, editors of eLife, and the four Reviewers for gifting us priceless time — time that could have been spent with the loved ones, on their own work, in a theater, under stars…
I thank James Boedicker, Chuxuan Sun, Sabrina Richards, Yan Fang, George Moore, and Alex Yuan for critiquing my story.
- Underlines contain clickable links to research articles relevant to this story.
- Suggestions and comments can be sent to firstname.lastname@example.org.
- For my other stories, see https://email@example.com.
- This story was featured under the Medium.com “Science” category. Here is the free link in case you want to share it with friends.