Storyboard your figures!

I think that one of the most important and useful exercise I can do when — before! — writing a paper, is to draw a storyboard. What is a paper storyboard? A list of figures, drawn on made-of-dead-trees paper, that represent what I expect them to look like given my previous knowledge of the things I am about to do in the paper.

I find storyboarding important for two reasons.

First, it helps me think about which data I want to collect, and how they should be presented. Drawing a mock figure requires to label the axes, figure out what the points will be, and so on. In some cases, it also helps deciding on the range of the parameters I want to look at.

Second, it helps me figure out what is surprising about the results. I operate under the (very strong) assumption that the mock figure I drew is what should be if the sum of all knowledge about the problem at hands is enough to describe it. This is a very strong and foolish assumption; but assuming it holds, when the actual figure do not resemble the mock one, it means one of two things.

I made a mistake. This happened before. This is the point where I triple-check (I have double-checked before out of principle) the maths, code, data, everything.

Or…

I found something new. If the observed outcome is not the predicted one, there is probably a story to tell. Is it qualitatively different? Quantitatively? The fact that there is a discrepancy is a good justification for digging further.

Another advantage to doing this is that it helps me decide when I am ready to really start working on a project. If I have a well defined question, I know the quantities I need to measure, and the quantities in response to which they should be measured. If I have read enough literature, I know the expected outcome. The most telling test of this is, Can I draw the figure showing the key result?. If I can, then it is time to start the actual paper.

I love exploratory analyses as much as anyone else; but it is easy to get lost in exploratory analyses. At some point, I need to stop exploring, and start building something. Drawing the figures I expect to see in the paper helps me do this.

To be perfectly honest, there is a risk with drawing the figures before even starting the experiment, simulation, etc, that will provide the actual data. Two risks, actually.

First, it is tempting to start repeating simulations, or tweaking parameters until the outcome matches the prediction. This is avoided, simply enough, by not doing it. If the only way to get the predicted outcome is to find just the right combination of parameters, surely the prediction is not as robust and general as it should.

Second, it sometimes constrains the reflection, because you essentially impose a problem on yourself — and this might not be the right problem to solve. This I avoid by proceeding iteratively. I start with Figure 1, do it for real, and after I have dealt with the outcome, I go on with the next step. It both helps me build a narrative, and make sure that any figure logically follows from the one before it.

Does drawing the figures in advance helps me write better papers? I don’t know. It certainly makes me more efficient, in that I know what to look for, and I can immediately tell whether the prediction was correct. It also helps me determine when I am ready to work on the real analyses, because I have both defined the problem, and what I expect the solution to be.

And it’s a good thought experiment.