How to analyse a research paper

When I began my MSc years ago, I sort of ran into research wildly and blindly. I grabbed paper after paper and viciously tried to read as many as possible. I wanted to rush through my MSc.

And like my lack of a bug solving strategy, I had a similarly adhoc paper reading/analysing “strategy”. My “anti-process” consisted of adhoc note writing, random highlighting, and scattered ideas. Sometimes I’d read full papers, sometimes only conclusions. Sometimes I’d read them top down, sometimes I’d read sections in an arbitrary order. Sometimes I wrote notes on the paper itself, sometimes I wrote them on a separate piece of paper, while other times I would write them in a google spreadsheet. The notes themselves were of varying formats and depths. Sometimes my notes were well written and understandable, sometimes (well most of the time) they were TOTALLY incoherent and useless.

I remember after my first few months of avid paper reading, I wanted to start writing a background chapter, only to find that I’d left a realm of disorganised, unprioritized chaos in my midst. I didn’t know where to start and where to stop and what gaps still existed. I wasn’t sure quite what I’d learnt and certainly didn’t know if I’d understood the field or not. How on earth could I be a good researcher if I couldn’t figure out a process for reading research?

It’s only in the past recent months (so 6 years later, post-MSc) that I’ve decided to spend time self-reflecting and in turn cultivating better paper reading habits. I’ve spoken to many of my favourite researchers in an attempt to understand their process and I am busy consolidating what they do into a process of my own. I’m hoping that I’m not the only sad sod who struggles to get much out of reading a paper and that maybe my failures and learnings will assist others.

Now, I’ll (hopefully) be doing a couple of blogs in this series, since there are many angles to reading research papers:

• How to chose what papers to read?
• How “deeply” should you read each paper?
• How to actually read the paper?
• How to analyse and structure your thoughts about a specific paper?

This blog deals mostly with the last of those 4: “How to analyse and structure your thoughts about a specific paper” and touches on “How to actually read the paper” since it highlights what you should be looking out for when reading.

Disclaimer: I have a background in Computer Science. So my strategy will likely only truly make sense if you’re in that sort of field. Hopefully some of the strategies are cross-disciplinary. Send me your thoughts if you see something relevant (or not) to your field

My New Paper Analysis Process

In my updated “process” I tend to do the following: After reading a paper (I’m trying top to bottom), highlighting and making arbitrary marks in the margins, I like to go through those margin notes afterwards, categorise them and collect them into a step by step analysis which addresses each specific theme/dimension. One day, once this has been further refined, I want to develop a template for analysing research papers that we can share and hopefully use to optimise our research reading process.

Being inspired by Julia Evans and her epic zines, I draw the little mind map below to describe the “themes” that I now look at when analysing a paper (one day I’ll convert it into a comic!)

Because my little mind map (a) needs to be digitised and (b) needs a fair amount of work still, I’m going to go through it. Let’s go!

1. Key Contributions

The key contribution is usually summarised in your abstract and conclusions. This is obviously of utmost importance, so I dedicate a few sentences to explaining it in my own words

2. Secondary Contributions

These aren’t always mentioned in the abstract so one picks them up while reading. This could be a unique strategy for analysing results or a new metric that may prove useful.

3. Personal Discoveries

A research paper is more than it’s contributions. Reading a research paper will add to various parts of your knowledge — things that have been cited from other paper’s which they are using in the present paper, or even useful paper writing styles. I like to classify my personal discoveries as follows:

• Unseen previous work, concepts, algorithms, and terminology.
• Interesting methodologies, datasets and benchmarks.
• Clever analysis techniques.
• Useful paper writing techniques and styles, e.g.: The choice to put related work at the end, or to add an appendix with in-depth details.

4. Questions

Reading a paper the first time will leave one with many questions. Often, the paper’s themselves do not have the answers. To answer the questions I need to read something else, look at the source code or sometimes reach out to an author. These are the types of questions I come out with:

• Missing background: “What is all of this?”
• Unknown terminology: “What is that?”
• Potential errors: “That doesn’t seem right!”
• Unexplained decisions: “Why did they chose to do that?”

5. Important References to Read

No researcher is an island, not even John Nash (who notoriously referenced only one person other than himself). While reading a paper, I note what body of work that paper is building on and highlight the important references. Now, in a perfect world, we’d read EVERY reference, but sadly, that is not practical and we need to focus our efforts. I like to list a couple of key references that I need to go back and read in order of perceived importance, so that I know which paper to pick up next.

6. Criticisms

Peer-review is not perfect, and our levels of scientific standards differ. I often spot problems about methodology or in author decisions that bother me and should be highlighted. Some examples of criticisms I regularly have:

• Bad or weak experimental methodologies. “They only ran one sample!”
• Incorrect analysis: “You cant draw that conclusion from that statistic!”
• Bad description of algorithms: “This is missing sections!”
• Missing or overloaded symbol definitions: “What the hell is ψ?”
• Bad writing: “Wait, what?”

7. Ideas

Our primary job as a researcher is to understand the knowledge base, ponder “what if” or “why” and then try answer it. So the ideas we come out of reading a paper are of utmost importance!

8. Rating

I’ve decided to rate each paper I read. This will help guide one through the field, highlight the best researchers, and understand the importance of a contribution. I use a simple 5 star rating. These are the axes I rate (let me know if you have more)

• Impact: Did this change the face of AI forever? Or was it a rehash of old algorithms?
• Methodology: How rigorous was their experimental techniques? How many samples did they run per configuration? Did they optimize their parameters?
• Analysis: How rigorous was the analysis?
• Quality: How easy was the paper to understand?
• Relevance: Is this relevant to my research?

So, there we have it…

Now that I’ve written this all up, I’ve essentially formalised a process for myself that I will follow. Additionally, I want to make sure the first top part of each paper analysis has the authors, the year, the journal, and my 5 star ratings.

This is very much a WORK IN PROGRESS, so if you have something to add that you think is valuable, please let me know and I will include it in my process. This is something I am actively working on trying to improve, so I welcome any comments, contributions and criticisms