Paper Notes Template
How We Read Papers
When Paper Club was in its infancy, all of us were new to reading academic papers. We were aware that papers require a different reading style than a novel or blog post, but struggled to identify concrete adjustments to make — which sections should we focus on? What content belongs in each section? What are common weaknesses and gotchas to look out for?
Then one day James Vanneman shared a blog post titled How to read and understand a scientific paper: a guide for non-scientists; in other words, exactly what we were looking for. We adjusted Professor Raff’s thoughts slightly to fit deep learning-specific content, and turned it into a template that frames our thinking when we approach a paper. It is not meant to be a rigid set of required Q&A, but rather loose guidelines to help us focus on the highest-value parts of each paper.
The template is shared below. We welcome any suggestions, and hope others can find as much use for it as we have.
Here are two examples of the template in action: fastText and GRUs vs LSTMs.
Overall impression: fill this out last; it should be a distilled, accessible description of your high-level thoughts on the paper.
⁉️ Big Question
“What problem is this entire field trying to solve?”
🏙 Background Summary
What work has been done before in this field to answer the big question? What are the limitations of that work? What, according to the authors, needs to be done next?
❓ Specific question(s)
What exactly are the authors trying to answer with their research? There may be multiple questions, or just one. Write them down. If it’s the kind of research that tests one or more null hypotheses, identify it/them.
💭 Approach
What are the authors going to do to answer the specific question(s)?
⚗️ Methods
What exactly did the authors do?
📓 Results
Summarize the results for each experiment, each figure, and each table. Don’t yet try to decide what the results mean, just write down what they are.
Things to pay attention to in the results section:
- Any time the words “significant” or “non-significant” are used. These have precise statistical meanings.
- If there are graphs, do they have error bars on them? For certain types of studies, a lack of confidence intervals is a major red flag.
- The sample size. Has the study been conducted on 10, or 10,000 people? (For some research purposes, a sample size of 10 is sufficient, but for most studies larger is better).
- Do the authors provide supporting data such as hyperparameters and training times?
- Do the authors provide code?
Do the results answer the SPECIFIC QUESTION(S)? What do you think they mean?
🤠 Conclusion
What do the authors think the results mean? Do you agree with them? Can you come up with any alternative way of interpreting them? Do the authors identify any weaknesses in their own study? Do you see any that the authors missed? (Don’t assume they’re infallible!) What do they propose to do as a next step? Do you agree with that?
👂 Questions
Drop any questions you have or would like to discuss here
⏩ Viability as a Project
Is the data available? How much computation? Can the problem be scaled down? How much code development is necessary? How much work to turn this paper into a concrete and useful application? How much will we learn? How do we prove success? What are the results of success?
🔁 Abstract
Does it match what the authors said in the paper? Does it fit with your interpretation of the paper?
🗣 What do other researchers say?
Who are the (acknowledged or self-proclaimed) experts in this particular field? Do they have criticisms of the study that you haven’t thought of, or do they generally support it?
📚 Other Resources
List any helpful references or citations that helped you understand the paper.
🤷 Words I don’t know
List and define any and all words you didn’t previously know.
