Paper Notes Template

How We Read Papers

Tiger Shen
Aug 20, 2017 · 3 min read
credit: Jeff Dean

When Paper Club was in its infancy, all of us were new to reading academic papers. We were aware that papers require a different reading style than a novel or blog post, but struggled to identify concrete adjustments to make — which sections should we focus on? What content belongs in each section? What are common weaknesses and gotchas to look out for?

Then one day James Vanneman shared a blog post titled How to read and understand a scientific paper: a guide for non-scientists; in other words, exactly what we were looking for. We adjusted Professor Raff’s thoughts slightly to fit deep learning-specific content, and turned it into a template that frames our thinking when we approach a paper. It is not meant to be a rigid set of required Q&A, but rather loose guidelines to help us focus on the highest-value parts of each paper.

The template is shared below. We welcome any suggestions, and hope others can find as much use for it as we have.

Here are two examples of the template in action: fastText and GRUs vs LSTMs.

Overall impression: fill this out last; it should be a distilled, accessible description of your high-level thoughts on the paper.

⁉️ Big Question

“What problem is this entire field trying to solve?”

🏙 Background Summary

What work has been done before in this field to answer the big question? What are the limitations of that work? What, according to the authors, needs to be done next?

❓ Specific question(s)

What exactly are the authors trying to answer with their research? There may be multiple questions, or just one. Write them down. If it’s the kind of research that tests one or more null hypotheses, identify it/them.

💭 Approach

What are the authors going to do to answer the specific question(s)?

⚗️ Methods

What exactly did the authors do?

📓 Results

Summarize the results for each experiment, each figure, and each table. Don’t yet try to decide what the results mean, just write down what they are.

Things to pay attention to in the results section:

  • Any time the words “significant” or “non-significant” are used. These have precise statistical meanings.
  • If there are graphs, do they have error bars on them? For certain types of studies, a lack of confidence intervals is a major red flag.
  • The sample size. Has the study been conducted on 10, or 10,000 people? (For some research purposes, a sample size of 10 is sufficient, but for most studies larger is better).
  • Do the authors provide supporting data such as hyperparameters and training times?
  • Do the authors provide code?

Do the results answer the SPECIFIC QUESTION(S)? What do you think they mean?

🤠 Conclusion

What do the authors think the results mean? Do you agree with them? Can you come up with any alternative way of interpreting them? Do the authors identify any weaknesses in their own study? Do you see any that the authors missed? (Don’t assume they’re infallible!) What do they propose to do as a next step? Do you agree with that?

👂 Questions

Drop any questions you have or would like to discuss here

⏩ Viability as a Project

Is the data available? How much computation? Can the problem be scaled down? How much code development is necessary? How much work to turn this paper into a concrete and useful application? How much will we learn? How do we prove success? What are the results of success?

🔁 Abstract

Does it match what the authors said in the paper? Does it fit with your interpretation of the paper?

🗣 What do other researchers say?

Who are the (acknowledged or self-proclaimed) experts in this particular field? Do they have criticisms of the study that you haven’t thought of, or do they generally support it?

📚 Other Resources

List any helpful references or citations that helped you understand the paper.

🤷‍ Words I don’t know

List and define any and all words you didn’t previously know.

Paper Club

Summaries, thoughts, and questions about foundational and cutting-edge academic papers about deep learning.

Tiger Shen

Written by

Hey, I’m Tiger and I like to write :)

Paper Club

Summaries, thoughts, and questions about foundational and cutting-edge academic papers about deep learning.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade