Learn how to read papers based on top researchers’ recommendations
How To Find, Read, And Master Research Papers
Introduction
I recently graduated from my CS bachelor’s program and will start my graduate program soon. In this period, I read some papers and articles and watched some tutorial videos to find out How to find, read, and master research papers, which is a crucial skill for every researcher. This article is a summary from Top researchers’ recommendations which I divided into three main sections. You can access to full resources from the references section.
More about the sources
Let me briefly introduce the sources of recommendations which is used in this article:
Andrew NG: Co-founder of Google Brain, Coursera, deeplearning.ai, and a professor at Stanford
Sebastian Ruder: Research scientist in the Language team at DeepMind, London
Pete Carr: Professor at the University of Minnesota
Srinivasan Keshav: Professor at University of Cambridge
Yannic Kilcher: PhD student at ETH Zurich
Step 1: How To Find Research Papers
This section contains recommendations from Andrew NG and Sebastian Ruder.
There are several sources that you can find papers related to your field, and keep updated.
- Twitter: You can keep updated by following related researchers. For example, Andrew Ng usually tweets about recent Conferences, new courses, and topics.
- ArXiv: ArXiv is a great source of papers and you can also stay up-to-date by using services such as daily arXiv digest, which Ruder describes it as “feels like drinking from a firehose”, arXiv sanity preserver and arXivist.
- Reddit: By joining related subreddits, you can be informed of the hottest topics in your field and participate in their discussions. Deep Learning and Machine Learning subreddits are examples of great subreddits that you can join if you are interested in.
- Google Scholar: You may have used Google Scholar to find related researchers and papers in your field. The interesting thing is that you can also follow your desired researchers and be notified when they publish a new paper. Besides, Google Scholar recommends you similar papers and scholars based on your Scholar alerts.
- ResearchGate: ResearchGate’s structure is similar to ArXiv and Google Scholar. Its website is also user friendly and you can easily navigate through citations and references and find the newest related research papers to your field.
- Top Conferences: There are several sources that you can find top conferences in your field, and follow their proceedings and read the accepted papers. I personally find the most prominent conferences related to my field from Guide2Research and Google Scholar Top publications. If you are interested in AI, AI Conference Deadlines is a great source that shows you the top conferences’ deadlines so you can get a rough intuition about which one is going to be held soon.
- Papers With Code: If you are a machine learning researcher, you can access the most novel papers following with their codes and benchmark from this source. The State-of-the-Art section is also an impressive source that provides the top models in each task.
- Friends and Reading groups: There are many reading groups in universities and you can be informed of state-of-the-art papers by joining them. Participating in one of these groups also motivates you and makes a routine that is beneficial for long-term research.
You will be more successful if you read steadily rather than putting an intense effort over one weekend
- Andrew Ng
Step 2: How to Read a research paper
Up to now, I explained how to find the related papers in your field and how not to miss the new papers. Before diving into analyzing the reading method of a research paper, some essential points should be considered:
How to start and note the paper?
Andrew Ng explains that while he almost handles his research tasks using his iPad, he always has some printed papers in his suitcase. He attributes his method to the ability to flip between pages and skimming more efficiently.
Sebastian Ruder uses Mendeley, which is a useful cross-platform paper management system that allows you to search in your library. I prefer this choice too, and during the ACL2020 meeting, I realized that many top researchers in the world are using paper management systems such as Mendeley and Zotero too. They also have tools for highlighting, noting, and sharing the paper. If you need more extended tools such as writing with your stylus pen, you can store your papers on cloud using Dropbox, attach it to your paper management service and edit it using pdf editing applications (I personally use Xodo PDF reader )
Pete Carr recommends students to have a notebook or index cards to summarize and note important parts of each paper, so you can refer to that whenever you want to have a quick review of the papers you have read. If you prefer the digital format, I recommend Microsoft OneNote. Mendeley also provides a note tab for each paper. Both of these applications allow you to search through your notes.
The faintest ink is more powerful than the strongest memory
- Chinese proverb
Why you are reading this paper
After you prepared your papers, stored them in your favorite system, and provided the required tools, there are some questions that you should consider before starting a paper. Andrew Ng suggests to keep these questions in mind when you start a paper:
- What did the authors try to accomplish?
- What were the key elements of the approach?
- What can you use yourself?
- What other references do you want to follow?
When you can answer these questions, hopefully, that will reflect that you have a pretty good understanding of the paper.
Ok, now we are ready to dive into papers. But the question is How? I explain it in three main stages, the first is just to get the gist of the paper, the second is for understanding it, and the third one which is described in the third section is for mastering the paper. The two first stages are enough for having a general understanding of a paper for your research projects, and you can save your time and skip the final stage if you are not a reviewer or you do not need to digest every word of a paper. The third stage might also vary based on the fields you are researching.
The Bird’s-eye View of the paper
After this stage, you will have a big picture of what is going to explain in the whole paper. Srinivasan Keshav says that this step gives a general idea of the paper so you can decide whether you need to go further in this or not.
- Title/ abstract/ figures
The first stage is looking at the title, affiliation, and the authors. Yannic Kilcher explains that the title will inform you what is especial of the model the authors are presenting and you can have a hypothesis of why this paper matters with a few keywords. Yannic usually skips the authors’ section because he cannot memorize all of the authors and the most famous authors work in a huge lab so it will not mean that the paper is going to be of any good or bad. However, he pays some attention to the affiliation because renowned ones such as Google and Facebook get more public attention and they get more scrutinized so there is more pressure on them to produce a better result.
One of the most important sections in a paper is abstract. In this section along with the introduction and the conclusion, authors try to convince reviewers that their work deserves to be accepted in their journal or conference. Try to read this section carefully, and you likely can find the answers to the first two questions that I mentioned before.
The next step is looking at the figures. Andrew Ng says that especially in Deep Learning, there are a lot of research papers where sort of the entire paper is summarized in one or two figures in the figure caption. Jordan B Peterson, psychologist and one of three “life changing” professors at the University of Toronto also says that about half of our brain is devoted to visual processing. So, we can build a rough structure of the paper-based on just looking and the images and the captions
2. Intro/Conclusions/Tables + Skim the rest
The detail of the reading methods in this part varies based on the references I have used, but the most important part is devoted a short period to grasp the big picture of the paper and Do not engage with details. Introduction and conclusion are the best sections for these goals. Authors extend their abstract in these sections to convince the reviewers that their contribution is significant enough to be accepted. You can also have a broader answer to the two first questions mentioned before. The tables also provide the result of the work in a coherent structure. Yannic Kilcher recommends being skeptical in this stage. What are the claims of the authors and did they prove that based on their experience? If you place yourself as a reviewer and try to convince yourself, you are more likely to have a better understanding of the paper.
Next, you should skim through the section titles and subsections, and after that, You should decide whether to leave the paper or continue reading it. The important point is that do not hesitate to leave a paper if you think you are not interested in it or it is not relevant. Andrew Ng suggests reading 10% of some related papers, and then decide to read one completely, add some more related papers based on the citations of the read one, and continue this cycle until you are satisfied with the subject you are learning about. Jeff Dean also expressed that he’d rather read ten papers superficially than one paper in-depth to get as much inspiration as possible as one can always go back to read a paper more deeply (he conceded that reading 100 abstracts might be even better). Sebastian Ruder says that by following this method, with a search-able paper management system you can always go back and reread the most relevant ones.
Understanding the whole paper
I suppose that you skimmed through some related papers and chose one to go deeper into it. First, skip the related work. you can skim through it but honestly speaking, authors try to cite some of their previous works, some friends, and probable reviewers. So, it will not give you much unless you are willing to add some other resources to your library. try to read the experiments and the proposed method carefully, but skip the mathematical proofs and the parts which do not make sense for you. Remember that you are reading the paper to answer your questions, and you should now focus on how you can use the information from this paper yourself. Abstraction might help you in this case. For example, if the authors are using a complex optimization method that is not familiar to you, you can look at it just as an optimization method, not more. If you need more detail, you can search or read about it later.
Step 3: How to master a research paper
If you are reading this step, it means that you are a reviewer, or you have found a paper that is so relevant to your research topic. In this case, you should understand everything in the paper, and the best method for achieving this goal is to assess yourself.
If your paper has an open-source code, you can try to run it. Dealing with parameters and the structure of the code helps you to understand the proposed method.
The final step is to re-implement the paper. This is the toughest stage and so challenging. If you are dealing with math, you should be able to re-derive it from scratch. If you are dealing with code, you should implement it yourself from scratch. For the claims of the paper, you should virtually re-implement it, and be able to present the paper as the author and try to do that in a convincing manner.
Congratulations! You are now able to start your research based on the recommendations of top researchers. Honestly speaking, this is my first blog post and my native language is not English. If you have more sources, recommendations, and comments, I appreciate if you write it in comments.
You can also reach me via LinkedIn, or Twitter.
References
[1]: Stanford CS230: Deep Learning | Autumn 2018 | Lecture 8 — Career Advice / Reading Research Papers
[2]: 10 Tips for Research and a PhD
[3]: Deep Learning Indaba 2018 edition 🌍
[4]: How to Read a Paper