Lessons learned from a PhD in Machine Learning
Part 2: How to get through
This is Part 2 in the series. You can find Part 1 here and Part 3 here.
Doing research
After having chosen a research topic and reviewed the literature, you are hopefully excited now to do your own research! Assuming that you have brainstormed some ideas to start with (either alone or with your advisor), a good habit is to vet these ideas by pitching them to some third party. Ideally, you want this to be someone who has some expertise in the area and is unlikely to steal your idea (most people won’t steal it, but it can’t hurt to be a bit careful). At the beginning of your PhD, the most obvious people for this role would be your colleagues in the group (or the department). If the ideas are fundamentally flawed, they should hopefully be able to spot that. Generally, this alludes to a more fundamental piece of advice, namely that you should try to talk to as many people as possible about your research as often as possible (again, assuming they are all people that are unlikely to scoop you). This will help you develop a better feeling for which ideas are useful, how the research community generally feels about different topics, and how to communicate your ideas better. Moreover, it can help you find people that care about the same questions as you.
Throughout my research, I found that one of the strongest factors impacting my research success were my collaborators. Talking to a lot of people helps you find good collaborators, which will then help you do better research. A natural place to start is within your group, but you should also be open to talking to people in other groups in your department, even if their interests are only tangentially related to yours. Many of the most interesting collaborations start this way!
When it comes to starting your own research, a good way is to choose some relevant papers in the area and re-implement them. You will probably need their methods as baselines anyway and you’ll be surprised how many things you can learn about a problem and a method by re-implementing it from scratch. Also, never be afraid to email the researchers who wrote the papers. You can tell them how much you like their papers and ask any questions you might have. In the worst case, they just won’t have time to respond, but you will still have made their day a bit brighter. In the best case, they will answer your questions and might even be interested in collaborating with you!
Finally, there will be times when you feel like you are not smart enough to do research and you don’t belong in academia and everyone else is just doing better research than you. This is called impostor syndrome and is unfortunately very common. Some other people have written great comforting posts about this (e.g., Charles Sutton) and what usually helps is, again, to talk to some other people (e.g., your colleagues) to realize that they actually all feel the same way and that research is just generally not always easy. After all, that is part of the reason why we do it in the first place.
Coding up experiments
If your area of research is machine learning (or some adjacent field), chances are that you will need to write some code to implement experiments testing different algorithms. The main language of interest will probably be Python, although Julia seems to have a growing community in machine learning too. I found that for preliminary prototyping and model or data exploration, Jupyter notebooks work really well, especially in combination with Jupyter Lab. However, you should be aware that notebooks also have many shortcomings (see this entertaining talk by Joel Grus), so the best practice is usually to regularly refactor your code into standalone Python scripts and modules once you’re done with the prototyping in the notebook. Generally, a good habit is to also refactor that code regularly, for instance, using the strategies of the Refactoring Guru. You should also try to write tests for your code, which will save you a lot of headaches later. Good packages for that are pytest or unittest. When it comes to testing the machine learning models themselves, that is a bit trickier, but you can get inspiration from the strategies applied by Jeremy Jordan or Eugene Yan.
Apart from coding up the models themselves, it often pays off to set up a nice experimental pipeline early, so you can then just scale up quickly to many baselines and datasets later. A tool that I’ve used a lot for that purpose is sacred, but you can also try Weights & Biases, or the recently open-sourced XManager from Deepmind. For plotting figures (preliminary ones, but also for the final paper), I mostly use matplotlib and seaborn. My main recommendations regarding these tools would be first to not treat the figure size as some random parameter but actually set it to the size that you will want in the final paper. For instance, in a NeurIPS paper, your figures should be 5.5 inches wide, and in an ICML paper, 3.25 inches for single-column and 6.75 inches for double-column figures. Sticking to these sizes in your plotting will make sure that the font sizes remain constant (and hopefully readable) across different figures in your paper (there are also libraries that can help you with that, e.g., tueplots). Secondly, you should try to make sure your figures look nice and clean, since this is one of the first things that readers (and reviewers) of the paper will notice. When in doubt regarding the layout of a figure, feel free to ask some colleagues for feedback or have a look at one of Edward Tufte’s books. You should probably spend at least an hour per final figure. Thirdly, you should always export the figures as a vector graphic (i.e., usually as a PDF file), so that you don’t get issues with the resolution in the embedded figure in the paper. Alternatively, you can also export the figure as a PGF directly (the TiKz file specification), but this is a bit more finicky and I would only recommend it to advanced LaTeX users. And lastly, make sure to never use Type 3 fonts in your figures. Annoyingly, Type 3 fonts still seem to be the default in matplotlib, but you can change this in your .matplotlibrc file or by calling matplotlib.rcParams[‘pdf.fonttype’] = 42 in your code (see a tutorial here).
Finally, a note on confirmation bias: What I have often observed (also in myself) is a pattern of coding up some experiment, realizing that the results don’t look as expected, then debugging the code until the results look nice, and then calling it a day and keeping the experiment that way. Of course, there is generally no reason to believe that after the results look nice, there are no more bugs left in the code, and indeed the nice results can be due to some bug in the same way that the previous bad results could have been. So this is just to say that it’s important to not stop debugging when the results look nice, but to continue to critically assess the correctness of the experiments and arguably even more so, the nicer the results look. If they look too nice to be true, that’s usually because they are. Of course, there are many incentives to stop the investigation when the results look nice and paper-worthy, but you should do yourself and the community the favor of staying vigilant and making sure to not fall prey to this confirmation bias. As nice as the prospect of writing a cool paper might seem, it will definitely be more annoying (and potentially damaging to your reputation) if someone else finds a bug in your code that invalidates your results after publication than if you find that bug yourself before writing the paper. Note that it can also often help to have a colleague or collaborator look over your code (or even do pair-programming), since they might find bugs that you would otherwise not see. If you are running short on colleagues and collaborators, you can alternatively also try rubber duck debugging.
Interacting with your advisor
Again, depending on how hands-on or hands-off your advisor is, you will interact with them more or less frequently. That said, you should never be afraid to ask for a meeting if you really need one. Especially at the beginning of your PhD, it is perfectly normal that you will need more frequent meetings to get up to speed. A common mistake is to get stuck on something and then never ask your advisor for meetings because you “don’t have anything to show”. Meanwhile, your advisor might assume that your research is going great because you never ask for help.
This mistake is founded on a common misconception, namely that those meetings are mainly for your advisor. They are not: they are for you! This implies that the main goal of the meetings should also not be to show your advisor all your impressive progress, but to actually use the time most efficiently to get help for your next steps. Often, this will of course entail some presentation of progress, but it is key to understand that this should only be included to the extent which is useful for you, and not for its own sake. In general, it should be in your best interest to ensure that your advisor is efficiently caught up on the current stage of your project and understands what your blockers or obstacles are and where you need help. You can use different tools to ensure that, but a reasonably low-tech solution is just a shared Google Doc, where you prepare an agenda for the meeting beforehand and keep meeting minutes and TODOs for the next time. You can also use this document to present some key research results, although often it can be more effective to prepare a short slide deck instead. You should of course also be mindful of your advisor’s time and prepare the meeting in such a way that it can feasibly finish within the allocated time slot. And similarly to how you should not be afraid to ask for an extra meeting, you should also not be afraid to cancel a scheduled meeting if you feel like you don’t need help at the moment. Your advisor will likely appreciate the donated time in their schedule.
Interacting with the community
While your advisor will generally be your first point of contact, you should also try to interact more broadly with the research community. As I alluded to above, this includes your colleagues in the group, people in the same department, possibly people at other departments at your institution, and people at other institutions. Again, trying to regularly talk to as many people as possible is a great way to learn things and possibly spawn new collaborations.
One effective way to engage with the community is to attend research talks, which are likely to be hosted by your department or institution (or even by other institutions online). Even though it might seem like the topic of a talk is not super relevant to your research and you really wanted to finish coding up some experiment instead, most of the time it is probably still a good idea to attend the talk. You will almost surely learn something, even if it is just some insight on how to give a good talk (or what to avoid to not give a bad one).
As you mature throughout your PhD, you will probably acquire some experience that can be useful to more junior members of the community. If a younger student reaches out to you, be generous with your time and try to help them. Remember: Most people are smart in academia; distinguish yourself by being kind.
Another way to interact with new people and learn a lot of things is to do research outside your normal group for a while. You can do this either by visiting another research group at a different institution or by doing a research internship in the industry. You should of course discuss these options with your advisor and the availability may differ depending on your home institution and stage in the PhD. Generally, I can wholeheartedly recommend both of these activities, but you will probably gain more from them a bit later in your PhD (i.e., not in the first year).
Finally, you should definitely set up a personal website for yourself as early as possible. This will make it much easier for people to find out about your research and get in touch with you. I used GitHub Pages for this, but there are many different options that are easy and free. Similarly, it is usually a good idea to have a Twitter account. While there are many discussions on that platform that are probably not worth following, and you should avoid getting sucked into it too much, many academics use Twitter to promote their recent work and it can be a great way to get more people to read your latest paper.
Teaching and supervising students
If you do your PhD at an institution of higher education (e.g., a university), there is often the opportunity (or even the requirement) to participate in the teaching efforts as a teaching assistant. This might seem like an annoying chore, which takes away time from your research, but it can be a great opportunity for you to learn something more deeply. Whenever I was supposed to teach something I was sure I understood well, I always noticed during the teaching preparation that there were actually details I didn’t fully understand and needed to read up on. When only studying a topic for yourself, it’s easy to fool yourself into believing you understood it, but when you need to coherently explain it to someone else, it becomes apparent what you do and don’t understand. I would therefore strongly encourage you to take part in teaching whenever you get the chance. It is also a great way to get to know younger students, which you can then potentially supervise.
At the beginning of your PhD, you will likely still require a lot of guidance and supervision yourself, so the prospect of supervising students might seem daunting. But after you’ve finished a few research projects, you will quickly realize that you might have follow-up ideas for those projects, for which you just don’t have enough time yourself. Those are usually great projects to give to younger students (e.g., for their bachelor’s or master’s theses), since they are well-defined and you will typically have the project-specific experience to guide them effectively through the research process. Supervising students thus (or co-supervising them with someone else) is not only a great way to get some help with your research, but (together with the teaching described above) is also an important part of the job of an academic. So getting some experience in teaching and supervision and figuring out whether you enjoy those things can play a big role in deciding later on whether you would like to try to stay in academia after your PhD or rather seek an industry position.
Project and time management
The great thing about a PhD is that you are basically your own boss. The awful thing about a PhD is that you are basically your own boss. So on one hand, you have a lot of freedom and can deeply focus on challenging questions that excite you, but on the other hand, you will have to manage your own time. This can be daunting at first, especially if you’ve been used to externally defined schedules for your whole life.
My first and foremost recommendation would be to maintain a healthy work/life balance. That doesn’t necessarily mean that you should never work on weekends (sometimes you might want to get some paper finished before a conference deadline) or that you should leave the office every day at 5 pm. But it does mean that you should allow for ample time off for leisure and relaxation. When it comes to your focus and creativity, as with physical exercise, the resting periods can be as important as the active ones.
I have spent many nights frantically and unsuccessfully trying to debug some code, just to return the next morning and spot the bug within five minutes. Sometimes your brain just needs the time off. Also, breaks can help you gain some perspective. Instead of getting lost in the weeds of your research, you should regularly take a step back and ask yourself “Is this the most useful experiment I could be doing for this project right now?” or “Is this the most useful project I could work on right now?”. Don’t be afraid to answer this question in the negative and then switch what you’re doing. Often, it can also help to have a little side-project (or two) to work on when the going gets tough in your main project and you just get too frustrated to work productively on it. That being said, you should probably try not to have too many projects in parallel. A good rule of thumb that worked for me was to have one main project (which I would spend at least half of my time on), one or two side projects (which would take up to a day per week each), and a few smaller collaborations (which take at most 2 hours per week each).
The most powerful tool you can use to structure your work and plan your time more efficiently is a calendar. Many people have noted before that calendars are strictly superior to to-do lists, and you should probably read Devi Parikh’s post on the matter. The main argument is that you actually live in a physical world in which time (which is reflected by your calendar) is one of the constrained dimensions, as opposed to a world where tasks are. The length of your to-do list is unbounded, whereas your day always just has 24 hours. So instead of putting tasks on your list, put them in your calendar.
Now, you might have noticed that this incurs an additional problem: while your to-do list was happy to just contain a description of the task, your calendar also wants to know how long it will take you to perform it. There is no secret trick for estimating this, and your estimates will likely be quite off in the beginning, but what I found helpful for certain tasks was to use the Pomodoro technique. It boils down to having you estimate how many blocks of 25 minutes (pomodori) you will need to perform a task, and then schedule that in your calendar. If you’re not done in the end, estimate the number of blocks you need for the remainder and schedule that for a later time. If you keep track of how many blocks you actually needed in the end for different tasks, you will soon find that your past durations are surprisingly predictive for future tasks. So you will ideally end up with a little look-up table of how many pomodori you need for different tasks.
The main trick to making this work reliably is to define the tasks in an atomic way. Don’t define a task as “write the paper X”, but define it as “draft an outline for the paper X”, “plot the figure for experiment Y”, or “write section 2.1 of the paper Z”. After doing this for a while, you will likely get proficient enough to automatically block the right amount of time in your calendar for the different tasks and it will help you to plan ahead properly and start different stages of the project on time, without having to rush before the deadline. Trust me, just the feeling of having your polished paper submitted well before the deadline, while your colleagues are still scrambling to get their last experiments in, is well worth it.
Finally, it can also be of tremendous help to not only keep a forward-looking calendar, but a backward-looking research diary where you log your progress. This will help you when you get lost among all the experiments you have already done or want to revisit some old idea you had at the beginning of the project in light of some new insights. Almost more importantly, it will help you when you feel like you haven’t achieved much in the last week (or month, or semester) and you can review all the things you actually did, which will usually turn out to be much more than you had remembered.
Writing a paper
So now that you have done some research and hopefully found some interesting results, it will be time to write them up into a paper! Many students wrongfully believe that their job as a researcher is to do research. It is not, in fact! Your main job as a researcher, which is also the reason why as an academic you are usually paid for it by tax money, is to communicate interesting novel research results to the world, so that they may help others. This necessarily presupposes that you do some interesting novel research in the first place, but it also means that if you don’t communicate your results, you might as well have just not done the research at all. This all is just to say: the world (and academia) doesn’t usually care about your experiments (and all the friends you made along the way), but it only cares about your publications.
While this may sound cynical at first, it also reveals a great opportunity: if your research journey was confusing, arduous, and convoluted, you now get a second chance to tell that story in a more straightforward way. You can spare other researchers the confusions you had to go through and present them with the shortest and most direct path to your conclusions. And, to put it in Charles Sutton’s words, you are granted “the privilege to participate in the worldwide, centuries-long conversation of the community of scholars”. If that isn’t something to be proud of, I don’t know what is.
Now, to get into the actual matter of how to write a good paper, I usually recommend Simon Peyton-Jones’s talk on the subject (he also has a great one on how to give a good talk, by the way). One main idea there is to come up with a story early and then plan the required experiments accordingly. Otherwise, it can happen way too easily that while writing the paper, you realize that half of your experiments are not really useful and a lot of potentially useful ones are missing. When drafting the story, I like to imagine how I would try to tell a colleague about this research project in front of a whiteboard. The order of things written on the imaginary whiteboard should then probably also be the order in the paper. Another great way to structure your paper (and also overcome writer’s block) is to write a rough outline first, in terms of some bullet points. You can then iterate on that outline with your co-authors until you all agree the story makes sense. Once you have that, you can just go ahead and, one by one, turn each bullet point into a paragraph. And before you know it, you’ll have a paper draft!
To make sure you end up with a polished and readable paper, you must hone your writing skills. While it may seem unfair that a major factor contributing to your paper’s chances for publication is your own command of the English language (especially if it isn’t your native language), English has firmly established itself as the scientific lingua franca, so we all have to deal with that for now. A good start on your journey towards better writing skills is to read the Elements of Style, and then probably to re-read it every now and then. Apart from that, some universities offer scientific writing classes, which you should definitely consider taking. Of course, you don’t have to take full responsibility for the paper’s writing alone, and you should make sure all the co-authors have at least read the final draft and have had a chance to give you feedback on the writing.
Speaking of co-authors, if you took the advice above to heart (the one regarding talking to many people), you will likely have some potential co-authors on your paper. When it comes to deciding whom to include, I would generally recommend erring on the side of inclusivity. Of course, you shouldn’t invite anyone to be a co-author if they have not contributed anything to the paper, but most meaningful contributions (advice on the project, help with the coding, analysis of the data, help with the writing, etc.) should warrant co-authorship. At the same time, people who only made small contributions might decline your invitation as co-authors, and that is also okay. You should make sure to discuss the authorship order well ahead of time, and not only on the last day before the deadline. If you want to make it clearer to readers who contributed what, consider writing an author contribution statement (e.g., using the CRediT system), which is encouraged by many journals and conferences these days.
Moreover, as I hinted at above, make sure to generously cite all the related work which is relevant to your paper. Again, err on the side of inclusivity. Many brilliant people get too little recognition for the research they are doing because their papers are somewhat overlooked by follow-up work, so give credit where it is due. It won’t diminish the value of your own contributions and will signal to the readers that you have a good knowledge of your research area.
Finally, once you have submitted your manuscript, consider making a preprint available on the arXiv. You should of course coordinate that with your co-authors and make sure that the venue you submitted to allows it, but it is usually a good idea, especially if you feel like your paper is in a nice and polished state (which it hopefully is, otherwise you probably shouldn’t have submitted it in the first place).
Using LaTeX
If you are writing a paper in any technical discipline, you are probably going to use LaTeX (or more specifically, LaTeX2e) to typeset your document. While most conferences provide you with a LaTeX template for your paper, there are still many subtleties to consider to improve your writing experience.
Generally, you will need to choose an editor for your LaTeX document. While some purists prefer to use a bare-bones text editor like vim, there are many alternatives with more appealing interfaces. For offline editing, I personally like Texpad (which is not free though). If you work on your paper with several co-authors, you might not want to use an offline editor anyway, but opt for a collaborative online editor. The most popular one these days is Overleaf. While it has some annoying little issues compared to offline editors, it does the job surprisingly well overall. It is also free in general, but you might want to convince your institution to get a Pro license for additional features. If you’ve never used LaTeX before, Overleaf also offers a nice short tutorial.
One of the main advantages of LaTeX is that you can make global changes to certain parts of the document quite easily. To use this power most effectively, you should get into the habit of defining macros. For instance, you can define certain mathematical notation as a macro, let’s say define \params to map to \theta if you want to denote the parameters of your model with the Greek letter θ. You then write all your equations using the \params macro and if you later decide that you actually rather would want the params to be called w, you just change the one line with the definition of the macro to be \mathbf{w} instead and it will change all the occurrences in all your equations automatically! If you’re overwhelmed with defining your own maths notation, you can also adopt the one from Ian Goodfellow’s Deep Learning book, which he has conveniently open-sourced. Another useful macro is for the name of your method. You can define \ourmethod to map to some preliminary name while writing the paper and if you later come up with a catchier name, you can easily update it.
LaTeX also offers the ability to use third-party packages in your document. One of the most convenient ones is cleveref. It makes it easy to refer to figures, tables, sections, equations, and anything else you have labeled with a \label{LABEL} command, using the simple command \cref{LABEL}, which then automatically expands to the correct reference in the text, following your defined preferences on abbreviation and capitalization.
One of the main headaches for new LaTeX users is bibliography management. While there are confusingly many different tools for this, the one that is mainly used in computer science publications is natbib. Generally, natbib requires a .bib file with BibTeX entries. You can get these entries for the papers you want to cite from Google Scholar, but there are often little mistakes, so make sure to check them carefully. A somewhat more reliable source for BibTeX entries (at least for computer science publications) is dblp. Once you have an entry in your .bib file for a paper that you want to cite, say the key is author2022paper, you have two major ways of citing it: within the text or as a parenthesis. You can set the within-text citation as “Following \citet{author2022paper}, we do …” and it will render as “Following Author et al. (2022), we do …”. The parenthetical citation is set as “… this has been shown in prior work \citep{author2022paper}” and renders as “… this has been shown in prior work (Author et al. 2022)”. Make sure to not confuse the two types, as it can be a bit confusing for the reader and impair the flow.
Finally, make sure again that you do not include any Type 3 fonts (mainly by stopping matplotlib from using them, see above). You can check this for instance by looking at the document properties of your rendered PDF in Acrobat Reader. For the initial submission of the manuscript, it might not be a problem, but many conferences and journals will not accept your camera-ready version if it includes Type 3 fonts and at that point, it will usually be much more annoying to remove them (by replotting the figures) than just avoiding them from the get-go.
Writing a rebuttal
Once you have submitted your paper to some venue, it will usually be sent out for peer review, to then return to you a few weeks (or months, in the case of some journals) later with the reviews. You will then usually have the chance to write a rebuttal, that is, a response to the reviewers.
First of all, you will probably be disappointed with some reviews. That is perfectly normal. After all, you spent a lot of time on this project and a lot of work on writing a nice paper, and then there are these reviewers who sometimes just don’t get it and ask silly questions and make unfair comments. Remember to not take that personally; the main problem is usually that the reviewers just didn’t have enough time to understand your paper properly. In the end, that is also not really their fault either, it’s just that the current system of peer review in machine learning is somewhat suboptimal in certain ways. There are initiatives to improve it, but for now, we’ll all still have to deal with that.
Several people have written about how to write a good rebuttal (e.g., Devi Parikh). The key is to understand that everyone involved is more or less doing this voluntarily. The reviewers and area chairs deciding about your paper are typically not paid to do this and volunteer to do so on their own time. So at the very least, you should be grateful for the time they are spending on your paper. Given that their time is limited, they will often overlook or misunderstand things. You should point that out to them in a friendly and respectful way. If several reviewers have the same misunderstanding, you should also consider that your paper might have some potential for clarification there. It helps to pretend that the anonymous reviewer is a friendly colleague of yours, whom you have given the paper draft for feedback. If the review reads particularly negatively, imagine the reviewer is a friendly colleague of yours who just happened to have a really bad day when they wrote the review. Chances are, this is not too far from the truth in the end.
Again, due to the time constraints of anyone involved, you should write your rebuttal as concisely as possible. You should briefly thank the reviewers for their time and then efficiently point out the main misunderstandings. Feel free to use some form of highlighting if technically feasible, so that the area chair can get a quick idea of your rebuttal without reading it in detail.
Even with the best rebuttal, your paper can still have a high chance of being rejected. This is mostly because the major conferences have decided to reject a lot of papers (around 80% of them, usually) and because the review system is inherently random. So don’t take a rejection to heart, just use the feedback you got from the reviewers to improve the paper and submit it to the next conference. To use a baseball metaphor I once heard, a good amateur player might have a batting average of 0.2, whereas a Major League player will be around 0.3. So if the average conference paper has an acceptance probability of 20% and your papers get accepted 30% of the time, that is something to be quite proud of, even though more than two thirds of your papers will still get rejected. Alternatively, you can consider submitting to a journal like TMLR, where the reviewers are less focused on novelty and state-of-the-art performance and the acceptance rates are higher.
Visiting a conference
If your paper gets accepted to a conference, you should usually get a chance to visit the conference and present it there. These days, many conferences are happening virtually, making the experience a bit less exciting, but they are slowly coming back to happening in person. This means normally, that you will travel to some interesting place somewhere in the world and spend a week in a large conference center full of other researchers. If your advisor or institution cannot fund your travels, you should be aware that most conferences offer travel awards to students and there are also many external travel grants, but you will need to apply well in advance for those. So make sure to discuss that early enough with your advisor.
Once you’re at the conference, there are plenty of talks, poster sessions, workshops, tutorials, coffee breaks, and parties, so it can be quite overwhelming. Keep in mind that nobody can ever take part in every event of the conference, so don’t overdo it. It is perfectly reasonable to take a break now and then and just stroll through the city to do some sightseeing or grab a coffee. In terms of the actual conference program, try to make yourself a schedule with things you don’t want to miss. As a general list of priorities, I would recommend: networking events, workshops, poster sessions, tutorials, and talks.
Networking events are probably the most valuable part of every conference (and sadly also the part that is hardest to emulate in a virtual one). They offer you the chance to meet other PhD students that might be willing to collaborate with you, industrial researchers that can possibly host you for internships, or senior academics that you may be able to visit for a research stay. Moreover, towards the end of your PhD, you might be able to find a postdoc or industrial position as your next job.
Workshops are nice for mostly the same reasons because they are basically networking events with a narrower subject focus and with a few talks and posters. They offer a great way to meet people from your particular research area and see some preliminary research that is often still a work in progress. Again, they are great to find new collaborators.
Main conference poster sessions are often really crowded and can be overwhelming and nauseating if you’re not prepared well. You should pick a few posters that you actually care about in advance and then maybe even skim those papers beforehand. At the actual session, you can then go to those posters directly and hopefully have a fruitful discussion with the presenter, asking all of your questions. If you have chosen posters that are relevant to your own research area, you may even learn some things that are not written in the paper and you can possibly also find new collaborators that way. If you feel like the poster presenter is a bit too stressed for an in-depth discussion and there are too many other people around, feel free to suggest to them to grab a coffee or lunch some other day of the conference.
Tutorials are often nice to learn about some area that you’re not really familiar with, but they are usually also available online afterward, so feel free to skip them if you have anything better to do.
Finally, talks at conferences are unfortunately often not super useful to attend. This is not necessarily the presenters’ fault (although it sometimes can be), but mostly just because they need to prepare a single talk to convey their research to an audience whose constituents range from experts in that particular subfield to first-year students in an entirely different area. I have personally found that I can often learn more about someone’s research from skimming a paper for 15 minutes than from listening to them explain it in 15 minutes, so I usually skip most conference talks. It is also often a good time to have a more relaxed discussion with other attendees in the coffee hall, which is much less crowded during the talk sessions.
Overall, just try to keep a good balance and have a fun time. And if you meet anyone whose research you liked, do remember to reach out to them after the conference and try to stay in touch. You never know where that may lead!
Writing a review
Once you’ve been through a few cycles of submitting and resubmitting papers and writing rebuttals, and hopefully got a few papers published, conferences will start to ask you to serve as a reviewer. This is an exciting opportunity to give back to the community! A good rule of thumb is that you should probably review for a conference at least as often as you have submitted papers to it, otherwise the overall peer review system will not be sustainable. That being said, if you have other critical commitments (e.g., writing up your thesis), it is also completely understandable to decline a reviewer invitation every now and then.
Once you’ve accepted the reviewing invitation, you often get asked to bid on interesting papers you’d like to review. This is a nice way to get an overview of what other people in your field are doing, so plan for some time to actually look at the papers. You should bid for anything that you would find interesting to read and anything that you think you are specifically qualified to review.
Before you then start reviewing your assigned papers, you should look at the reviewer guidelines of the specific venue. You can also look at the ones from other venues (for instance the ICML one), which have some good general guidance on reviewing. Moreover, you should look at the review form to know what you will be expected to write in the end.
When actually reviewing the paper, try again to pretend it’s a draft that you got from a friendly colleague, who is asking for your opinion. Regardless of how terrible you think the paper is, try to stay friendly and respectful in your review. Also, don’t necessarily ask yourself how you would have approached the problem and then criticize the authors for not having done the same. Just try to assess whether the way they chose is scientifically valid and whether their conclusions are supported by their evidence. Finally, the venue will want you to gauge how likely it is that the community would benefit from having this paper published. Try to decide that without relying too much on the “novelty” or “performance” of the research, but rather ask yourself whether you could imagine that a significant subset of attendees at the conference (or readers of the journal) could learn something useful from the paper. If so, you should probably recommend acceptance.
If you are going to recommend rejection, make sure to give the authors clear guidance on what they would need to improve for you to recommend acceptance. If these are minor improvements, they can sometimes even be made within the rebuttal period, so be prepared to reassess your evaluation afterwards. Also, consider the possibility that you have missed or misunderstood something (that happens often enough), so be open to having the authors point that out to you. Finally, do take part in the discussion with the other reviewers, since they might have caught (positive or negative) aspects of the paper that you haven’t.
This was Part 2 of the article series. You find links to the other parts of this series at the top. If you want to get in touch with me, follow me on Twitter or use the contact form on my website.
