Lessons learned from a PhD in Machine Learning
Part 1: How to get in
This is Part 1 in the series. You can find Part 2 here and Part 3 here.
At the time of this writing, I have recently finished my PhD in machine learning at ETH Zürich. In retrospect, I have the subjective feeling that I’ve learned a thing or two over the last four years, which I intend to share with you in this series of posts. I would expect this guidance to be most useful for prospective PhD students in machine learning in Europe, but some of the aspects might also transfer to other disciplines and regions. As with all advice on the internet, you should feel free to heed any, all, or none of it. I should also mention that many other smart people have written similar documents before. Some that you might want to look at include the ones by Andrej Karpathy, Sebastian Ruder, Diana Cai, and Jason Hong’s collection of other pieces of advice. Finally, this writing does not assert any claim of completeness; it is merely a collection of assorted musings I found worth sharing.
Should you do a PhD?
The obvious first question to answer is whether you should try to pursue a PhD in the first place. I believe that this mainly depends on your motivation. Does doing open-ended research for a few years and becoming an expert in some narrow scientific niche sound fun to you? Then you should definitely go for it. On the other hand, if your motivation to do a PhD is mainly due to getting some fancy title, getting better jobs later on, making your family happy, following your friends’ examples, or any similarly extrinsic factors, you should probably not do it. Doing a PhD is already hard enough, and can be rather frustrating at times, so if you’re not driven by some intrinsic motivation for research, you will probably not enjoy it very much.
Another way to put this is: Do you rather enjoy developing useful products/services or understanding phenomena very deeply? In the former case, industry might be a better place for you and your job prospects will likely not improve much with a PhD, in the latter, academia can be a great place and a PhD can open the access to research positions in industry for you. However, there is of course also a financial aspect and you should be aware that the PhD route will not be the most economically rewarding (at least in the short term). So if you feel like you would not enjoy living a student lifestyle for a few more years and you have some things that you would like to spend more money on, this can also be a valid reason against doing a PhD.
That being said, I really enjoyed my PhD and I think it was one of the best times in my life. I would encourage you to do some research in academia before making your decision, either through a thesis project at your university or a research internship at another institution. Ideally, you should also try to do an internship in the industry, to get a feeling of what an alternative career could look like. If you then feel more excited about the academic research and are keen to do more of that, you should definitely go for the PhD.
Finding the right group
Your first choice to make if you want to do a PhD is the choice of the group. This can seem rather daunting, because you may have many (potentially conflicting) preferences regarding the advisor, the institution, and the geographical location.
Regarding the institution, it is of course true that all else being equal, a more renowned institution might look better on your CV than a less renowned one. That being said, all else is usually not equal and the institution is probably the least impactful of those three factors on your experience and wellbeing, so I would largely ignore it initially and only use this criterion to break ties. Note, however, that I am mostly thinking about research careers here, whereas for a career in the industry, the institution on your CV might matter more.
When it comes to the location, this can actually have a stronger impact than you might think, and it’s usually worth considering. You will spend the next few years of your life there, so if you don’t like the place, no advisor in the world can make up for that. However, I can’t help you much with the choice of location; you’ll have to ideally visit and talk to people there and see how you feel about the place.
Regarding the advisor, this is the most strategic choice and will have the strongest impact on your research. The first broad decision should be whether you rather want a “hands-on” or a “hands-off” advisor. This depends mostly on your preferred mode of working, but also on the amount of your research experience and determination of your research interests. A hands-on advisor will probably meet with you between once and several times a week and will be closely involved with your research. Typically, these are more junior advisors with small groups, who still have many own research ideas that they will want you to work on. This is good, if you don’t yet have much research experience yourself and/or don’t have any strong idea of what you would like to work on. In contrast, a hands-off advisor will meet you every few weeks to months and will generally not get too involved with your research. Typically, these are more senior advisors with larger groups who give you more freedom to pursue your own ideas. This is good if you already have some research experience and ideas about what you would like to work on. That being said, all advisors naturally lie on a spectrum between these two extremes and you might also prefer some advisor who is more of a middle ground between the two. Note that, coming back to the previous point about institutions, you will generally find more hands-off advisors at bigger, more renowned institutions and more hands-on advisors at smaller ones, so this may also play a role in your institutional choice.
Once you have made a selection of potential advisors, you will first want to look at their most recent publications. Do you find them exciting? Would you have liked to be involved in any of this work? If not, they are probably not the right advisor for you. Next, you should reach out to the potential advisors and talk to them and, almost more importantly, talk to other students in their groups. The students will generally be able to tell you more about what the actual PhD experience in this group is like and they generally have no incentive to deceive you. Moreover, if there are postdocs in the group, you should also talk to them, since they might be able to co-advise you to some extent, depending on how busy the actual advisor is. If you have the option, the best is to visit the group in person and have a day where you meet all of them, before you make your final decision.
One thing to note is that of course, not all the groups you might like to work with will also have the resources (or intention) to hire you. So depending on your personal background and where you apply, you should be prepared to possibly get rejected. This is quite normal in academic careers and happens to the best. So you should not take those rejections to heart, but rather mitigate the risk by applying to several different places and also try to increase your chances of getting accepted. Avenues towards this typically include studying at a renowned university and getting good grades, but more importantly, getting research experience and getting to know people in the field. If you have the opportunity to work on a research project for your bachelor’s or master’s thesis where you are studying, you should take this rather seriously and try to at least get a small publication out of this (e.g., in the form of a workshop paper or a technical report on the arXiv). You can also try to get a research internship at an academic institution or a company, which might lead to a small publication or a co-authorship on a bigger one. Moreover, trying to do your thesis project or research internship in a group where you would also be interested to do a PhD can often significantly improve your chances of getting a position there. Overall, the better you get to know the people in your prospective group and the more you can convince them that you are able to do good research, the more likely it is that you will get an offer for a PhD position. Finally, you can apply for PhD fellowships (depending on your country and institution), which will ease the financial burden for your prospective advisor and thus also make it more likely for them to be able to hire you.
Finding a research topic
Now that you have found an advisor and started your PhD, you will need to find a research topic to work on for the next few years. Depending on how hands-on or hands-off your advisor is (see above), they will be more or less involved in this choice. A great timeless piece of advice on choosing research directions is this talk by Richard Hamming. The key is to read a lot of papers from a broad set of topics and talk to a lot of people to find out what are important open problems that might be worth working on. Mainly, you will need to find a problem that excites you, that ideally your advisor and group have some useful expertise in, that is hard enough to be worth working on for several years, but crucially easy enough for you personally to make some progress. The last point is often hard to gauge, but it means that for instance if you’re personally terrible at maths, you shouldn’t pick a complex theory problem to work on, just because it’s an important one. Rather, try to pick a problem where you feel like you might have a competitive advantage compared to other students. That being said, your skills can of course develop over the course of your PhD and you can collaborate with other people with complementary skills to yours, so don’t get discouraged if you don’t yet feel 100% prepared for all facets of your chosen project. In fact, you should collaborate with other people with complementary skills to yours: they’ll make your research life much nicer and more productive (and hopefully you’ll make theirs).
Reviewing the literature
Once you have chosen a project, you should carefully review the relevant literature. While previously, in the project-finding phase, your goal was to read broadly, now you should read in-depth. In particular, this usually means that you should start with textbook introductions, then read review papers, then seminal research papers from the past, and finally the most recent research papers from the last few years. To build your reading list, it can also help to follow relevant citations in the papers that you read, or use Google Scholar to find follow-up papers that cite the one you are reading. Especially if you already have a specific idea to work on, carefully look through the research in that sub-area. Nothing is more annoying than starting to work on a great idea and then finding out later that it has already been published, or worse yet, having that pointed out to you by a reviewer of your paper.
The key skill to develop in this context is how to efficiently skim a paper. Regardless of your research area, there will likely be more papers there than you could ever hope to read in detail. So you should read them in different levels of detail depending on how relevant they are for your particular project. You will want to read the abstract for all of them, briefly skim over the main results and figures for many, read the methods and experiments sections for some, and read the whole paper in detail for only a few of them. While reading, you should keep some notes on all the papers and ideally organize them using a reference management software. I use PaperPile, but there are also many other (cheaper) options (e.g., Zotero).
This was Part 1 of the article series. You find links to the other parts of this series at the top. If you want to get in touch with me, follow me on Twitter or use the contact form on my website.
