Optimizing the research process

Paper deadlines often feel like a mad dash after the bus while my pants are falling down. After a recent submission, with the usual crunch of experiments and down-to-the-wire drafts, I’ve been thinking, is there a better way to do research?

Let’s assume that you’ve got a proposed problem, motivation, and starting idea. You will be tempted to immediately implement and test your idea, but don’t. Instead, front-load related work experiments, to better motivate and define your approach and move that computational load well ahead of deadline. Also, perform continuous ablation experiments to systematically validate your ideas and produce usable results for the final paper. This process is recommended for both academic research and R&D in commercial ventures.

1. Define experiments: What benchmarks and metrics will best define success? What will be used for validation/ablations and for final results?
2. Find applicable works: If you had to solve your problem with existing works or slight variations on them, which ones would you try?
3. Evaluate applicable works: For each, evaluate on all benchmarks/metrics and find representative qualitative examples.
4. Identify strong and weak points of existing work: What are the addressable open problems?
5. Define and evaluate your baseline: The baseline is a working and testable system that excludes your key ideas but can be modified to incorporate them.
6. Refine and implement your proposed solution(s): Every time you incorporate an idea or tune a parameter, record the experimental results on validation/ablation sets.
7. Evaluate your complete system on all benchmarks/metrics.

While not purely sequential, each step should at least be started before working on the next. While doing all of this, maintain three key documents to track research progress and prepare for the paper:

1. Online quantitative results sheet
2. Online qualitative results slides
3. Running draft of paper to organize method, key results, and findings

If you follow this process, you will make more stable progress, and writing the paper will be like a stroll in the park. Comment with your own tips of how to manage the research process.