A Home For Results in ML
Rapid progress in machine learning has led to an exponential growth in the number of machine learning papers, with a new paper published every 15 minutes on average. This speed of development presents a challenge for our field: how do we stay up-to-date and filter through a growing mountain of papers and repositories?
Our goal at Papers With Code is to capture all the results of machine learning in a single place. A home for machine learning results makes it possible to stay connected with progress in the field without getting left behind. This means everyone — researchers, engineers and hobbyists — can benefit from the latest knowledge, improving accessibility and ultimately accelerating progress itself.
We took the first steps last year with our leaderboards feature. This was a community effort to collate all published results in ML in a single place, with a machine-readable format and a free license. We’re very grateful to the thousands of contributors who added results for their papers (and other papers) to create the world’s largest database of results in machine learning.
We want to go even further, so today we’re introducing several exciting new updates for Papers With Code:
- A New Results Interface that directly links results to tables in arXiv papers — to become a primary source for results in machine learning.
- An ML Extraction Algorithm that semi-automatically extracts results from papers — to achieve much higher precision than ever before.
- A Big Database Update — 800+ new leaderboards, 5500+ new results — to enable many more task comparisons between methods.
New Results Interface
One of the main requests of researchers for our leaderboards feature was more clarity about where results come from in the paper. So today we are rolling out our new results interface that directly links results to their original tables in arXiv papers.
To see how this works, visit an example leaderboard, such as the ImageNet leaderboard, and click the results icon on one of the rows. This will take you to the table in the paper where the result comes from, as the animation below illustrates:
The new interface also acts as our new paper result editor, where the community can add results from their own papers — and directly link them to the tables inside! This interface is currently only available for arXiv papers with LaTeX source. We’re looking for feedback on this new editing interface, so please give it a go and let us know your thoughts.
Automating Results Extraction
For the past year, we’ve been researching ways to automatically extract results from machine learning papers. Today we have a new human-in-the-loop system for results extraction in production. Our model generates proposals for every arXiv ML paper which a human can accept or reject. We’ve made this system practically viable by significantly increasing performance over the previous state-of-the-art. We believe this will improve results quality and coverage, allowing us to keep the community notified on new results in machine learning, even in niche and specialized subfields.
We’ve published our methodology on arXiv, and are open sourcing our pipeline on GitHub. We are also releasing a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of models on this task. We hope this helps stimulate interest in this machine learning task and is helpful for the community!
A Big Database Update
Our new interface and extraction model have allowed us to scale up the resource: as of today there are 800+ new leaderboards and 5500+ new results. This goes a long way towards achieving comprehensiveness — but our work is not done yet!
We call on all ML paper authors, engineers and hobbyists to submit results from both your own papers and any papers you read!
Our database is open and free for everyone to contribute to. All of the data is licensed under a free open data license — and you can download all data in JSON format here. Continued contributions by the community will keep this resource going, and improve accessibility and knowledge transfer in our field!
Summary and Next Steps
A few years ago, keeping track of progress in machine learning was hard. Thanks to the contributions of our community on Papers With Code, you can now type in a benchmark on Google and find the best methods in seconds:
We hope our new features lead to an even more comprehensive experience: so even niche areas of machine learning can have tools to better summarize progress and compare different methods. These features are live today! We invite you to browse our catalogue at paperswithcode.com/sota and use search to find your paper and add results!
It is worth quickly noting the limitations of leaderboards. Leaderboard metrics are often just point estimates, and numerous factors influence the final value — e.g. extra training data, training duration, and data augmentation choices. Additionally, dataset biases may mean leaderboard progress isn’t the best way to measure research progress. So we’ll go deeper later this year to enable even richer comparisons between machine learning methods - beyond simply what is the state-of-the-art for a given benchmark — to help researchers understand what generalizes and is working well.
Taken as a whole, we believe the changes released today are a big step towards having all ML results in one place. We hope the community finds it useful to have a place where results live, so they can more easily compare methods and understand which techniques are working in different areas of ML. Finally, we look forward to bringing you more powerful features like this in the near future to help you understand the world of ML research.
- The Papers With Code team
