Leveraging ML to Fuel New Discoveries with the arXiv Dataset

Photo by Glenn Carstens-Peters on Unsplash

For nearly 30 years, arXiv has served the public and research communities by providing open access to scholarly articles, from the vast branches of physics to the many subdisciplines of computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics.

The sheer number of arXiv research papers is both beneficial and challenging. Whether it’s a graduate student ramping up in her respective field, an established professor delving into adjacent ones, or researchers searching for big picture insights for the public good, this rich corpus of information offers significant, but sometimes overwhelming, depth.

To help make the arXiv more accessible, we present a free, open pipeline on Kaggle to the machine-readable arXiv dataset: a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more.

“Having the entire arXiv corpus on Kaggle grows the potential of arXiv articles immensely,” said Eleonora Presani, arXiv Executive Director. “By offering the dataset on Kaggle we go beyond what humans can learn by reading all these articles and we make the data and information behind arXiv available to the public in a machine-readable format.”

Kaggle is a destination for data scientists and machine learning engineers seeking interesting datasets, public notebooks, and competitions. Researchers can utilize Kaggle’s extensive data exploration tools and easily share their relevant scripts and output with others.

“ArXiv is more than a repository of articles, it’s a platform for knowledge sharing,” said Presani. “This requires constant innovation on how we present and interpret the knowledge we make available. Kaggle users can help push the limits of this innovation and it can be a new outlet for our community to collaborate.”

“With large data sets, there is generally an expectation that there are likely to be overlooked discoveries, connections, innovative tools or perspectives, which may lead to additional insights, not just in the original subject, but across other fields of study, enabling yet more discovery and innovation,” said Steinn Sigurdsson, arXiv Scientific Director.

Call to Action

Our hope is to empower new use cases that can lead to the exploration of richer machine learning techniques that combine multi-modal features towards applications like trend analysis, paper recommender engines, category prediction, co-citation networks, knowledge graph construction and semantic search interfaces.

An example of such a semantic search application built on top of a specific corpus would be Google’s COVID-19 Research Explorer, a tool that helps researchers pore through the CORD-19 dataset — a repository of 190,000+ science articles on COVID-19. Interfaces built on datasets such as this leverage advanced NLU techniques to understand a user’s intent behind a query. Ultimately, this can enable more efficient research by surfacing relevant data and evidence to complex, scientific questions. We hope the release of the machine-readable arXiv dataset will inspire the creation of similar NLU tools over this new corpus.

Alex Alemi, Senior Research Scientist at Google, has also pursued more exciting applications of ML using the arXiv. As described in the paper, On the Use of arXiv as a Dataset , Alex and his colleagues sought to propel the arXiv as a benchmark for large scale, multi-relational tasks, such as with graph neural networks. “I’m excited to see the research community rise to the challenge of a rich, multifaceted dataset with so much real-world practicality, and the new questions this will raise,” says Alex.


The dataset is now available on Kaggle and will be updated weekly. Please send us your feedback and stay tuned for additional updates!


Special thanks to Paul Ginsparg, the noted physicist and visionary behind the arXiv, and Jack Hidary of X, The Moonshot Factory, who worked to bring this collaboration to fruition.

This dataset would not be possible without the helpful input of many people. Much appreciation to (in alphabetical order of last name): Alex Alemi, Timo Bozsolik, Alison Fromme, Peijen Lin, Brian Maltzan, Eleonora Presani, Steinn Sigurdsson and Joe Tricot.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Kaggle Team

Kaggle Team

Official authors of Kaggle winner’s interviews + more! Kaggle is the world’s largest community of data scientists. Join us at kaggle.com.