What is wrong with data challenges

Networking at the first CDS RAMP in January 2015, with HiggsML winner Gábor Melis

Gábor Melis won the HiggsML Kaggle challenge last year. He gave a talk at the follow up NIPS workshop, expanding on his winning solution, and wrote a paper in the subsequent proceedings. We learned a great deal from him about winning challenges. He also gave us his code, deep neural nets written in LISP, which we could never compile, not even after working with him for a couple of weeks in January.

By the way, he was hired by Google Deep Mind this summer.

This story is a perfect illustration of what data challenges are for. For the participants it is an extended job interview. Winning, but even finishing in the top 1% in any of the challenges, means guaranteed job offers from your dream employers. For the organizers, public or private, it is publicity, demonstrating both coolness and openness.

What you should not expect from organizing a challenge?

  1. It will not deliver a workable solution to your problem, not even a prototype.
  2. You will have no access to the data scientists participating in the challenge, unless of course you can hire them.

What about the larger community? Data scientists will profit from getting to know interesting domain problems, both from domain sciences and industry, although most of the times challengeable problems are diluted or overly abstract versions of the real task. They can also benchmark their novel techniques in a fair experimental environment. Novice data scientists can also learn an amazing quantity of practical tricks from the forums, not described in textbooks, rarely taught in classrooms. Yet:

  1. Challenges are, by design, competitive. They don’t simulate either the industrial workplace or research, let alone the open source world, where collaboration is the ruling paradigm.
  2. Creative ideas rarely bubble up. The winning technology has been a set of tricks, interesting if you never heard about them, but rather stable since the mid 2000s.

People spend two to four months hammering out the details of their models, fine tuning the cross validation and blending schemes. Gábor’s solution was delicately designed, but there was nothing new in it. His 200 ideas didn’t work. Nor the novel ideas of anybody else. Challenges funnel everybody into a narrow set of standard practices. Deep into the leaderboard there are arguably tons of interesting things tried, which might improve the state of the art. They have no chance to bubble up without the sophisticated engineering mastery required to be in the top 1%. The only way to find anything creative is to painstakingly go through the forum and dig out the hidden treasures as my management scientist colleague, Akin Kazakçi did in this paper. The point is, none of these “side outcomes” (working prototype, code, creative ideas) are inherent parts of the current mechanism, they all need considerable additional work to be generated and formalized.

On Saturday, December 12, we are running a workshop on data challenges at NIPS with several panel discussions. Our main goal is to brainstorm about the formats and mechanisms. Open innovation has come up with an amazing breadth of schemes, including hackatons, fablabs, participative science, crowdsourced car building, crowdsourced hedge funds, coding contests, and matching seekers and solvers. What can we learn of these formats, and how can we adapt it to the data science field? What are the use cases? Publicity? Headhunting? Matching experts to problems? Training? Collaborating? Optimizing? Benchmarking? Innovating?

PS: By the way, I’m a huge fan of Kaggle. It is easy to criticize and throw ideas out there, it is another one to implement those ideas in a scalable business model. Funding crowdsourcing schemes is a major issue, and in the academic world we often overlook the constraints that come with it.