Inevitable downfall of a Kaggle Grandmaster who tried to win at any cost
On Jan 10, 2020, the Kaggle community was shocked by an announcement that a top Grandmaster (then ranked #3 out of more than 120-thousand competitors worldwide) was found to have cheated on a competition to win 1st place; his team was disqualified retroactively from the competition. It was also announced that this Grandmaster was permanently banned from the platform.
This was devastating and upsetting news to the Kaggle community. Being a Kaggle competitor myself, I sincerely hope that damages to the Kaggle brand will be minimal from this, and Kaggle continues to attract many organizations to host competitions on its the platform AND top-notch, world-class competitors who diligently and passionately work on them. I truly believe in Kaggle’s mission and the Kaggle community has had tremendous impact on myself and countless others. I would hate for this incident to deter Kaggle and the community from advancing AI forward.
Background
Back in December 2018, a Malaysian pet wellness site Petfinder.my hosted a Kaggle competition which attracted more than 2000 teams to compete over a 3-month period to work on the following noble goal of helping homeless pets by leveraging AI:
“In this competition you will be developing algorithms to predict the adoptability of pets — specifically, how quickly is a pet adopted? If successful, they will be adapted into AI tools that will guide shelters and rescuers around the world on improving their pet profiles’ appeal, reducing animal suffering and euthanization.”
Discovery of Cheating
This competition completed more than 9 months ago in March 2019. Why did it take this long to discover that the 1st place team was cheating?
In Kaggle competitions, the Kaggle staff and the host perform verification of the models submitted by the competitors. In this verification phase, cheats get discovered and get disqualified; this unfortunately happens from time to time. But the 1st place team passed this check and was awarded the prize money of $10K USD. The discovery came when Benjamin Minixhofe, one of top finishers in this competition, was volunteering to help Petfinder.my productionize the top teams’ models. He found that 1st place team’s model does not work well and he started investigating their code. It turns out that this was non-trivial as the cheat was specifically designed to evade discovery through obscurity.
Summary of Cheating
The competition host from Petfinder.my summarized the cheat as follows:
Here is what the Bestpetting team did in the PetFinder.my contest:
- They fraudulently obtained adoption speed answers for the private test data (possibly by scraping our website)
- These data and answers were then encoded, obfuscated and hashed into an ID field that was disguised as part of their external “cute-cats-and-dogs-from-pixabaycom” dataset
- When processing the data, these hashed IDs were decoded and answers were retrieved for the predictions
- Only some of the encoded answers were used, so as to keep their final score “realistic”
- These processing codes were meticulously hidden and obfuscated under many nested layers of functions and codes, intentionally designed to be highly unreadable and seemingly mundane
The summary is that they collected the answers to the problem by violating an explicitly-stated official competition rule to not use any data from Petfinder.my (see EXTERNAL DATA section), hid these answers, and used these answers only some of the time to boost their score. Benjamin stated in the Notebook that if they didn’t cheat, they would have placed around ~100th in the competition.
History of Cheating and Testing the Boundary
Google Landmark Retrieval 2019 Incident
It was not the first time that this now-banned Grandmaster was disqualified due to cheating. In the Google Landmark Retrieval 2019 competition (which I also competed in), his team placed 1st but later found to have cheated and disqualified. There is a thread where he and other competitors debate the practice of finding answers through scraping. In his words:
“totally agree, it’s not useful and not scientific at all, as well as poorly designed competitions
my logic is always the same: if there is a flaw or leak, somebody will definitely try to use it and if rules will allow it afterward then having an honest solution will be a disappointing disadvantage. so there are only two options left: not to participate or use the flaw”
“nevertheless, I haven’t seen a lot of leaks/flaws posted during the competitions — people are trying to win by any edge they could find, it’s a competition after all”
“no, what I’m trying to say: there is a flaw in competition design at Kaggle and there is already a solution to this problem by Google. until they fix it, we will always see solutions like mine and will never be sure about the rest of the LB (only winning solutions were checked)”
Quora Insincere Questions Incident
On the Quora Insincere Questions competition, he publicly disclosed how his team placed 1st on the public test set by simply scraping for answers from quora.com. This disclosure happened after the competition was already finished.
He claims that the reason his team did this was because they didn’t like the restrictions imposed by the competition rules, so they “decided to have fun”.
The technique did not work for the private test set which determines the final standing, so they ended up finishing 1023rd in the competition. Looking at the discussion thread, folks who responded to the thread were mostly about being impressed and praising their cleverness, and perhaps such positive reactions were encouraging and enabling. Had their method worked on the private dataset and placed them near the top, had the cheat been discovered? Maybe not, as we can now see from the Petfinder.my competition incident that post competition verification is not foolproof if the cheat is cleverly hidden. Or if the cheat could not be hidden due to the competition format (no external dataset), they might have tried to place well enough but not so well to stay under the radar. If they could have gotten away with it, I doubt he would have posted and disclosed the method but that’s just my guess. AFAIK there were no consequences from this incident as their entry was never removed from the leaderboard.
Aftermath
He was swiftly let go from his employer H2O.ai after getting permanently banned from Kaggle. H2O.ai has a number of Kaggle Grandmasters on its roster and advertise this heavily to demonstrate their expertise and establish trust. Employing a Grandmaster known to have cheated and banned would be a PR nightmare for the company. It must suck to be permanently banned from something he cared so much about and to lose a job at the same time but actions have consequences…
He publicly apologized via Twitter saying:
“1. We, as a team, would like to apologize to
@myPetFinder @kaggle and the DS community for all the wrongdoings. I would also like to apologize to @h2oai and all the Kaggle competitors for putting their reputation at risk.”
“2. We will return the prize money asap. For me, it was never about the money but rather about the Kaggle points: a constant struggle of becoming #1 in rating had compromised my judgment.”
“3. Now, after the ban, I feel almost relieved that the race is over, and there are still a lot of thoughts to process and reflect upon. I hope at least some of you forgive me and hope other competitors will learn from my mistakes.”
We are human and make mistakes, especially when we start chasing numbers (money, rank, likes, views, or whatever) and forget why we are doing the thing we do to begin with. I applaud him for publicly apologizing and I hope he makes sound choices going forward to avoid such mistakes.
Thoughts
There’s no question that this individual is smart, creative, and very determined to accomplish his goals. It is unfortunate that he deliberately chose to use his intelligence and creativity to improve his Kaggle world ranking even if it meant resorting to cheating and at the expense of others. I wish he could have applied his talent toward productive goals of developing models that work well to solve important problems or to focus on pushing the boundaries of the state of the art by developing new techniques, rather than treating Kaggle as a game to be won.
It is unfortunate that his cheating has had so much negative impact. For one, it prevented appropriate rewards to legitimate and hardworking competitors who were playing by the rules. For example, because of the disqualification, a competitor on the fringe got bumped up a place and earned his Grandmaster title that he should have received to begin with had cheating not taken place. I am sure there are many others who missed their in-the-money or gold medal by one spot. On a bigger scale, what he did was damaging to the reputation of Kaggle as a platform and the reputation/motivation of Kaggle competitors.
The Kaggle community attracts many smart people from around the world. It is not unusual for competitors to find exploitable aspects of some competitions. The common practice has been to report such exploits publicly and/or to the organizer for fairness to all participants as soon as they are discovered. I hope that this practice is followed by everyone going forward.
There will always be exploits even if much care and effort is taken to prevent exploits. However, there are ways to improve the system to be less exploitable and deter cheating. For example, we can allow top finishers to check each other’s solution after the competition for up to X days and have an appeal process (competitors would be VERY motivated to do thorough reviews to uncover cheating for those who finished above them). Or we can give everyone a chance to review fellow competitors code (at least for some type of competitions), as I have previously suggested on Kaggle.
I believe in Kaggle’s mission and future. I have benefited so much personally from competing, learning, and meeting new people, as did hundreds of thousands of others. I hope the ban of the high-profile Grandmaster sends the right message to maintain the integrity of the platform and that any negative impact is contained.
May the Kaggle community keep thriving to push AI forward!
