SIGIR and plagiarism: an open letter
Dear IR community,
Recently an anonymous Reddit user posted a thread on /r/MachineLearning accusing a SIGIR 2019 paper of plagiarizing a RecSys 2018 paper. I am writing this letter because I want to explain the steps we take to prevent plagiarism, how we work with the ACM to address plagiarism claims, and finally what we have been doing in this particular case.
The SIGIR conference is run by the ACM SIGIR group, and ACM SIGIR is under the US-based non-profit Association for Computing Machinery (ACM). SIGIR must abide by all the policies and procedures of the ACM, as the ACM is the organization that is legally liable for any action SIGIR takes or anything that happens at a SIGIR conference.
We do not tolerate plagiarism
We want to be totally clear on this: SIGIR does not tolerate plagiarism, falsification, or misrepresentation. The SIG and the ACM have a number of policies and procedures to catch and respond to incidents, as described in detail here:
https://www.acm.org/publications/policies/plagiarism-overview
Conferences have pre-publication checks to catch infractions, including using iThenticate to compare submissions to each other and to published papers. Conferences with overlapping review periods coordinate with each other, even across the bounds of the ACM, to identify possible cases in submitted papers. Peer review serves as another check. There is another check on misrepresentation after paper acceptance but before final publication, as authors can make changes up until the final version being due.
When a paper is flagged during the pre-publication timeline, it is subject to immediate removal from the submission pool. The conference organizers have full discretion over this process. They can also work with the ACM and other organizations to ensure that the authors face penalties. And authors most certainly have faced penalties for infractions in the past. Again, we take this very seriously.
Nevertheless, a paper will sometimes get through. It’s an IR problem, so of course it faces precision-recall tradeoffs! When a paper does get through the pre-publication checks, is recommended for acceptance by peer reviewers, passes post-acceptance checks, and finally appears in print, then the conference organizers are seriously constrained in what they can do. In fact, the only formal action they can take is filing a claim with the ACM.
And they do: the ACM handles hundreds of plagiarism claims each year. The ACM has the expertise and experience to handle these claims in an unbiased way. But this takes time. Authors of papers that reach this stage must be given due process: the ACM policy defines five levels of seriousness along with penalties for each level, and gives the accused the right of appeal. It is extremely important to respect this process.
We do not engage with public anonymous accusations
Plagiarism is a very serious accusation. It can damage or destroy careers. It is not an accusation that should be made lightly nor disputed publicly. Unfortunately, it is all too easy to make damaging accusations from an entirely risk-free position, with a free anonymous/pseudonymous account on an online social media platform.
We do not engage with public, anonymous accusations. We cannot stop people from making them, but we do not respond to them.
In fact, the ACM policy requires that the claimant provide their real name. It also provides assurances of confidentiality, detailed in the document linked above. There are many good reasons one might not want to reveal their identity even with those assurances. Anonymity can be an important protection, but it does negate the ability to file a formal claim. But there is an answer to this that does not require a public accusation: talk to people privately. Talk to local colleagues and seniors; perhaps you will find someone willing to submit a formal claim. If not, and you remain sure of the case, start expanding outside your local circles — but privately. Reach out to the PC chairs of the conference. Reach out to the SIG executive committee. If all else fails, reach out to the ACM and explain why you prefer to remain anonymous — the ACM also investigates ethics issues which may be driving a desire to remain anonymous. But there is never a good reason to level a public accusation.
What we have done
We have not been idle. In the spirit of the paragraph above, most of our response has been private. We have:
1. re-run pre-publication checks. We verified that this paper would not have triggered any pre-publication plagiarism alert.
2. reviewed the papers ourselves. This was to identify the portions of the paper we might want to flag to the ACM, who will not accept the anonymous report.
3. posted in the Reddit thread to inform others of the official policy and procedure, and to invite anyone who so wishes to file a claim.
4. asked the Reddit forum moderators to close the thread. The Reddit forum has community standards, which this thread does not appear to meet. As of this writing the thread has not been closed.
5. spoke in person to the co-chair of the ACM Publications Board. This is the group that handles plagiarism claims. He reiterated that the only formal action we can take is to submit a claim.
6. filed a formal claim. I am in contact with the ACM about it, but I cannot talk about the details.
What we cannot do
We cannot say whether plagiarism occurred or who is responsible. Since the paper is published, that is not for us to decide. The ACM policy is very clear on this. The reason for this policy is clear as well: if what we say, or any actions we take based on our belief, conflict with the ACM’s ruling, the ACM is open to liability. Furthermore, even if we wanted to make a statement, we are not in a position to be unbiased towards respected members of our community.
We cannot take the paper down while the case is being adjudicated. It has been published by the ACM. Only the ACM can take it down.
We cannot stop people from discussing the accusation anonymously in public forums. But we do strongly recommend not participating in such discussions, as participation lends credence to the manner in which the accusation was made.
We cannot stop people from making anonymous public accusations in the future. But we do not respond to such accusations as if they are legitimate.
Finally, some perspective
SIGIR and the ACM care deeply about plagiarism and all kinds of misrepresentation. They are growing problems across dozens of ACM SIGs and hundreds of conferences. Ensuring that the right decisions are made takes time. It never hurts to just take a step back and wait.
Thank you all for reading, and I hope to see you soon at one of our upcoming ACM SIGIR sponsored or co-sponsored events in 2020: AFIRM in Cape Town, WSDM in Houston, CHIIR in Vancouver, JCDL in Wuhan, SIGIR in Xi’an, ICTIR in Stavanger, and CIKM in Galway!
Ben Carterette, Chair, ACM SIGIR
