Comparing the Uncomparable to Claim the State of the Art: A Concerning Trend

Benjamin Marie
19 min readAug 16, 2022

Spotting evaluation errors and speculative claims in GPT-3, PaLM, and AlexaTM.

Step 1: put numbers in the magic hat ; step 2: take a magic wand to tap the hat ; step 3: state of the art go out of the hat ; Party smiley

In AI research, authors of scientific papers often choose to directly compare their own results with results published in previous work, assuming that these results are all comparable. In other words, researchers perform a simple copy of previous work’s…

--

--

Benjamin Marie

Ph.D, research scientist in NLP/AI. Medium "Top writer" in AI and Technology. Exclusive articles and all my AI notebooks on https://kaitchup.substack.com/