![[PAPER] Major benchmarks are found to be polluted, with up to 12% of questions broken](https://external-preview.redd.it/dA5pwmkvUvz3lNMrhJR9OFc-ChoginftXNYjPKO_HAo.png?width=640&crop=smart&auto=webp&s=3d79bed8e3bb1e04f8652af6778a0784cccf7717)
I was very curious why all the models were topping out on GPQA-Diamond around 92 or 93% (AA) and spent the last few weeks pouring over GPQA (Diamond and Extended), and then expanded to auditing MMLU-Pro and MMMU-Pro.
It was quite frankly shocking just how many questions were malformed, had wrong answer keys, or questions with more than one realistic answer. In fact, on GPQA-Extended, MMLU-Pro and MMMU-Pro, ~12% of questions were verifiably broken! Once fixed, the top models hit around 98%.
Full paper released here: https://github.com/adamallcock/answer-key-audit
As part of this process, I have also shopped -Clean versions of all four benchmarks with the broken items removed, but also a full flagged-candidate ledger so you can see exactly why. I've also included dual original-vs-cleaned scoring, lm-eval-harness tasks and Hugging Face datasets.
Quite frankly, I have absolutely no idea how to get this out into the hands of industry so they can start using the -clean version.