I want everyone to take a look at this graph for a second.
ARC-AGI 3 was intentionally not allowing the reasoning agent to maintain its context across actions. It was effectively making the model forget what it had already figured out, over and over again, then scoring that crippled version as if it represented the system’s actual intelligence.
Once OpenAI allowed the agent to preserve its reasoning and compact older context, which is exactly how real world frontier agents work, its score nearly tripled while using far fewer tokens.
Compaction is a basic part of how a real world agent would function. Humans similarly write notes and preserve what they have learned. Nobody would test a human by erasing their memory after every action and then claim the result tells us their true capability.
The reality is that ARC-AGI 3 is not measuring general intelligence. In the real world, if an agent using reasoning and compaction could function in virtually the same way as a human would, that would be called AGI. The already existing agent can do 3x the score while using 6x less tokens, so the benchmark is intentionally dishonest as a measurement of general intelligence. A human is not required to reset its memory each time it starts a new puzzle, so this is absolutely egregious in my opinion.
The fact that an AI can do this much better just by remembering what it had already figured out is the true testament to how far in context learning has come. I was already not a fan of ARC-AGI after the quadratic penalty was applied for taking extra steps, but this just confirms my view that this benchmark strayed from the initial goal: measuring general intelligence of frontier models.
We're still going to saturate it anyways, and it's good that there are still tough benchmarks out there, but I just had to share that this is not a good look for this particular benchmark.