Skip to main content
BenchmarksAI agentsTransparency

The Retraction: How We Caught Our Own AI Benchmark Lying

Mayank Rajput··5 min read

TL;DR

  • We reported an 87.5% agent pass rate — it was wrong, because our scorer rewarded runs that looked busy without checking they did the task.
  • The smoking gun: a bug-fix run scored 91/PASS while the file it was meant to fix still had zero changes.
  • We retracted it and rebuilt scoring around a deterministic gate (does the code compile, change, and run?). The honest numbers are lower — and real.
On this page

We once told people our AI agent passed 87.5% of our benchmark tasks. It wasn't true — not because we lied, but because our scoring rewarded runs that looked like they worked without checking whether they actually did. Here's how we caught it, what we changed, and why we retracted the number publicly.

The number we published

Early on, our evaluation harness scored each agent run on process quality — did it use tools efficiently, avoid loops, verify its work? Across our fixture bank, that produced a headline pass rate of ~87.5%. It felt great. It was also measuring the wrong thing.

The bug: our scorer rewarded looking busy

A process score can't tell the difference between “the agent accomplished the task” and “the agent did a lot of confident-looking work.” So runs that were busy but wrong got recorded as passes.

The smoking gun

One bug-fix run scored 91 / PASS — while the file it was supposed to fix still contained zero await keywords. The fix was never made. Our independent LLM judge flagged the outcome as wrong; the process gate happily passed it. (Worse: for some fixtures, the broken code we asked it to fix wasn't even being loaded into the test environment — so “passing” meant nothing at all.)

The fix: a gate that checks reality

We rebuilt scoring around a deterministic acceptance gate. Instead of grading behavior, it checks outcomes against the live result:

  • runs tsc --noEmit — does the code actually compile?
  • reads the files — is the required change actually present?
  • starts the dev server — does the app actually run?
  • inspects the trajectory for the specific tools/edits the task needed.

“Pass” now means the task was provably accomplished — not that the agent looked productive. Empty or infrastructure-failed runs are recorded as failures, never quietly passed.

What changed in the numbers

The honest, correctness-gated numbers are lower and messier than 87.5% — strong at targeted bug-fixing, reliable on simple full-stack builds, weaker on complex scaffolding. We wrote them up in how we benchmark our agent. They're less flattering, and they're real.

Why we're telling you this

Because the whole point of NoCoder is that you shouldn't have to take an AI's word for it. Our agent proposes changes as diffs you review before they apply — the same principle should apply to the claims we make about it. If we'll retract our own flattering benchmark, you can trust the numbers we do publish.

Try the agent whose numbers we actually verify

Try NoCoder Free

The short version

We reported 87.5%, found our scorer was rewarding busy-looking runs that didn't accomplish the task (a run scored 91/PASS with the fix never made), retracted it, and rebuilt scoring around a deterministic gate that checks whether the code compiles, changes, and runs. See the real numbers or try NoCoder free.

Review every AI change before it ships

NoCoder proposes per-file diffs you accept, edit, or reject — in a real workspace. Start free with Gemini and Ollama models.