How We Benchmark Our AI Agent — And Publish the Real Numbers

Every AI app builder claims its agent is good. Almost none publish a number you can check. We do — including the runs where our agent fails. Here is how NoCoder measures its own agent, the real (unflattering) results, and why we think publishing them is the honest thing to do.

How we measure

Every evaluation runs the full production agent — the same orchestrator, tools, prompts, and memory that serve real users — against a fixture task inside a disposable workspace pod. Then we score it two independent ways:

• A deterministic acceptance gate (pass/fail). Not “the agent looked busy” — it actually runs tsc --noEmit, checks the files exist and contain the required change, starts the dev server, and inspects the trajectory. Pass means the task was provably accomplished.
• An independent LLM judge that assesses the outcome separately and is cross-checked against the gate.

We also keep a holdout set the improvement loop never sees (to catch overfitting), and infrastructure failures are recorded as failures — never quietly passed.

The time we caught ourselves overstating — and fixed it

What the agent is actually good (and bad) at

Correctness-gated results on our internal fixture bank:

• Targeted bug-fixing: ~85–90% pass. Given real broken code, the agent reliably produces a fix that compiles clean and makes the specific change.
• Simple full-stack app builds: reliable. It scaffolds working SPAs and can go prompt → full-stack app → live deploy with a real database (validated end-to-end).
• Complex full-stack scaffolding & consistency: weaker. Heavy multi-part builds and run-to-run consistency are where it still misses — and we say so publicly instead of hiding it.
• Code retrieval (RAG): recall@10 0.94, MRR 0.91 on our file-retrieval fixtures.

The part no competitor publishes: a closed self-improvement loop

A failed run feeds a reflection step that proposes a prompt change; the candidate is validated against the correctness gate (with the holdout set guarding against regressions) before anything reaches production. The platform measures itself continuously and improves from its own failures — and we publish the trajectory, methodology, and misses.

Want the control that these numbers are built around? Every change the agent makes is a per-file diff you review before it applies. See the AI app builder comparison or try NoCoder free.