Coding agents: 99% in the lab, 26% in real life. Buy the process

You know the feeling. The agent builds the new landing page in ten minutes. Looks nearly done. Then the rest takes two days: the button that breaks on mobile, the form that won't submit, the copy in the wrong block.

That "nearly" now has a number.

What the study measured

Researchers, from Alibaba and several Asian universities according to The Decoder, published Co-Design Bench. The key figures from the paper:

• 273 tasks, curated from more than 37,000 requests across over 26,000 real coding sessions. The selection favored cases where users had to rephrase or roll back changes.
• 2,331 hidden acceptance criteria written by experts. The finished app gets tested in a browser, and a second AI judge checks passing results for mock-ups and hardcoded shortcuts.
• Ten models in the same agent harness. The best one meets 69.9% of criteria on average but fully solves only 26.4% of tasks.

According to The Decoder, older benchmarks sit above 99%; here scores drop by more than 35 points. It gets hardest when one task touches layout, design, interaction and content at once. In other words, when it looks like actual website work.

70% done on a website is 0% live

What I find telling: the gap between 70% of criteria and 26% of finished tasks isn't a rounding error. It's the distance between demo and launch. No customer sees 70% of a landing page. They see a broken one.

For marketing teams that means the agent doesn't remove the work. It moves it, from building to checking. Skip that in the plan and you're inflating your productivity on paper and paying it back in approval loops. I build my own website with Claude, which is exactly why I care more about the process than the model: clear acceptance criteria, screenshots on phone and desktop, a human who signs off.

The underrated number: the setup beats the model

The second finding matters even more for buyers. The same smaller model scored 75% or 56% on large overhauls depending on the environment, and the model ranking itself shifted with the environment. The authors recommend always reporting model and harness together.

To me that's the real reality check for any tool selection. You're not buying a model. You're buying a system of model, tools, prompts and review steps. Compare only leaderboard positions and you're comparing engines while ignoring the car.

The catch

Each task uses one fixed request; follow-up questions and mid-task corrections weren't tested. Real work is iterative, which helps the agent. And the grading is done by AI judges that can make mistakes too.

What I'd do now

• Write acceptance criteria first. Five to ten checkable points per page (mobile, form, load time, copy) before the agent starts.
• Budget review time. Every AI-built change gets fixed time for testing and approval in the plan, not "if there's time."
• Test tools in your own setup. Three real change requests from your backlog, same criteria, several tools. That tells you more than any leaderboard.

Bottom line: The agent hands you 70% in minutes. The last 30% decides whether you ship, and that part is still yours.