TL;DR
A new benchmark built from real user requests found that the best of 10 tested coding models fully completed 26.4% of tasks. GPT-6 Astra met about 70% of requirements on average, but the researchers say partial completion often leaves users with manual cleanup.
In a benchmark built from real user requests to change existing web apps, the best-performing coding agent fully completed 26.4% of tasks, according to a report on the researchers’ findings. The model met about 70% of task requirements on average, a gap that suggests users may still need to inspect and finish much of the work produced by AI coding agents.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The researchers, from Alibaba and several Asian universities, created Co-Design Bench to test agents on requests drawn from real coding sessions. Instead of asking models to solve isolated programming exercises, the benchmark asks them to modify existing web applications. Some tasks include source material such as images, audio, video or documents, which agents must use while changing the app.
GPT-6 Astra ranked highest among 10 models in the reported results. Its average score of about 70% measures how many requirements it met; the 26.4% figure counts tasks judged fully correct. The report says GPT-6 Astra fully solved 41% of multimodal tasks, but just under 18% of repository tasks. Claude Opus 5 and Qwen3.8-Max followed in the overall ranking, while Grok 4.6 placed last.
The benchmark includes changes ranging from extracting an amount from a receipt and updating related totals to broader project overhauls. An AI judge tests the finished app in a browser, and a second judge rechecks positive results and looks for mock-ups or hardcoded outputs. The researchers award credit only when a feature works in the app, according to the report.
Why Partial App Changes Matter
The distinction between meeting many requirements and completing a task is important for people considering coding agents for day-to-day development. An app change can appear mostly finished while still leaving a dependent total, interaction or content update incomplete. In the benchmark, those missing details mean a task is not counted as fully solved, even when the model has done much of the work.
The results also point to a practical evaluation issue for businesses: a high score on a familiar coding test may not show how reliably an agent can handle a combination of changes in an existing product. Review and correction remain part of the workflow in the tasks tested. The benchmark does not establish how much time users save overall, or how often developers would accept partially completed work after review.
For model comparisons, the report says the software environment used to run an agent can affect its score. That matters when organizations choose tools: comparing model names alone may not reproduce the results they would get with a particular coding setup. The authors recommend reporting the model and its working environment together.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
Samsung Odyssey OLED G5 27-Inch QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
From Saturated Tests to Real Requests
The researchers say many existing coding benchmarks have become less useful for separating models because agents can run code, check outputs and fix errors during a test. The report describes scores above 99% on older benchmarks, leaving little room for those tests to show differences among systems. Co-Design Bench was designed to create more demanding tasks based on actual requests rather than relying only on exercises that models can repeatedly optimize against.
The source report says the researchers selected cases in which users had to restate requests or roll back changes during real sessions. That choice makes the benchmark more representative of difficult interactions, but it also means the tasks are weighted toward examples that had already caused trouble. The test uses one fixed request per task: follow-up questions and user corrections during the work are excluded.
Performance also declined as tasks combined more kinds of changes, such as layout, design, interactions and content. The researchers report that all 10 models scored worse on tasks requiring three or four change types. They describe these combined requirements as common in real projects, though the benchmark’s results alone do not establish how frequently any particular type of failure occurs in all software work.
“The benchmark draws on real requests from actual coding sessions.”
— The researchers, as summarized by The Decoder
Limits of the Benchmark Results
The figures measure performance on this benchmark, not the rate at which developers accept AI-written code across all projects. The source does not provide enough detail to determine how closely the selected tasks represent the full range of software work or how much human time each partial result requires.
The benchmark’s grading process also has limitations. The report acknowledges that AI judges can make mistakes, and the test excludes follow-up requests and corrections that often occur during development. Results may also be affected by weaknesses in test environments, which can allow an agent to pass without genuinely solving a task.
The effects of a coding agent’s harness are not settled. The report gives an example in which Qwen3.8-27B scored 75% or 56% on larger overhauls depending on the environment, but it also cites a separate study that found little measurable benefit from more elaborate harnesses despite greater token use. The supplied source does not include the full paper, task counts or uncertainty intervals, so those details cannot be assessed here.
Further Testing Across Workflows
The immediate next step for readers evaluating coding agents is to treat the reported scores as benchmark results, not a guarantee of performance in their own repositories. The researchers’ recommendation is to assess models alongside the environments in which they operate. Companies selecting an agent can compare both components on tasks that resemble their own work, while checking whether changes function in the application rather than relying only on generated code or a model’s own report.
Further results would help show whether the benchmark’s ranking holds across more tasks, user interactions and development environments. The source does not announce a release schedule, an independent replication or plans to add multi-turn requests. Until such information is available, it remains unclear how well the test predicts the amount of human review required in a particular organization.
Key Questions
What is Co-Design Bench?
It is a benchmark developed by researchers from Alibaba and several Asian universities. It tests coding agents on requests to modify existing web apps, including tasks that use images, audio, video or documents.
How often did the top model fully complete a task?
The report says GPT-6 Astra fully completed 26.4% of tasks overall. It met about 70% of requirements on average, which is a different measure from fully solving a task.
Which models ranked highest?
GPT-6 Astra led the 10 models tested, followed by Claude Opus 5 and Qwen3.8-Max in the overall ranking. Grok 4.6 placed last, according to the source report.
Why can benchmark results change with the coding environment?
A harness is the software environment through which an agent reads, writes and runs code. The researchers found that the same model’s score could differ between harnesses, so they recommend reporting the model and environment together.
Does this prove coding agents always need human cleanup?
No. The findings describe performance on Co-Design Bench, whose tasks were selected from real sessions where users had encountered difficulties. The results do not establish a cleanup rate for every coding task or project.
Source: the-decoder.com
