A new benchmark paper suggests that today's best coding agents are far better at satisfying parts of a request than at delivering a complete, working result. In Co-Design Bench, the strongest of ten model endpoints passed an average of 69.9% of acceptance criteria per task, yet it fully satisfied only 26.4% of tasks. The gap between partial and full success is the paper's central finding.
What the Benchmark Measures
The paper, arXiv:2610.35138, describes Co-Design Bench as a reference-free benchmark for multimodal collaborative coding, meaning it tests whether a coding agent can turn a real user's request, which may include screenshots or visual context, into a working revision of an application.
The benchmark contains 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Each task pairs a normalized request with a runnable pre-edit application. Grading relies on 2,331 expert-written acceptance criteria that are hidden from the agent. Tasks are organized around three capability areas: Grounded Understanding, UI Transformation and Functional Implementation.
Because the criteria are hidden and there are no gold-standard solutions or target screenshots, the benchmark aims to reflect how real requests are phrased rather than how a lab would prefer them to be phrased. The authors frame it as a testbed for the distance between "partly satisfies the requirements" and "delivers the whole task."
The Results
With a shared agentic harness, the best endpoint averaged 69.9% of criteria satisfied per task. Whole-task completion, which requires every criterion for a task to be met together, stopped at 26.4%. That means roughly three of every four tasks had at least one requirement left unmet, even for the strongest system.
The benchmark has two tracks, Multimodal Editing and Repository Editing. According to coverage of the paper, performance was lower on repository edits that spanned three or four types of change than on edits involving one or two. That pattern matters for real software work, where a single request often touches interface code, logic and configuration at once.
Why It Matters
Many coding benchmarks report pass rates on isolated problems. Co-Design Bench uses requests drawn from real users, which makes it closer to the daily work of a development team. Its most useful signal is the distinction between per-criterion scores and full-task completion. A tool that gets most of the way there can still cost a developer time if the last few requirements take as long to fix as the first ones took to build.
For teams adopting coding agents, the practical advice is to measure completion rather than partial credit. Acceptance tests written before the task starts, and checked after the agent finishes, will reveal whether the output actually meets the brief. Human review remains essential for the criteria an agent is most likely to miss, especially those involving layout, interaction or edge cases.
The result also gives researchers a clear target. Improving the average score is not enough. The benchmark points toward work on planning across multiple change types and on verifying that every requirement is met before an agent reports success.
Limits of the Evidence
The coverage available at publication does not give the full per-model breakdown in the abstract, and it does not name the lab or labs behind the ten endpoints. Readers should consult the full paper before comparing specific systems. The pass rates describe this benchmark's tasks and grading method, and they may not transfer directly to other benchmarks or to a particular product.
Even so, the core finding is consistent with how many engineering teams describe their experience: agents are fast at producing most of a change and slower at finishing the last parts reliably. Co-Design Bench gives that experience a measurable form.
How Teams Can Use the Finding
The benchmark's design offers a template for internal evaluation. Teams can gather recent requests from their own developers, write hidden acceptance criteria for each one, and score agents on both per-criterion and whole-task results. Tracking the two numbers side by side shows whether a model upgrade actually reduces the amount of cleanup work, or whether it only improves the average score.
It also helps to separate tasks by complexity. The reported drop in performance on repository edits with several change types suggests that large, cross-cutting requests deserve closer human supervision than small, single-purpose ones. Splitting a big request into smaller steps, each with its own check, may recover some of the lost completion rate, although the paper itself does not test that approach.
