🔍 Read the full analysis: FrontierCode: Understanding The Maintainer’s Role In AI Contributions on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 AI models on repository tasks judged against maintainer-style review criteria. Claude Opus 5.5 ranked first, but passed every blocking check in 57.7% of attempts; the results also show a large cost gap between leading models.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a repository maintainer would accept a proposed change, and reported results on October 12, 2026. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in 57.7% of attempts, indicating that even the leading model did not consistently produce changes meeting the benchmark’s required criteria.
FrontierCode’s results cover the benchmark’s 100-task Main set. Artificial Analysis ran the models with the same open-source agent harness, Stirrup, allowing up to 250 turns per attempt and three attempts per task. Its review stage used GPT-6 Luna (medium) as the judging model. The benchmark’s full collection contains 150 tasks, according to the source report.
GPT-6.1 Sol placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. The next reported results include GPT-6 Astra at 47.0%, Claude Sonnet 5.5 at 45.8%, Xiaomi’s MiMo-V2.6-Pro at 45.4%, DeepSeek V4.1 Flash at 44.0%, Kimi K3 at 43.6% and Qwen3.8 Max at 43.2%. The report says nine of the 100 tasks were not solved by any model on any attempt.
The ranking measures more than test success. FrontierCode checks hidden tests, command results, whether an agent’s own tests fail on the original code, adaptive tests, limits on files changed and diff size, and an LLM review against maintainer criteria. A failed blocking check gives an attempt a zero; passing all blocking checks yields a score from 70% to 100%, depending on quality checks. The source report says attempts flagged by Cognition’s trajectory scanner for looking up solutions also receive zero.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode focuses on a practical gap between working code and an acceptable pull request. A patch can pass tests and still create review burdens if it changes unrelated files, lacks useful tests or fails other project-specific requirements. By making those issues part of the score, the benchmark tests a closer approximation of a maintainer’s decision than a simple test-pass count.
The results do not establish how often AI-generated changes would be accepted across all software projects. They do show that, on this benchmark and under its stated evaluation setup, the top model cleared all blocking checks in 57.7% of attempts. That makes human review relevant to interpreting claims that coding agents can independently prepare merge-ready changes.
Cost also varies sharply among the reported models. Artificial Analysis lists Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81; the report says Sol-to-Opus gains 3.1 score points for about 12.5 times the cost. These figures give teams a benchmark-specific way to weigh performance against expense, but they do not by themselves show what a model will cost or achieve in a different workflow.
Products Worth Considering
Apple iPad Air (2025) 13‑inch Wi‑Fi 256 GB – Space Gray (M3 chip)
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Apple iPad Air 13-inch (M4), 128GB, Wi-Fi 7, Space Gray
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Changes
FrontierCode was developed by Cognition and open-source maintainers, according to the source report. It was announced in June 2026, followed by version 1.1 in July with rules against looking up solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The report says maintainers spent more than 40 hours on each task and that tasks are private to limit the chance models encountered them during training.
That design makes the benchmark’s emphasis on review criteria central to interpreting its leaderboard. The tasks draw on established codebases, while checks cover correctness, regressions, test quality, scope and a review against maintainers’ stated criteria. Artificial Analysis evaluated only the 100-task Main set in this run, not all 150 tasks in the full benchmark.
The scores also differ from general model rankings in the source report’s comparison. Claude Sonnet 5.5, for example, ranked second among the models listed on the Artificial Analysis Intelligence Index but fifth on FrontierCode. DeepSeek V4.1 Flash ranked higher on FrontierCode than its general intelligence position in that comparison. This is evidence that the two evaluations rank models differently, not proof that either result predicts performance on every coding task.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
Limits of This Model Comparison
The published results do not show whether FrontierCode scores predict acceptance rates in live software projects, where maintainers, codebases and review practices vary. The source report also does not provide enough detail to establish how results would change across all 150 tasks, with different agent tools, or under other review setups. The 100-task evaluation is a defined benchmark run, not a universal measure of coding ability.
Several models had no reported FrontierCode score as of October 13, according to the report: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the reported ranking does not cover every model the source author considered relevant. The source’s comments about what missing models might score are expectations, not measured results.
Cost figures and scores are tied to the evaluated configurations and setup. The report does not establish that the same per-task costs or model ordering will hold for other task mixes, agent harnesses or deployment conditions. It also does not specify, in the supplied material, independent replication of the results.
More Model Results to Watch
The next useful comparison will be results for models absent from the October 13 leaderboard, particularly the named Gemini, Muse and Mistral systems. Until those evaluations are published, their standing relative to the 23 tested models remains unknown.
Readers following the benchmark can also watch for further runs, expanded coverage of FrontierCode’s 150 tasks and details that help compare results across evaluation setups. For now, the October results offer a snapshot of performance on the 100-task Main set; they do not establish how quickly scores will change or when another evaluation will appear.
Key Questions
What is FrontierCode measuring?
It evaluates AI-generated changes to real repositories using checks for correctness, regressions, test quality, scope and an LLM review against maintainers’ criteria. Its central question is whether a change meets the benchmark’s acceptance requirements, not merely whether it passes ordinary tests.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the October 2026 results with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts in the reported evaluation.
Does the result mean AI coding agents can work without review?
No. The benchmark reports that the leading model failed to clear every blocking check in 42.3% of attempts. The results support treating review as relevant to this benchmark, but they do not determine acceptance rates across live repositories.
How many models were tested, and how many tasks were included?
Artificial Analysis tested 23 models on FrontierCode’s 100-task Main set. The full benchmark has 150 tasks, according to the source report.
Which models were missing from the reported leaderboard?
The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026. Their performance on the benchmark was not established in the material provided.
Source: ThorstenMeyerAI.com
