🔍 Read the full analysis: A Closer Look At CWE-bench V2’s Expanded AI Coverage on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI’s CWE-bench v2 makes 100 security-fix tasks, a grader and an evaluation harness publicly available for testing coding agents. The release reports a leading pass@1 score of 67.8%, but every evaluated model failed some tasks on every attempt, and the results do not establish performance in production security work.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from real open-source repositories, along with its grader and evaluation harness. The release reports that the top evaluated model achieved 67.8% pass@1; the authors also say every model failed some tasks on every attempt, underscoring that strong benchmark results are not the same as dependable performance across security work.
The benchmark asks a coding agent to work in a checkout of a real repository, interpret a short instruction, identify a security weakness and fix it while preserving existing functionality. The 100 tasks span 91 CWE weakness categories. Collinear AI says each task is packaged as a self-contained Harbor environment, and that the dataset includes a registry file for running tasks from Hugging Face. Users must accept the dataset’s access conditions before running it.
The release reports evaluations of 11 models. Gemini 4 Argon led at 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%. Pass@1 measures success in one attempt; pass@4 reports success when a model can make up to four attempts. The authors say at least one evaluated model solved 95 of the 100 tasks, but that figure does not mean a single model solved all 95.
The release also compares reported price and performance. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a stated cost of $0.04 per rollout. Claude Opus 5.5 scored about ten percentage points higher at $1.83, while Gemini 4 Argon was slightly ahead of Claude at a reported $3.92. These are the announcement’s estimates, not fixed prices: model pricing and evaluation setup can affect the cost figures.
What the Scores Say About Security Fixes
CWE-bench v2 offers researchers and developers a common set of tasks for comparing whether coding agents can find and repair security weaknesses without breaking expected behavior. Making the tasks, grader and harness public gives outside teams a way to inspect the evaluation and try to reproduce results under shared conditions. That can make comparisons more transparent than relying on vendor demonstrations or results from incompatible tests.
The reported scores also show the limits of current performance on this particular set. The leading pass@1 result remains below 70%, and the release says the top model fails 19% of tasks on every attempt. That is evidence of unresolved failures in the benchmark, not a measurement of the rate at which AI systems would fail across all real-world security work. A high score alone cannot show that an agent is safe to use without review or can handle the range of codebases, threat models and deployment conditions found in practice.
The cost comparisons may help teams weigh performance against expense when testing models, but they should be read as release-specific estimates. They do not by themselves establish which system offers the best value in other setups or for a particular security workflow.
Products Worth Considering
Bose S1 Pro+ All-in-One PA Speaker
Apple Watch SE (2nd Gen) GPS 40mm with Starlight Aluminum Case and Sport Band (S/M)
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
As an affiliate, we earn on qualifying purchases.
How Version Two Tests Repairs
Version 2 retains the prior task format and expands the public collection to 100 tasks. According to the announcement, the authors reviewed tasks by hand, checking that instructions were realistic, that the expected behavior could be inferred, and that verifiers assessed outcomes rather than one preferred code structure. They also say they checked that starter code fails, the reference fix passes, and existing functionality remains intact.
The release says the team used checks intended to reject memorized solutions, including known upstream patches, and drew on LLM judge panels while shaping programmatic checks for difficulty and fairness. The authors describe scoring as fully programmatic, with curated checks intended to support repeatable runs. These are descriptions of the benchmark’s design from its creators; they are not independent validation of every task or comparison.
Collinear AI maintains a separate held-out set for tracking progress on tasks not seen by evaluated models. The announcement says teams can request access by contacting the company. Public results are tracked through two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space that also covers API-only models. Contributors can submit results through pull requests.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The announcement does not supply enough methodological detail to independently assess every model comparison. Exact model configurations and the conditions behind each cost estimate are not fully specified in the material provided, and the release says leaderboard results can change as submissions arrive. Independent reproduction of the published scores would help establish how stable they are under different evaluation runs.
The held-out tasks are not public, so outside researchers cannot directly inspect or reproduce evaluations on that set. It is also unclear how closely results on these 100 tasks predict production performance. Hand review and programmatic checks can strengthen a benchmark, but they cannot represent every repository, vulnerability or deployment environment. The reported results should be treated as performance on this test set, not proof that an agent can safely carry out security work broadly.
Further Runs and Task Review
Researchers and developers can run models against the public tasks using Harbor after accepting the dataset access conditions and setting up Hugging Face authentication. Collinear AI invites feedback and task contributions, and results can be proposed for the public leaderboards through pull requests. Teams interested in testing against the private held-out set can contact the company, according to the release.
Additional submissions may alter the public rankings, while feedback could expose weaknesses in task design or verification. The next useful evidence will include independent reproductions, clearer reporting of model configurations and costs, and results showing whether benchmark performance tracks outcomes in security reviews beyond the curated task set.
Key Questions
What is CWE-bench v2?
It is a public benchmark from Collinear AI comprising 100 security-fix tasks based on real open-source repositories. The release also provides a grader and an evaluation harness.
Which model scored highest in the reported results?
Gemini 4 Argon led the reported evaluation at 67.8% pass@1 and 81.0% pass@4. These are results on the benchmark’s task set, not a general measure of security capability.
Did one model solve 95 tasks?
The release says at least one evaluated model solved 95 of 100 tasks. It does not say that one model solved all 95; the statement should not be read as a shared total or as a claim that the leading model solved every task.
Can researchers reproduce the results?
The public tasks, grader and harness are available for running evaluations, subject to accepting the dataset’s access conditions. The separate held-out set is not public, and the announcement does not provide all details needed to independently assess every model configuration and cost estimate.
Do the scores show that AI agents are ready for security work?
No. The results describe performance on this curated benchmark. The release reports failures even among the leading models, and it remains unclear how well these scores predict behavior across production software and security conditions.
Primary source: Hugging Face · via ThorstenMeyerAI.com
