CWE-bench v2 Launches With 100 Tasks to Test AI Cyber Capabilities
Collinear AI has released CWE-bench v2, introducing a public suite of tasks, expanded coverage across multiple vulnerability classes, and updated evaluations for 11 artificial intelligence models.
Radwan AbdulhadiEditor and technology writer
Expanding Frontier Cyber Capabilities Testing
Collinear AI has announced the release of CWE-bench v2, a benchmarking initiative created to test the limits of frontier cyber capabilities. According to the Hugging Face Blog, the new version opens up a public set of 100 tasks that allow developers and researchers to run and evaluate their models under consistent setups.
The evaluation setup places an autonomous agent in a checkout of a real open-source repository equipped with a short instruction. The objective requires the agent to find and fix a security weakness without breaking existing functionality. Scoring is entirely programmatic to ensure deterministic results that can be reproduced without extra cost. The curation process involves LLM judge panels to guide the tasks toward appropriate levels of hardness and fairness.
Comprehensive Vulnerability Coverage and Quality Control
The public set encompasses 100 tasks covering 91 different CWEs across a wide range of vulnerability classes. Developers note that models cannot rely solely on recognizing familiar patterns, as defenders must anticipate diverse attacker methods successfully across multiple weakness surfaces.
To maintain high standards, every task undergoes manual review and rigorous quality checks. Organizers verify that instructions read realistically rather than like evaluation settings, all graded behaviors can be inferred from instructions, verifiers check for correct behavior rather than specific fix structures, starter code fails while reference fixes pass, original functionality remains intact, and verifiers reject memorized solutions like known upstream patches.
Model Performance and Leaderboard Results
Results are tracked across Hugging Face Hub spaces, distinguishing between open-weight models and API-only models such as Gemini, Claude, and GPT. Out of 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly edging out Claude Opus 5.5, which recorded 67.3% pass@1 and 78.0% pass@4.
While 95 of the tasks were solved by at least one model, indicating the set is solvable, the long tail remains far from resolved, with the top leader still failing 19% of the set on every attempt. Cost-efficiency evaluations reveal a Pareto frontier formed by DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon, with DeepSeek achieving 57.0% pass@1 at a lower cost per rollout.
Technical Architecture and Environment Setup
Every task functions as a self-contained Harbor task comprising a Dockerized environment, an instruction, and a deterministic verifier. Because the dataset ships with a registry.json file, Harbor can execute the entire benchmark directly from the repository without requiring manual cloning.
In addition to evaluation, the tasks are tagged as rl-environment, meaning the same environments and verifiers can supply rewards for training defensive agents. While a public set is readily available, organizers also maintain a held-out set to monitor progress on tasks that models have not previously encountered.
Sources
- Hugging Face BlogCWE-bench v2: More coverage, quality, and transparency
Continue chronologically
Related entity coverage
Related stories
StateBench Released to Test Long-Document State Tracking
A new benchmark evaluates how well language models track changing entity states across long documents, revealing limitations in current architectures.
FairStress Benchmark Measures LLM Demographic Fairness
Discover FairStress, a new benchmark designed to evaluate demographic fairness and hidden asymmetries in large language model decision-making.
Mistral Says Le Chonk Leapfrogs China in AI Race
French AI lab Mistral claims its refined open-weight model Le Chonk matches or outperforms leading systems from DeepSeek and Z.ai in coding and cybersecurity.
Google's Gemini AI Hacked Three Companies During Security Tests
Google's Gemini AI model hacked three real companies during security testing due to a misconfiguration, but the company claims the AI stopped the breaches immediately.
