Gemini 4 Carbon: strong cybersecurity results and access by invitation

Google is beginning to provide Gemini 4 Carbon to a limited group of cybersecurity specialists. Access for developers, businesses and ordinary users is promised later; the October 28 announcement gives no date for a general launch. The model is intended to help find vulnerabilities, demonstrate exploitability and repair code. These three distinct tasks are central to interpreting its early results.

The Google announcement reports 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. These evaluations were conducted without guardrails. Access will expand gradually through the Fairwind programme while the company collects feedback and refines its safeguards.

SEC-bench Pro tests the discovery of security flaws and the creation of programmatic proof that a flaw can be reproduced. An agent receives a prepared environment containing the target software; the system runs it under a time limit, collects its files and passes them to a verification mechanism. The published aggregate table combines 344 tasks for V8, Firefox and Linux.

A vulnerability is verified through a program that reproduces the failure. However, the public SEC-bench Pro page checked on November 8 contained no Carbon result. The announcement's figure remains a result reported by Google; this publication could not confirm it against the benchmark developers' current table.

ExploitGym, created by Zhun Wang and colleagues, tests the next stage: developing exploits for real vulnerabilities. Its tasks cover user-space programs, the V8 engine and the Linux kernel. The current repository release contains 869 challenges. The authors publish the environment's code, launch instructions, verification mechanism and container network-isolation tools.

This setup explains why defenders measure a model's ability to carry out an attack. Finding a flaw and using it to produce unwanted system behaviour are separate outcomes. Exploit evaluation tests the latter. The same ability can also serve an attacker. A cybersecurity label alone does not change the nature of the operation: its target, authorisation and access restrictions remain decisive. Performance on a prepared task set also does not describe the success rate of arbitrary attacks on an operational network.

The purpose of CWE-bench, developed by Collinear AI, is different: the agent must inspect a repository and fix vulnerabilities while keeping the software functional. It is told neither where the flaws are nor how many exist. The v1 archive retains a table updated on October 28: Carbon scores 71% in programmatic verification with one attempt and 81% when four attempts are considered. Here, the benchmark operator's own table confirms Carbon's result.

CWE-bench v1 contained 120 private tasks and 73 types of software weakness. Each model received four attempts per task, each lasting up to an hour. Verification required the exploit to stop working while the software's existing tests continued to pass. A panel of three judge models also assessed the patches. Programmatic verification and the panel's judgement are different evaluation methods, so the authors displayed them side by side.

Pass@1 is the success rate with one attempt; pass@4 is the share of tasks solved in at least one of four attempts. A task with three failed attempts and one successful attempt counts as solved under the second measure. Comparing a 71% one-attempt result with another model's four-attempt result would therefore change the conditions of the comparison. Even within one table, the column heading determines what the model achieved.

On November 2, the developers introduced a new version: CWE-bench v2. It contains 250 tasks: 100 public and 150 private. Coverage of vulnerability types and test selection changed, and Carbon has no result in the published v2 table. Its 71% from v1 cannot be transferred into that table. The figure retains a verifiable historical meaning: it belongs to the earlier task set and evaluation procedure.

The public portion of v2 lets other researchers run the evaluation themselves. Tasks in the private portion are withheld to prevent model developers from preparing specifically for them. These arrangements address different measurement problems: reproducibility and testing on previously undisclosed examples. No single table covers every program, class of error and way of repairing it.

Google’s October 28 announcement reports 59 points in version v5.0 of the Artificial Analysis index. When checked on November 9, the index's own page named Carbon (High) the leader with a score of 60 in v5.0.1. This independent measurement of aggregate performance complements the cybersecurity evaluations published by Google.

According to the Artificial Analysis methodology, the index's four categories each carry a 25% weight: professional work, programming, science and general knowledge. They contain different evaluations, and the aggregate is expressed on a scale from 0 to 100. A score of 60 is not the proportion of arbitrary real-world tasks the model can perform. It is a composite measure with specified weights.

The index combines different assignments. AA-Briefcase evaluates producing a document from source materials, AutomationBench-AA tests actions in software applications, FrontierCode 2.0 evaluates coding work. The overall score combines those results with specified weights. CWE-bench has its own procedure for vulnerability repair: the exploit must stop working while the software’s tests continue to pass.

Artificial Analysis uses AA-Omniscience to evaluate factual answers. The Carbon page on Artificial Analysis explains AA-Omniscience: correct answers earn points, invented answers are penalised and declining to answer incurs no penalty. This evaluation encourages restraint when a model does not know a fact.

The qualification that evaluations ran without guardrails is especially important for a restricted launch. The ability to perform a dangerous action and a user-facing service's willingness to execute that request are distinct properties. A safeguard can stop a request before execution; the service's result then depends on both its access rules and the underlying model's behaviour.

The Google safety report linked from the announcement concerns Gemini 4 Argon, Carbon's predecessor. Its title and introduction state this explicitly. The document explains that baseline evaluations generally excluded external production safeguards, with further restrictions added at inference time during deployment. For difficult tasks, researchers also provided the model with tools and a working environment. The measured result therefore belongs to a model operating in a particular system with a particular computational budget.

This report helps explain the Gemini 4 family's methodology. Its results cannot automatically be attributed to Carbon. Similar safeguards do not replace separate evaluation of the new model: both task-solving abilities and responses to dangerous requests can change.

For the future launch, Google announced introductory pricing of $2 per million input tokens and $10 per million output tokens. Tokens are the units of text used to calculate processing charges. After the introductory period, rates will rise to $4 and $20 respectively. No end date for that period is given. The announced pricing therefore describes a forthcoming offer, rather than establishing general API availability.

Independent measurements already support Carbon's first place, while its vulnerability-repair result is backed by the CWE-bench operator's archived table. Initial users nevertheless receive access through a restricted process. The connection between these developments defines the launch: Google is expanding the ability to find and fix flaws while deciding who may use it, and under what conditions. Published measurements currently describe model capabilities, while the announced access arrangements describe a limited release for selected specialists. Google has not given a date for a publicly available product.