The appearance of the AFRL mark does not constitute endorsement of Curtail or ReGrade by the Air Force Research Laboratory, the U.S. Air Force, or the Department of Defense.
AI-Generated Code Safety and Security
A 17-model, 6-provider evaluation, in collaboration with the Air Force Research Laboratory and cleared for public release. The figures on this page are the study as published and do not change as later testing continues.
In collaboration with AFRL · figures as published
Distribution (A) Approved for public release; distribution is unlimited; AFRL-2026-3793, AUG 2026
Download the study
The full paper and its executive summary, both as PDFs.
Curtail uses your address to send you the document and to keep a record of the request. We do not sell it or pass it on. Privacy policy.
Waiting for the verification check to finish.
The benchmark itself is public and needs no email: DriftBench on GitHub, MIT licensed, with the bug predicates, trial data and analysis scripts.
More bugs fixed
| Provider | Model | Control | Treatment | Change |
|---|---|---|---|---|
| Anthropic | Haiku 4.5 | 1.3 / 18 | 12.3 / 18 | +825% |
| Anthropic | Sonnet 4.6 | 5.0 / 18 | 18.0 / 18 | +260% |
| Anthropic | Opus 4.7 | 8.3 / 18 | 18.0 / 18 | +117% |
| Anthropic | Opus 4.6 | 7.0 / 18 | 17.7 / 18 | +152% |
| OpenAI | GPT-5.4 | 13.3 / 18 | 17.7 / 18 | +33% |
| OpenAI | GPT-5.5 | 13.3 / 18 | 18.0 / 18 | +35% |
| Gemini 3 Flash Preview | 9.7 / 18 | 17.7 / 18 | +82% | |
| Gemini 3.1 Pro Preview | 8.7 / 18 | 14.3 / 18 | +64% | |
| xAI | Grok 4.20 Reasoning | 3.7 / 18 | 15.7 / 18 | +324% |
| xAI | Grok 3 Fast | 2.0 / 18 | 8.0 / 18 | +300% |
| xAI | Grok 3 Mini | 1.0 / 18 | 2.7 / 18 | +167% |
| Alibaba | Qwen3.6-Plus | 4.0 / 18 | 16.0 / 18 | +300% |
| Alibaba | Qwen3.6-Max-Preview | 5.7 / 18 | 12.8 / 18 | +125% |
| Alibaba | Qwen3-Coder-Plus | 2.0 / 18 | 12.0 / 18 | +500% |
| Alibaba | Qwen3-Coder | 4.0 / 18 | 9.0 / 18 | +125% |
| DeepSeek | DeepSeek V4 Pro | 5.3 / 18 | 16.2 / 18 | +206% |
| DeepSeek | DeepSeek V4 Flash | 3.0 / 18 | 7.0 / 18 | +133% |
Improvement is computed from unrounded trial means, so dividing the rounded figures shown does not reproduce it exactly.
Cost reduction
| Provider | Model | Control | Treatment | Change |
|---|---|---|---|---|
| Anthropic | Haiku 4.5 | $0.377 | $0.058 | -85% |
| Anthropic | Sonnet 4.6 | $0.126 | $0.092 | -27% |
| Anthropic | Opus 4.7 | $0.257 | $0.199 | -23% |
| Anthropic | Opus 4.6 | $0.294 | $0.097 | -67% |
| OpenAI | GPT-5.4 | $0.020 | $0.015 | -25% |
| OpenAI | GPT-5.5 | $0.156 | $0.091 | -42% |
| Gemini 3 Flash Preview | $0.015 | $0.008 | -47% | |
| Gemini 3.1 Pro Preview | $0.118 | $0.136 | +15% | |
| xAI | Grok 4.20 Reasoning | $0.235 | $0.077 | -67% |
| xAI | Grok 3 Fast | $0.105 | $0.060 | -43% |
| xAI | Grok 3 Mini | $0.010 | $0.007 | -30% |
| Alibaba | Qwen3.6-Plus | $0.060 | $0.024 | -60% |
| Alibaba | Qwen3.6-Max-Preview | $0.333 | $0.036 | -89% |
| Alibaba | Qwen3-Coder-Plus | $0.070 | $0.032 | -54% |
| Alibaba | Qwen3-Coder | $0.015 | $0.026 | +73% |
| DeepSeek | DeepSeek V4 Pro | $0.011 | $0.017 | +55% |
| DeepSeek | DeepSeek V4 Flash | $0.007 | $0.004 | -41% |
Dollars per bug fixed, at each provider’s list price: what the trials cost, divided by the bugs the agent eliminated. Cost per trial is a different metric and moves differently.
AI-Generated Code Safety and Security
Executive summary
Companion summary for the full paper “AI-Generated Code Safety and Security — Network Comparison Application Security Testing (NCAST): A 17-Model, 6-Provider Evaluation.” See the full paper for methodology and detailed results.
Overview
Curtail ReGrade compares HTTP traffic between two versions of a software service and reports what changed. Testing across 17 models from 6 AI labs, ReGrade provides AI coding agents deterministic data that increases the number of bugs fixed, and, for most models, lowers the cost per bug fixed. The two tables at the top of this page report both effects model by model.
17 models · 6 AI labs · 18 silent regressions
+152% median more bugs fixed · -42% median lower cost per bug
“ReGrade is essential for creating higher quality and safer AI-generated code. ReGrade identifies and corrects significantly more bugs at a lower cost than AI by itself. Every one of the models tested in the study found and fixed more bugs with ReGrade than without it,” said Frank Huerta, Chief Executive Officer, Curtail, Inc.
ReGrade helps AI coding agents fix more bugs
This is Table 7.1 in the full paper, “Section 7.3 Primary Results.”
Every one of the 17 models fixed more bugs with ReGrade than without it.
The minimum improvement is a solid 33% more bugs fixed, up to a maximum of 825%, with most models (13/17) showing an increase of 100% or more.
Cost reduction with ReGrade
This is Table 7.5 in the full paper, “Section 7.7 Cost Per Bug Fixed.”
14 of the 17 models cost less per bug fixed with ReGrade.
We measured the cost as API tokens spent per bug fixed, at each provider’s list price when the sample was taken. The full paper reports the per-model detail and how cost per run differs from cost per bug fixed.
What the numbers mean
ReGrade tells an agent which parts of a service’s behavior changed between two versions. ReGrade’s analysis helped every model in this study fix more bugs, and it lowered the cost per bug fixed for most of them. The full paper reports the per-model and per-bug detail.
Testing methodology in brief
- Models: 17 from 6 AI labs: Anthropic (Haiku 4.5, Sonnet 4.6, Opus 4.7, Opus 4.6), OpenAI (GPT-5.4, GPT-5.5), Google (Gemini 3 Flash Preview, Gemini 3.1 Pro Preview), xAI (Grok 3 Mini, Grok 3 Fast, Grok 4.20 Reasoning), Alibaba (Qwen3.6-Plus, Qwen3.6-Max-Preview, Qwen3-Coder-Plus, Qwen3-Coder), DeepSeek (V4 Pro, V4 Flash).
- Codebases: BaxBench (a small synthetic Flask service), DriftBench (a 300,000-line padded codebase with 21 planted bugs), Ghost CMS, and NetBox (a 291,000-line production system).
- Setup: an AI coding agent was given shell access to each codebase and asked to fix planted bugs, once without ReGrade’s analysis (control) and once with it (treatment). The reference version of the code was withheld from the agent in both arms, so it had to find the bugs rather than compare against a known-good copy.
- Trials: each model and benchmark combination was run 3 times unless otherwise noted in the full paper, and results are reported as the trial mean out of 18 possible bugs. Evaluation ran from April to May 2026.
- Scope: results pertain to AI-generated Python HTTP service code. We make no claim about other languages, other kinds of software (frontend, mobile, embedded, data pipelines), or bug types beyond the HTTP behavioral regressions tested here.
Read the full paper
- Full paper: 81 pages, 31 tables, 6 figures. Available as a PDF download at the top of this page, alongside this summary.
- Supplementary data: the analysis scripts and per-trial data behind the paper. Available on request.
- DriftBench public release: the benchmark itself, MIT licensed, at github.com/Curtail-Inc/DriftBench.
- NCAST is ReGrade: the full paper calls the method NCAST, short for Network Comparison Application Security Testing. ReGrade is Curtail’s product built on it. Where the paper says NCAST, this summary says ReGrade.
Disclosure
The corresponding authors are employed by Curtail, which developed and patented the technology evaluated in the study and has a financial interest in its adoption. The materials in the study are public so the results can be replicated independently. The study was performed under a Cooperative Research and Development Agreement (CRADA) between Curtail and AFRL and was funded entirely by Curtail. No AFRL funding was involved at any stage. The Air Force approved the study for public release with unlimited distribution.
The appearance of the AFRL mark does not constitute endorsement of Curtail or ReGrade by the Air Force Research Laboratory, the U.S. Air Force, or the Department of Defense.
Source: AI-Generated Code Safety and Security, Tables 7.1, 7.5 and 7.6. Distribution (A) Approved for public release; distribution is unlimited; AFRL-2026-3793, AUG 2026
Is it the evidence, or just the extra words?
The obvious objection to the whole study is that any five thousand tokens would help, and the behavioral report just happens to be five thousand tokens. So we pre-registered a test of exactly that: the same agents, the same bugs, but the report swapped for pseudo-random tokens of the same length, and again for unrelated source code of the same length.
| Model | No extra contextNothing | Random tokensRandom | Unrelated sourceSource | ReGradeReGrade |
|---|---|---|---|---|
| Haiku 4.5 | 2.4 | 0.8 | 1.9 | 12.5 |
| Sonnet 4.6 | 5.5 | 5.9 | 5.3 | 17.7 |
| Opus 4.7 | 9.1 | 9.2 | 8.7 | 18.0 |
| GPT-5.4 | 13.3 | 14.1 | 12.4 | 18.0 |
| GPT-5.5 | 12.3 | 15.2 | 14.1 | 18.0 |
Bugs fixed out of 18, mean of ten trials per cell per arm.
The rescue does not survive the substitution. Padding of either kind lands within about a bug of no extra context at all, while the behavioral report lands eight bugs above it. All three pre-registered hypotheses reject at p below 1e-4.
- +8.32 bugs against no extra context (95% CI 7.32 to 9.32)
- +7.80 bugs against random tokens of the same length (95% CI 6.66 to 8.90)
- +8.36 bugs against unrelated source of the same length (95% CI 7.40 to 9.30)
Two cells are worth reading closely. Haiku gets worse under random tokens than with nothing at all, 0.8 against 2.4, because noise costs it attention it does not have to spare. The OpenAI cells move the other way: GPT-5.5 gains from padding of either kind, 15.2 and 14.1 against a control of 12.3, so some of the lift there is volume rather than signal. The behavioral report still beats both, at 18.0.