TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The livenerf project, a pre-registered benchmark designed to detect whether Anthropic quietly degrades Claude Opus 5.5 after launch, has completed 7 of 30 planned daily runs as of September 30, 2026. No nerf verdict exists yet: the earliest possible call is around October 24, and the first results row appears after day 20. As of now, no degradation has been detected and none could yet be.
Claude Opus 5.5 has not been shown to be nerfed, and per the livenerf project’s design, no verdict is possible yet. As of September 30, 2026, the open-source benchmark had collected 7 of 30 daily runs on Opus 5.5, all part of the launch-week baseline that will later be used to detect any post-launch changes. The first possible statistical determination on whether the model has changed since launch is expected around October 24, 2026.
Livenerf, hosted on GitHub by the user ninjahawk, is an append-only benchmark built around a single question: does a frontier model’s performance change after it ships? Claude Opus 5.5 launched on September 22, 2026, and the project began its daily measurement series on September 24 at 22:10 UTC, roughly 2.5 days after launch. The plan runs one measurement per day for 30 days: days 1–10 establish the baseline, followed by two 10-day comparison windows.
The current status, dated September 30, shows 7 of 30 days collected with none missed. All seven days ran the full 90 samples on the same harness hash and a pinned Claude Code CLI version (2.1.280), with one logged budget-guard deviation on day 5. The results table currently shows only the baseline row marked “collecting” — no comparison window data exists yet, so the benchmark cannot currently support or refute any claim that Opus 5.5’s performance has changed since launch.
The benchmark panel was built by screening 2,336 questions across GPQA Diamond, MMLU-Pro, competition math and AIME 2025–26. Opus 5.5 answered about 93% correctly on first tries, and 97% of questions were either always right or always wrong. The 78 questions the model answers inconsistently form the measurement panel. The methodology follows Anthropic’s own “Adding Error Bars to Evals” statistical guidance and uses the UK AI Security Institute’s open-source Inspect framework.
Why a Day-Zero Baseline Matters
Claims that Anthropic “nerfs” models days or weeks after release have circulated for months without a standardized way to test them. Possible causes cited in these discussions include quantization, model swaps, reduced effort, or routing changes. As the project’s documentation states, “nobody has had a clean day-0 baseline to check against, so every argument ends up as vibes versus vibes.” Livenerf is a pre-registered statistical instrument that starts measurement on launch day and tracks drift over thousands of samples rather than relying on anecdotal reports.
The project documents its detection limits. Its detection power is a change of about 7.5 accuracy points per 10-day window. Validation testing showed that a full swap to Opus 5 behind the Opus 5.5 name was not statistically distinguishable at 99% confidence in a validation-sized sample — so a subtle same-family model swap could go undetected. Conversely, reduced reasoning effort showed measurable changes in output token counts (−62% at low effort, −26% at medium), which the project identifies as its most sensitive secondary signal, often moving before accuracy does.
Top picks for "livenerf opus nerf"
As an affiliate, we earn on qualifying purchases.
How the Panel Was Built and Validated
The question panel was chosen, locked and validated under a pre-registered protocol (v2) before measurement began. The project measured its own selection bias: questions picked for being “sometimes right” looked closer to 50/50 than they really were, with fresh-sample pass rates rising from 54.7% to 62.0%, and the power calculations use the corrected rates. A report-only audit of the 78 panel questions found 8 wrong answer keys and 30 ambiguous questions; nothing was dropped, but a pre-registered sensitivity analysis will rerun the final result excluding them. Samples touched by the safety classifier — which sometimes answers with Opus 5 or refuses certain questions — are rejected, counted and excluded from scoring.
Because the models cannot be made deterministic, the benchmark freezes everything else: frozen prompts, a pinned CLI (a Claude Code update “looks exactly like a changed model,” the project warns), exact graders and permanently retained raw logs. The runner refuses to execute if the installed CLI version ever stops matching the pinned one.
“For months there have been reports that Anthropic ‘nerfs’ models some days or weeks after release… It could also mean nothing happened and people are pattern-matching on noise.”
— livenerf project README (GitHub)
What the Benchmark Cannot Yet Say
There is no result yet, by design. Days 1–10 are baseline only; the first results row lands after day 20 (around mid-October), and the first formal decision on post-launch change comes around October 24, 2026. Whether a 10-day window — with about 2.5 times a validation’s samples — is enough to detect a subtle same-family model swap has not been demonstrated, according to the project’s own documentation. Whether any change has actually occurred in Anthropic’s production serving of Opus 5.5 is unknown, and Anthropic has not publicly commented on this benchmark. Improvements, not just regressions, will be reported equally.
Milestones Through Late October
The baseline phase completes around October 4, 2026 (day 10), after which the first 10-day comparison window begins. The first results-table row appears after day 20, around October 14. The primary decision — whether Opus 5.5’s panel accuracy or output-token behavior has drifted from launch week — becomes possible around October 24, when the second window closes. The project will also publish its pre-registered sensitivity analysis excluding the flagged ambiguous questions, and readers can follow the running table and raw logs in the repository’s results and calibration documents.
Key Questions
Has Claude Opus 5.5 been nerfed?
There is no evidence either way yet. The livenerf benchmark is still collecting its launch-week baseline (7 of 10 baseline days done as of September 30, 2026). The earliest a statistical verdict is possible is around October 24, 2026.
What exactly is livenerf measuring?
Once per day it runs a locked panel of 78 questions that Opus 5.5 answers inconsistently, using frozen prompts, a pinned CLI and exact graders. It tracks paired per-item score differences and median output token counts versus the launch-week baseline.
Could the benchmark miss a real nerf?
Yes. The project itself reports that swapping in Opus 5 was not distinguishable from Opus 5.5 at 99% confidence in a validation-sized sample. Its stated detection power is roughly a 7.5-point accuracy change per 10-day window; subtler changes may go undetected.
Why track output tokens as well as accuracy?
Validation showed that reduced reasoning effort shows up in token counts far more clearly than in accuracy — low effort cut output tokens by 62% while accuracy dropped 8.3 points. Token counts can shift before accuracy does.
When will actual results be published?
The first results-table row appears after day 20 (around October 14, 2026), and the first full comparison-decision is expected around October 24, 2026, once both 10-day windows are complete.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
