Anthropic Turned Fuzz Tests Into Proofs. What Claude’s Formal Verification Work Shows, And What It Doesn’t

🔍 Read the full analysis: Anthropic Turned Fuzz Tests Into Proofs. What Claude’s Formal Verification Work Shows, And What It Doesn’t on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Anthropic’s Frontier Red Team and formal-verification collaborator Theorem reported on October 22 that Claude generated proofs for 239 fuzz-test entry points across 123 open-source libraries, establishing that the tested compiled programs do not crash on inputs under 64 MB under the project’s stated assumptions. The work matters because it applies machine-checked formal verification at substantial scale, but its results concern specific RISC-V builds and defined properties—not proof that the libraries are free of bugs.

The team used existing fuzz tests as specifications rather than trying to define all aspects of what each library should do. Fuzz tests provide small program entry points that can be supplied with different inputs; in this project, the property to prove was that an instrumented program would not crash for any input below the stated size limit. The work covered compiled binaries, not just source code.

According to Anthropic’s account, the libraries and tests were compiled into RISC-V executables and instrumented with AddressSanitizer, which makes certain memory errors cause a crash. Theorem supplied formal models of RISC-V instructions and Linux processes. Claude generated proofs in the Rocq proof assistant, and the Rocq checker mechanically checked whether those proofs followed from the specifications. If a proof attempt exposed a failure, the team could investigate a possible bug, patch the code and regenerate the specification.

Anthropic said the effort produced about 180 million lines of Rocq proof over several weeks. A typical proof was about 600,000 lines, while the median binary contained 18,000 RISC-V instructions. The company said it found hundreds of bugs, including cases involving large or specially structured inputs and sequences of calls that fuzzing had not reached. Those figures and descriptions are reported by Anthropic; the source account does not provide an independent audit of every finding.

What Machine-Checked Proofs Add

The project targets a known limitation of fuzz testing: fuzzers try many inputs and follow reachable code paths, but a successful test campaign cannot establish that every possible input has been covered. A proof of the stated property can provide a stronger guarantee for the particular program and conditions modelled. Anthropic’s examples include bugs requiring inputs around 20 MB, a decoder case involving a 64 MB input and more than 2 GB of memory, and a flaw dependent on calls across two handles to the same file.

That distinction could matter for software security. If developers can use automated systems to produce proofs at lower cost, they may be able to establish certain safety properties in more projects than traditional verification methods allow. But this report does not show that the economics have been solved: Anthropic says proofs could consume hundreds of millions of tokens, although newer models and reuse of shared proofs can reduce the cost. The result is evidence of a possible workflow, not a demonstration that every codebase can be verified cheaply.

The method can also cover more than memory safety where a test encodes another condition. Anthropic gives a compression test that checks whether decompression restores the original input; proving that test’s no-crash behavior can establish the round-trip condition as represented in that test. The proof still applies to the formalized specification, so properties that tests do not express remain outside its scope.

From Fuzz Tests to Rocq

Formal verification uses mathematical specifications and machine-checkable reasoning to show that a program satisfies a defined property. Its expense has limited its use, particularly when both the specification and proof require extensive expert work. Anthropic cites the seL4 microkernel as an example of the effort involved: its account says building it took two person-years and verification took 20 person-years. This project’s approach narrows the task by treating existing fuzz tests as the starting specification.

Anthropic described the work as, to its knowledge, the first large-scale use of AI to apply formal verification to production software at assembly-code level. That is the company’s characterization, not an independently established industry-wide finding. It said it is publishing proofs that did not require patches, previously unavailable trusted components, and specifications for fuzz tests it could not prove.

““discovered hundreds of bugs””

— Anthropic, describing the work

Which Builds the Proofs Cover

The main limitation is the gap between the verified setup and typical deployments. The proofs cover single-threaded, static RISC-V binaries linked against musl, according to the source account. Most users instead encounter software built for x86 or ARM and commonly linked against glibc. The report does not establish that the same results hold for those builds, other operating systems, or concurrent execution.

The guarantee is also bounded by the formal models, specifications and trusted tools. It means the specified program satisfies the checked property under the model’s assumptions—not that its behavior is correct in every respect. The source says the trusted base includes the RISC-V and Linux-process models, theorem statements and Rocq checker, along with hardware, the operating system and the OCaml compiler on which Rocq relies. It is not clear from the supplied material how the full set of reported bugs was independently validated or how costs compare across projects.

Publication and Wider Testing

Anthropic said it is making available the proofs that succeeded without code patches, trusted components that were not already public, and specifications for tests it could not prove. Those materials should let researchers and developers examine the proof setup and assess how much of the process can be reproduced. The source does not give a schedule for broader verification work or a complete breakdown of token and computing costs.

The next test for the approach is whether it can be applied beyond these particular RISC-V builds while retaining reliable, checkable results and a manageable cost. Until then, the findings support a narrower conclusion: AI-assisted proof generation can help analyze specific compiled programs and surface hard-to-reach bugs, while the scope of each guarantee depends on the program, model and property being proved.

Key Questions

What did Claude prove?

Anthropic says Claude generated Rocq proofs establishing that programs at 239 fuzz-test entry points would not crash on inputs under 64 MB, under the project’s stated models and assumptions. The guarantee is limited to that property and setup.

Does this mean the libraries are bug-free?

No. The proofs address defined properties for specific compiled binaries. They do not establish that every library behavior is correct or that no other kinds of bugs exist.

How did the work relate to fuzzing?

The team used existing fuzz tests as specifications and attempted to prove that the instrumented programs would not crash for any input within the stated size limit. Unlike fuzzing, which samples inputs, a successful proof covers the property across the inputs represented by its formal model.

Do the proofs apply to software people commonly use?

Not directly to every deployed version. The reported proofs concern single-threaded, static RISC-V binaries linked against musl. Anthropic’s account says most users run x86 or ARM builds, often linked against glibc, which were not covered by those proofs.

What remains to be tested?

Developers and researchers can examine the published proofs and components, and future work can test whether the process extends to other architectures and build configurations at manageable cost. The supplied account does not give a timeline for that work.

Source: ThorstenMeyerAI.com