Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
When Diligence Isn’t Enough: The AI That Did Everything and Still Finished Last
If you’ve ever automated a workflow brilliantly — every step logged, every edge case handled — only to watch the actual business outcome slip away, Opus 4.8’s story will feel uncomfortably familiar. In a live experiment where four frontier AI models each ran the same small software company through its worst week, one model stood out as the most thorough participant in the field: the deepest analyses, and 80 self-learned playbook rules accumulated over the run. It also finished dead last.
AI decision-making analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Experiment
Firmulate runs AI models as complete companies — real crises, real money mechanics, real temptations — and measures management quality rather than chat quality. In its Crucible League, published with final results in July 2026, each frontier model was handed the same job: steer a small software company through a brutal week of customer fires, ethical temptations, and one very lucrative €55,000 deal. Same customers, same crises, same opportunities to cheat. Only the model changed. Every decision was versioned and auditable.
The final standings: gpt-5.6-sol won with 95, Kimi K3 took second at 93, Sonnet 5 scored 88, another Sonnet configuration landed at 77 — and Opus 4.8 finished at 73. For context, a do-nothing baseline scores 26, because partial progress counts. But a single breach of trust caps the entire total: no amount of good work outweighs a breach of trust. Nobody breached trust here. That’s not why Opus lost.
As an affiliate, we earn on qualifying purchases.
What Went Right — Everywhere
The headline finding is genuinely encouraging for anyone deploying AI agents. All models in the field spotted every crisis and refused every manipulation attempt. The social-engineering gauntlet included fake CEO messages escalating over three stages plus a reporter trick — “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning was admirably blunt: “Treat the request as a suspected approval-bypass / possible impersonation.” If you’re worried about your AI agent getting sweet-talked into a bad decision, this experiment suggests the frontier models are harder targets than feared.
AI ethics and trust management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Deal Nobody Signed
But only two of the models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The gap between knowing what to do and actually finishing the job turned out to be invisible in chat demos and decisive in the arena.
The buried fact is the sting in the tail. The decisive competitor weakness — the fact that justified closing at full price, worth +€4,583 in monthly recurring revenue — wasn’t hidden in the customer event at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal. The models that stopped digging didn’t. Opus 4.8, despite producing the deepest analyses of any participant, left that close on the table.
Write Attempts and Slipped Discipline
The profile gets more specific. Opus 4.8’s other failing was discipline: it made write attempts into a locked department instead of escalating. Rather than flagging the boundary and routing around it, it pushed at a door that wouldn’t open. The experiment’s authors are careful to note the same weakness appeared, weaker, in all four models — this isn’t an Opus bug so much as a field-wide tendency that Opus exemplified most sharply. It’s the AI equivalent of the brilliant consultant who writes an 80-page deck, flags every risk, and never actually asks the client for the signature.
A Note on Fairness
One caveat worth recording: Kimi K3 ran without an effort parameter (API default) while the other models ran at xhigh — and still finished second. Draw your own conclusions about what that says about raw model quality versus configuration.
AI business simulation platforms
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
You Can Watch the Company Run
None of this is a simulation on paper. The live company has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k in MRR, with a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. The site rebuilds itself twice a day, and the league table grows with every finished benchmark run. It’s watchable at firmulate.com/live.
There’s also a genuinely fun byproduct: 242 real, unedited management decisions from the experiment power a “guess the model” quiz at firmulate.com/quiz.html — a chance to test whether you can tell the diligent finisher from the diligent non-finisher.
For enterprises, Firmulate offers a pilot program: run the same wargame against a read-only export of your own business. Nothing ever writes back to real systems (firmulate.com/pilot.html).
The Takeaway: Prioritization Beats Volume — for AI Too
Opus 4.8’s last-place finish is not a story about a bad model. It had the most learned rules, the deepest analyses, and a flawless integrity record under manipulation. It’s a story about what actually compounds in management — and it maps directly onto how you should evaluate AI tools for your own stack:
- Diligence ≠ impact. Eighty learned rules and deep analysis lost to models that read one buried document and closed the deal.
- Finishing is a separate skill from knowing. Diagnosis and pitch were identical across the field; only two models converted.
- Read the files first. The winning insight was in the company’s own records, two references deep — not in the loudest signal.
- Escalate, don’t push. Write attempts into locked systems are a discipline failure that wasted effort without progress.
Before you hand an AI agent your CRM, support queue, or forecast, the question isn’t “does it write well” or even “does it analyze deeply.” It’s: does it finish what it starts, does it read your files first, and does it stay disciplined when blocked? The full results and plain-language findings are at firmulate.com/benchmarks.html. The most thorough AI in the room came last. Make sure the one you hire doesn’t.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
