🔍 Read the full analysis: OpenAI Is Training Agents Inside Your Software. Read The Fine Print On Ironclad. on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI described training and evaluating a frontier model in hosted copies of Ironclad’s contract-management software, using 11 legal, commercial and procurement tasks. GPT-6 Astra met an average 55% of task criteria, while its estimated completion time was simulated rather than measured in customer workflows. OpenAI is inviting a small number of software companies to propose similar research partnerships.
OpenAI said on October 6 that it trained and evaluated a frontier model inside hosted copies of Ironclad’s contract-management software, using tasks based on legal, commercial and procurement work. The results show progress on operating specialized business software, but the model met an average of 55% of task criteria, and the reported task times are simulated estimates—not measured customer productivity gains.
Ironclad employees and OpenAI staff who use the product selected 11 tasks, including setting up nondisclosure agreements, creating procurement approval processes and updating reusable contract language for a selected jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes for each task. The tasks were scored against rubrics containing 8 to 50 criteria, depending on complexity.
OpenAI said Ironclad provided hosted product copies for model practice. It built synthetic training tasks from contracts filed publicly in the SEC’s EDGAR database, which it said were filtered to remove personal information. OpenAI stated that the work did not use its customer data, its internal contracts or non-public Ironclad customer data. The company identified GPT-6 Astra as the first frontier model trained through this approach.
OpenAI reported that GPT-6 Astra met an average 55.0% of rubric criteria, compared with 41.6% for GPT-5.6 Sol in a high-compute setting. An internal OpenAI model used during Astra’s development reached 63.7%. Astra met about 94% of criteria on one showcase task, but that single result is not the average across the evaluation. OpenAI also reported estimated times of 19.2 minutes for Astra and 37.0 minutes for Sol; it says those figures are simulated from assumed processing and generation speeds, not observed time savings for customers.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Software Workflows Need Reliable Rules
The results matter because business software tasks often depend on several rules being followed together. A procurement workflow, for example, might require Finance approval above a spending threshold, a Security review for certain requests and Legal review when terms are nonstandard. A system that satisfies some criteria but misses a required approval may still produce an unusable or risky workflow. The reported 55% score is not a 55% task-completion rate, and it does not establish that the model can safely take over contract work.
For companies using agents, the evaluation points to a need to inspect individual failures, not just an average score, and to keep people responsible for checking consequential work. OpenAI’s post itself says that losing track of a business rule limits the tasks an agent can reliably handle and that human oversight remains important. Ironclad CTO Sunita Verma likewise emphasized preserving the controls teams rely on.
The collaboration also has implications for software vendors. Training inside a product may help models handle workflows that general-purpose evaluations do not capture, while giving vendors evidence about where agents fail. But capable agents could also become a new way users interact with software, reducing reliance on its screens. In that case, a vendor’s durable value may depend more on its business rules, records, audit trails and controls than on its interface. That is an implication, not a result demonstrated by this study.
As an affiliate, we earn on qualifying purchases.
How the Ironclad Evaluation Was Set Up
The October 6 post described two OpenAI developments; the Ironclad collaboration received less attention than the company’s announcement about 722 mathematics manuscripts. Some AI news trackers inferred from the word “Ironclad” that it referred to a hardened agent framework. In this case, it is the name of a contract-management software company, and the post concerns model training and evaluation inside its product.
OpenAI framed the work around teaching models to understand business rules, perform multi-step tasks in specialized software and check completed work against requirements. The evaluation is limited to the 11 research tasks chosen for the project. OpenAI’s invitation to a small number of other software companies signals interest in extending this method to additional products and difficult workflows; it does not mean broad deployment or proven benefits across business software.
“Agents have to preserve the controls teams rely on.”
— Sunita Verma, Ironclad’s chief technology officer
As an affiliate, we earn on qualifying purchases.
What the Scores Do Not Establish
The published average does not show which criteria Astra missed on each task, whether particular failures could bypass a required approval, or how performance would change across different organizations and workflows. A rubric score measures criteria met; it is not the share of tasks completed correctly. The showcase result of about 94% applies to one task and cannot establish general reliability.
The timing figures also do not establish real-world savings: OpenAI described them as simulations, and the 11 research tasks are not a representative measurement of all Ironclad customer work. The source material does not specify a timeline for commercial availability, customer deployment, or independent evaluation. OpenAI’s stated data boundaries describe this research setup; they do not, by themselves, answer what data or safeguards might apply in future partnerships.
As an affiliate, we earn on qualifying purchases.
OpenAI Seeks More Software Partners
OpenAI said it is inviting a small number of software companies to work on tasks current agents cannot reliably complete. It asked prospective partners to bring a concrete example of a failing task, people with deep knowledge of the work, a secure testing environment and data suitable for research. The company has not named additional partners or announced a schedule for further evaluations.
For organizations considering agent use, the next useful evidence would include task-by-task results, the specific rules missed, how human review is built into the workflow and whether performance holds under customer conditions. Until such evidence is available, the Ironclad results support a research finding—measurable progress on selected tasks—rather than a claim of safe, autonomous contract operations or verified productivity gains.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is Ironclad in this announcement?
Ironclad is a contract-management software company. OpenAI described training and testing models in hosted copies of its product, not the launch of an agent framework named Ironclad.
What does GPT-6 Astra’s 55% score mean?
It is the average share of evaluation rubric criteria met across the tasks. It is not the percentage of tasks completed and does not show that the resulting work is ready to use without review.
Did Astra cut customer work time by nearly half?
That has not been established. OpenAI’s 19.2-minute figure is a simulated estimate based on assumed processing and generation speeds, not a measured customer result.
What data did OpenAI say it used?
OpenAI said it created synthetic tasks from publicly filed SEC EDGAR contracts, filtered to remove personal information. It stated that the project did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Can companies deploy these agents without human review?
The reported results do not establish that. OpenAI said human oversight remains important, and the average score leaves open which requirements were missed on individual tasks. Reliability in customer workflows remains unproven by this evaluation.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
