🔍 Read the full analysis: How Z.ai’s Legal AI Earns A Near-Top Benchmark Rank With A 1-In-5 Pass Rate on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived results. The score means it passed every criterion on about one in five tasks, although it passed 92.5% of individual criteria; benchmark results do not establish reliability for live legal work.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s archived leaderboard for Harvey’s Legal Agent Benchmark, with a 19.17% all-criteria pass rate. The result is a strong relative placing on a demanding test, but it means the model met every requirement on roughly one in five tasks—not that it completed most legal work reliably.
Z.ai describes the model, also called GLM Legal, as a legal-workflow system for document understanding, legal reasoning and task execution, including research, contract review and drafting initial drafts. Its October 13 AutoClaw blog post says the model is built on GLM-5.3-Flash and opened a beta by application. Vals lists the release date as October 12. Z.ai says it used legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows.
The benchmark figure reflects all-pass grading: a task counts as complete only when every rubric criterion passes. Vals’ archived leaderboard reports GLM Legal at 19.17%, alongside a 92.5% individual-criterion pass rate. The leaderboard describes the score as an average of results from two AI judges. Z.ai’s documentation rounds the task score to 19.2% and says the results were independently evaluated by Vals.
Z.ai also claims a 12.5 percentage-point improvement over the base model and “state-of-the-art performance among open-source models in this evaluation.” The archived table supports the arithmetic: GLM Legal scored 19.17%, compared with 6.67% for GLM 5.3 Flash. The open-source comparison is Z.ai’s characterization; the source material does not define its comparison set, and it reports no published weights for the legal model.
Why One-in-Five Passes Matters
The benchmark highlights the difference between getting many details right and delivering a complete legal work product. A model can pass 92.5% of individual criteria yet fail most tasks under the all-or-nothing rule if it misses even one required item. In legal work, that omitted item could be a deadline, citation, risk or recommendation. The task score is therefore not interchangeable with a general accuracy rate or a measure of how useful a draft might be.
For law firms and corporate legal teams, the result offers a comparison point, not proof that the system is ready for unsupervised use. Z.ai recommends using the model as an agent connected to services for laws, cases and company data. That raises practical questions about client-data handling, source verification and human review. A favorable leaderboard rank does not answer those procurement and governance questions.
The economics may also draw attention: Vals’ archived table lists GLM Legal at $0.15 per test, below the prices shown for the higher-ranked entries cited in the source material. Z.ai’s pricing page separately lists token rates of $0.37 per million input tokens and $1.25 per million output tokens. These figures use different pricing descriptions and do not, by themselves, establish the cost of deploying the model in a firm’s workflow.
Products Worth Considering
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source benchmark on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. Each task provides an instruction and matter materials and asks an agent to produce a work product for review. More than 75,000 expert-written criteria assess details such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
Harvey’s grading rule requires every criterion to pass for a task to count as complete. Its published example is a deal-team report that identifies eight of ten risks: under the benchmark’s approach, it is materially incomplete rather than an 80% completion. The public leaderboard uses a private holdout set; Vals says its holdout contains 120 tasks, with internet access disabled. Two AI judges grade submissions, and the reported task pass rate averages their results.
Harvey’s own first baseline, published May 26, reported a best overall all-pass score of 7.1% among the frontier models it tested. Harvey said those models completed less than 10% of tasks end to end in aggregate. That is a separate run and should not be treated as a directly comparable head-to-head result with Vals’ later leaderboard.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.””
— Z.ai, in its model documentation
What the Score Cannot Establish
The leaderboard is a bounded benchmark result, not evidence of performance across all legal matters, jurisdictions or firm workflows. It is not clear from the supplied material how the model performs on tasks outside the holdout set, how results vary across repeated runs, or whether it can maintain the same performance when connected to changing external legal databases.
The benchmark score also cannot show whether citations are authentic and support the statements attached to them in routine use. The source article describes a four-task spot check in which the author says GLM Legal cited rulings that could not be found; the author explicitly characterizes this as a small spot check, not a benchmark. That observation should not be generalized into a measured error rate. The article also reports that none of the Z.ai pages it reviewed carried a “not legal advice” notice and that they framed outputs as drafts for legal professionals.
Other open questions include how client information is processed, what retention and access controls apply, and what review Z.ai expects before generated material is used or sent externally. The supplied material does not provide answers to those operational questions or independently verify the model’s training claims.
Questions for Legal Teams
Z.ai’s AutoClaw announcement says the beta is available by application. The next useful evidence will be clearer information from the company about access, data handling, deployment controls and the model’s expected human-review process. Any further leaderboard updates should also be read with attention to the version tested, the holdout set and the grading method, rather than the rank alone.
For organizations considering a trial, the reported results point to practical checks: test the model on representative matters, verify every citation and legal proposition, and establish who reviews work before it reaches clients or counterparties. GLM Legal’s seventh-place ranking is a notable benchmark result; whether it can meet a firm’s requirements remains a question for testing and due diligence.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model passed every required criterion on about 19% of benchmark tasks in Vals AI’s archived results. Missing one or more criteria made a task a failure under Harvey’s all-pass grading rule.
How can it pass 92.5% of criteria but only 19.17% of tasks?
The measures use different units. The 92.5% figure counts individual criteria; the task score counts only tasks where every criterion passed. A single missed requirement can lower the all-pass task result.
Where did GLM Legal rank?
Vals AI’s archived leaderboard, last updated October 18, 2026, lists it seventh among 81 models, at $0.15 per test. Rankings can change as the leaderboard is updated.
Does the benchmark show the model is ready to give legal advice?
No. The result measures performance on a defined set of benchmark tasks; it does not establish suitability for independent legal work. The source material describes Z.ai’s outputs as drafts for legal professionals, and firms would need to verify citations, conclusions and data-handling practices.
Is Z.ai’s 12.5-point improvement supported by the leaderboard?
The arithmetic matches the archived figures: GLM Legal scored 19.17% and the base GLM 5.3 Flash model scored 6.67%, a difference of 12.50 percentage points. This is a leaderboard comparison, not a guarantee of improvement on other tasks or deployments.
Source: ThorstenMeyerAI.com
