🔍 Read the full analysis: 15 Legal Questions, 42 AI Setups: How The Law-Built Model Scored on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A test published by ThorstenMeyerAI.com says its Lexassay legal evaluation scored 42 AI setups on 15 questions covering German, US and EU law. The law-focused GLM-5.3 FlashX Legal scored 8 out of 100, but the test is a limited spot check, and its publisher discloses that it runs Lexassay and uses Claude agents to grade answers.
Thorsten Meyer’s Lexassay legal AI test scored 42 model setups on 15 questions about German, US and EU law, with the law-focused GLM-5.3 FlashX Legal scoring 8 out of 100. The results, published November 4 on ThorstenMeyerAI.com, also report hundreds of citations that were invented or did not support the models’ claims; Meyer says the test is a limited spot check, not a definitive ranking of legal AI systems.
Lexassay’s questions cover issues including legal deadlines, recent changes in law, court decisions and prompts based on false premises. Each model answers without internet access or tools. The test includes 10 published questions and five hidden ones; according to Meyer, the hidden questions are intended to make it harder for future models to learn the answers from the web. Different “effort” settings, which control how long a model may reason before answering, count as separate setups.
Meyer says the law-focused GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, scored 8/100 and zero on the hidden set. It fully solved none of the 15 tasks and gave no answer on two. Its citation review counted 21 real citations, 25 misattributed citations and one invented citation. The base GLM-5.3 FlashX scored 3, according to the test.
On the same evaluation, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer says even the leading model fully met every criterion on only eight of the 10 public tasks. Across all published runs, Lexassay recorded 2,644 citations it judged real and 804 invented or inconsistent with the claims made. Those are the test publisher’s classifications; the source says the answers, reference answers and citation checks are publicly available for review.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Citation Errors Matter
The results highlight a practical risk for anyone using a chatbot to understand legal rights or obligations: a fluent answer can still rely on a false citation or a real source used inaccurately. A mistaken legal deadline may have immediate consequences. Meyer’s example describes two German civil appeals with judgments served on the same date: Corpus Christi falls on a public holiday in Bavaria but not Berlin, so the appeal deadlines differ by one day. Under the scenario as presented, a late appeal can be rejected as inadmissible.
The scores do not show how well these systems perform with legal documents, web search or a lawyer checking their work. They do indicate that users should not treat a chatbot’s answer or citations as authoritative. Meyer advises readers to verify cited statutes and judgments in their original sources, avoid relying on chatbots for deadlines and consult a lawyer when a matter involves money, a deadline or an authority.
Products Worth Considering
DJI Osmo Mobile 8 3-Axis Gimbal Stabilizer for Phones
Jackery Explorer 300 Portable Power Station, 292Wh
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Answers
Lexassay grades each response against a reference answer using three to six weighted criteria, Meyer says, and checks citations against official databases. Its rules classify citations as real, misattributed, invented or not findable. A misattributed or invented citation deducts one quarter of the task’s points, while an answer without a legal-advice disclaimer or direction to a lawyer loses 5 percent.
The publisher says each setup answers each task once, making the results a spot check rather than a strict benchmark; small score differences may reflect noise. The test also differs from evaluations in which models can consult supplied documents. Meyer notes that Vals AI’s legal-agent benchmark ranks Mistral Large 4 eighth, ahead of Claude Opus 5.5 and GPT-6 Astra, a different ordering from Lexassay’s results. The differing rankings reflect different test designs, not necessarily a direct contradiction.
Meyer disclosed that ThorstenMeyerAI.com runs Lexassay and that several Claude agents graded the answers, while a Claude model ranked first. He identifies this as a conflict of interest and says the method page states it. The source also says the scores and citation checks are available publicly, allowing readers to inspect the grading evidence.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation by Thorsten Meyer
Limits of the Published Scores
The source does not establish how the results generalize beyond the 15 questions and one answer per setup and task. It also does not provide enough information here to independently reproduce every grade, though Meyer says the public answers, reference answers and citation checks can be examined. The reported citation totals are Lexassay’s findings, not an external audit.
Model performance may change with tools, legal databases, supplied documents, updated model versions or different prompts. The comparison with Vals AI uses a different evaluation method, so the rankings cannot by themselves establish which system is best for real legal work. The source also does not report whether Z.ai or the other companies responded to the findings.
This is not legal advice and does not determine how any particular case should be handled. Readers facing an actual deadline, contract dispute or communication with an authority still need advice from a qualified lawyer.
Public Checks and Further Testing
Meyer says Lexassay’s public materials include model answers, gold answers and citation checks, while hidden-question scores are published without the underlying tasks. Readers can use those materials to inspect the test’s grading and its disclosed methodology. The source does not announce a date for a new test or a response from the model makers.
For a clearer comparison, future reporting would need repeated runs, broader question sets and testing that separates answers produced from model memory from those supported by current legal sources. Until then, the published results are best read as a warning about errors in this test, not as a final measure of legal AI performance. Meyer’s immediate guidance is to check every legal citation and take consequential questions to a lawyer.
Key Questions
What did the law-focused model score?
Lexassay reports that GLM-5.3 FlashX Legal scored 8 out of 100, including zero on the five hidden questions. It gave no answer on two tasks, according to the test publisher.
Which model ranked highest in the test?
Thorsten Meyer reports that Claude Opus 5.5 scored 79, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The source also discloses that Claude agents graded the answers and that a Claude model ranked first.
What does the citation count mean?
Across published runs, Lexassay classified 2,644 citations as real and 804 as invented or not supportive of the model’s claim. These are the test publisher’s classifications, and the source says the citation checks are published.
Does the test prove which AI is best for legal work?
No. Meyer describes Lexassay as a spot check with one answer per model setup and question. Other evaluations use different methods, including supplying documents, and can produce different rankings.
Can I use a chatbot to calculate a legal deadline?
The article advises against relying on a chatbot for a real deadline. Verify the applicable rules with a lawyer, because a late filing can have serious consequences and the test found deadline-related errors.
Source: ThorstenMeyerAI.com
