🔍 Read the full analysis: The Best AI Model Setup For My News Pipeline Isn’t My Current One on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer reports testing 71 model configurations on real news-writing tasks from September 8 to October 13, 2026. GPT-6 Astra scored highest, but at about 100 times GPT-6 Luna’s reported per-article cost; the test also found no reliable benefit from higher reasoning effort and identified serious quality problems in two fallback models.
Thorsten Meyer says GPT-6 Astra produced the strongest articles in a test of 71 AI model configurations for his news pipeline, but he is keeping the much cheaper GPT-6 Luna in production. The benchmark, published October 13, compared models on news articles generated from real sources and found that Astra’s reported cost was about 100 times Luna’s per article.
Meyer’s test used his production prompt with two real news sources: one 6,340 characters long and one 619 characters long. Each configuration generated English and German versions. Two AI judges scored articles on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also assessed for language quality. Meyer says the leading configurations then underwent a robustness run across 10 additional articles, mostly based on short sources.
GPT-6 Astra scored 4.00 out of 5 in both languages in the initial test and achieved 3.93 across 12 articles in the robustness run. Meyer reports that its source-fidelity score was 5.00 in the standard test. GPT-6 Luna scored 3.67 in English and 3.68 in German initially, then 3.67 in the robustness run, with source fidelity of 4.71. Meyer says every Luna article in the test produced output his pipeline could use after its normal repair step.
The reported trade-off was cost and speed: Astra averaged about $0.12 per article and 50 seconds, while Luna cost roughly $0.0012 to $0.0013 and took 17 seconds. Meyer said he decided on October 7 that Astra was too expensive for routine news production. These figures come from his test and operating experience; the source does not provide independent verification of the costs.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
The Cost of Higher Article Scores
The results address a practical choice for publishers using AI to produce articles at volume: whether to select the model with the highest benchmark score or one that meets a quality threshold at lower cost. In Meyer’s reported results, Astra performed better, but Luna’s per-article cost was far lower and its articles remained usable in the test. That makes the decision dependent on a publisher’s own quality requirements, budget and tolerance for errors, rather than a general model ranking.
The reported fallback failures also matter because a pipeline may route work to a backup model when its primary service is unavailable. Meyer says GPT-4.1 nano produced fabricated quotes and received a source-fidelity score of 1.00, while locally run gpt-oss-120b produced no valid articles in four attempts. A fallback that generates fluent but unsupported material can create a publication risk, not just a temporary drop in writing quality.
The test is a single publisher’s evaluation, not a broad independent comparison of every model or news workflow. Its findings can inform other teams’ testing, but they do not establish that Luna is the best choice for publishers with different prompts, sources, languages or review processes.
AI model training and benchmarking tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Testing Models on Real News Tasks
Meyer’s central point is that broad capability rankings do not necessarily predict performance on a specific production task. He writes that he uses public leaderboards but says they cannot answer which model should write the articles published by his sites. His comparison instead used the prompt and sources from his own workflow.
The sample combined a relatively detailed product filing with a short political news item. Meyer says most of his news begins with short sources, making the shorter item a more representative test for his pipeline. He also set a rule that score gaps under about 0.3 points across the two initial test articles should be treated as noise, and says the robustness run provides a more reliable ranking.
Among other results, Meyer reports that GPT-6 Luna’s six reasoning-effort settings showed no consistent quality advantage over the production setting, labeled “none.” Maximum effort cost about 12 times as much and took about 12 times as long as none, according to his measurements. He also found Claude Opus 5.5, which he says he uses for software development, scored below Luna as a news writer in this test. Those are task-specific results from Meyer’s evaluation, not general judgments about the models.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the October 13 report
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Benchmark
The findings come from one publisher’s prompt, workflow and source set. Although Meyer says the strongest configurations were tested on 10 additional articles, the report does not establish how the rankings would change with other publishers’ prompts, longer investigations, breaking-news material, additional languages or human editing.
The article describes two AI systems as judges and says they saw the original source, but the provided material does not explain how judge disagreement was handled or whether human editors independently reviewed the scores. Meyer reports 95% bootstrap intervals for the robustness run, but the results remain an evaluation of a limited sample rather than a universal ranking. It is also unclear from the report how the selected production model will be monitored over a longer period, or whether model updates will change its cost or performance.
As an affiliate, we earn on qualifying purchases.
Monitoring the Production Route
Meyer says GPT-6 Luna is the model currently used for production and that he removed GPT-4.1 nano and gpt-oss-120b from the news route on October 7 after the test exposed fabricated quotes and invalid output. The report does not identify a replacement backup model, so the full failover arrangement remains unclear.
The immediate next step for other publishers considering these findings is to test candidate models on their own sources and production prompts, with checks for unsupported claims and output validity as well as writing quality. Meyer’s results suggest that any replacement or fallback should be evaluated separately rather than assumed to perform well because of its general ranking. No further testing date or planned model change is specified in the report.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which model scored highest in Meyer’s test?
GPT-6 Astra scored 4.00 out of 5 in English and German in the standard test and 3.93 across 12 articles in the robustness run, according to Meyer.
Why did Meyer keep GPT-6 Luna in production?
Meyer says Luna scored 3.67 in the robustness run, produced valid articles throughout the test and cost about $0.0012 to $0.0013 per article, compared with Astra’s reported cost of about $0.12.
Did higher reasoning effort improve the articles?
Meyer reports no reliable improvement across Luna’s six reasoning-effort settings. Maximum effort took and cost about 12 times as much as the production setting, while its score did not show a dependable gain.
What problems did the test find in fallback models?
Meyer says GPT-4.1 nano generated invented quotes and received a source-fidelity score of 1.00. Locally run gpt-oss-120b produced no valid articles in four attempts, including output-format problems and attributed quotes he said were invented.
Do the results establish which model is best for every publisher?
No. The test used Meyer’s own prompt, two source examples, English and German editions, and a limited robustness sample. Its scores describe that evaluation and do not establish performance for other workflows.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
