TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Memdoor’s authors report that a workflow using five Qwen3.8-27B model roles scored 9.8 out of 10 across five runs on each of the first 10 problems in their LiveCodeBench Hard list. That was close to their own harness’s 9.6, but the authors say the small sample does not establish a broad advantage, and Fable 5 was not rerun in their harness.
Memdoor’s authors report that a workflow assigning five roles to Qwen3.8-27B averaged 9.8 out of 10 on five runs of each of the first 10 problems in their LiveCodeBench Hard list, slightly above the 9.6 average they report for their own harness on the same sample. The result is a limited comparison, not evidence that the workflow generally outperforms Fable 5: the authors say their benchmark covered only 10 problems, and Fable 5 was not run in their harness.
The workflow divides work among a planner, two ideators, a solver, a manager and a repair worker. Each role starts with fresh context and communicates through files on disk rather than relying on a shared chat history. The steps are specified in eight YAML files, and the workflow itself does not name a particular model; the benchmark used Qwen3.8-27B for its model calls, according to the project report.
For each problem, the workflow produces a plan and two proposals in parallel, then creates a solution, runs it against public samples, reviews the test output and can request a repair. A final verification command determines whether the run passes. The score is based on hidden tests, while the workflow’s test step checks the supplied public samples. The repository says it retained all 50 runs, including requests and replies, programs, grader output and cost records.
The reported figures are 9.8 for the workflow, 9.6 for the authors’ harness and 8.0 for Fable 5 on this 10-problem comparison. The Fable 5 figure comes from the authors’ files, rather than a new run in Memdoor’s harness. The authors report a median run time of 40 minutes and a cost of $0.39 per run, or $35.79 across all 50 runs. They say the workflow used OpenRouter, not a local machine.
What Multi-Role Runs Can Show
The comparison speaks to a practical question in AI-assisted programming: whether dividing a problem among model roles can improve results compared with a single-pass approach. Memdoor’s workflow adds planning, independent algorithm proposals, review and repair around the same named model. Its reported score is close to the authors’ harness result, suggesting that this structured process can reproduce that result on the selected tasks.
That is relevant to developers evaluating agent workflows and inference costs, but the figures do not establish that five roles are better than one across coding benchmarks. The test set is small, the scores are close, and the comparison with Fable 5 uses previously reported data rather than a controlled run in the same setup. The authors themselves describe the 9.8 and 9.6 results as being within noise. The run time and per-run cost also matter: this is a multi-step process, not simply a single model response.
Products Worth Considering
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Set Up
The report is presented as a Memdoor implementation of the GVS5H method attributed to Lee (2026). The authors compare their workflow and their own harness using Qwen3.8-27B, with their prompts, loop and a 128K reply cap. They say they regraded the results with a corrected grader and provide result files in the repository. Their full-list figures are different from this smaller check: the report cites 92.4 for Fable 5 and 90.4 for their harness across 100 problems, based on their patched verdicts file.
The reported workflow result covers only the first 10 problems in the authors’ LiveCodeBench Hard list, with five passes per problem. The authors say the contests date from spring 2025 and that possible training-data contamination has not been ruled out for any of the compared results. These limits make the result best read as a reproducibility and workflow check, rather than a new broad leaderboard claim.
“9.8 and 9.6 are within noise.”
— Memdoor authors
Limits of the Ten-Problem Check
The available report does not show whether the result holds across the authors’ full 100-problem set, other coding benchmarks or different model versions. Nor does it provide a same-harness rerun of Fable 5: that score was taken from the authors’ existing files. The comparison therefore combines results with different evaluation histories, even though the Memdoor authors say they used their corrected grader for the reported regrading.
Possible benchmark contamination also remains unresolved, according to the authors, because the contests date from spring 2025. The report describes two cases in which a correct program was overwritten and later recovered, and one problem where the manager introduced a wrong counterexample and the team rewrote a correct program. Those examples point to workflow failure modes, but the source does not provide enough information to determine how often such errors would occur on a larger evaluation.
Reproduction and Broader Testing
The immediate next step for readers assessing the claim is to inspect the project’s 50-run records and parity notes, which the authors say include each request, reply, program, grader output and cost. The repository also describes how to run the workflow with a task file and paired input-output samples. Those instructions document reproduction of the workflow; they do not by themselves validate its benchmark score independently.
A clearer comparison would require testing the workflow and alternatives on the same larger problem set, under the same model access, prompts, grading rules and run budget. Until such results are available, Memdoor’s report supports a limited finding: its five-role Qwen3.8-27B setup came close to the authors’ harness on this 10-problem sample. The source material does not announce a date for further testing or a broader evaluation.
Key Questions
What did the Memdoor workflow score?
The authors report an average of 9.8 out of 10 across five runs on each of the first 10 problems in their LiveCodeBench Hard list.
Did the workflow beat Fable 5 in a direct test?
Not in a new, same-harness comparison. The Fable 5 score of 8.0 came from the authors’ existing files; Memdoor’s report says Fable 5 was not run in its harness.
How does the workflow use five model roles?
It assigns planning, two parallel idea-generation roles, solving, management and repair tasks across separate steps. The agents use fresh context and communicate through files on disk.
Does the result establish a general coding advantage?
No. The authors describe the sample as only 10 problems and say the 9.8 and 9.6 scores are within noise. They also have not ruled out possible contamination from the contests’ spring 2025 release period.
Source: github.com
