🔍 Read the full analysis: 180B-Parameter MoE Inference: Working Within 32GB Of RAM With Transformers on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Hugging Face developer has proposed an experimental Transformers option that streams selected mixture-of-experts weights from storage into a bounded device cache. In a reported OLMoE test, it reached 34.6 tokens per second versus 38.1 for the standard Transformers path; results for the motivating 180B-model, 32 GB Mac use case have not been reported.
A Hugging Face developer has proposed an experimental SSD-backed expert-streaming option for Transformers, as detailed in the original analysis, aiming to let mixture-of-experts models run when their weights exceed available memory. In a reported test on OLMoE-1B-7B, the method generated 34.6 tokens per second, compared with 38.1 for the standard Transformers path; the source does not report a completed test of a 180B model on a 32 GB Mac.
The approach targets a memory constraint in mixture-of-experts (MoE) models. Each token is routed through a subset of a model’s experts rather than all of them. The developer says experts account for more than 90% of the model weights, creating the possibility of loading only selected experts into faster memory as generation proceeds. The proposed system keeps a bounded cache on the device and fetches other experts from storage when needed.
The developer first tested a simple implementation that ran experts one at a time in a Python loop. Even with experts already cached, that version reached 10.1 tokens per second, compared with about 32 for vanilla Transformers. The revised design stores experts in a shared, preallocated device tensor and uses a table to map expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together, reaching 34.6 tokens per second against the reported 38.1 baseline.
The source says the change was submitted to Transformers’ from_pretrained flow in pull request #50717 and points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work involving Transformers, a model-specific package and Metal kernels, with AI used to help build the code. The developer says their role included selecting experiments, examining profiles and deciding which implementation to keep.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
The Promise of Local MoE Inference
If it works on models that exceed available memory, expert streaming could let people run some larger MoE models on a laptop or a single GPU without keeping every expert in fast memory. That could expand local inference options for developers and researchers who are limited by device memory rather than by whether the model can be loaded in principle.
The reported result is encouraging but narrow: it shows that a particular cache design can approach the baseline speed on the tested OLMoE setup. It does not establish that storage-backed inference will remain fast when experts must be fetched repeatedly, or that the method will work at similar speeds on a 180B model with 32 GB of unified memory. Storage speed, cache size and how often routing changes the required experts could all affect performance.
Products Worth Considering

Apple MacBook Air 13-inch with M5 Chip, 16GB Memory, 512GB SSD, Liquid Retina Display, and 12MP Center Stage Camera – Midnight

DJI Mini 4K Drone with 4K UHD Camera, Under 249 g

Jackery Solar Generator 1000 V2 with 100W Solar Panel
As an affiliate, we earn on qualifying purchases.
How the Cache Design Changed
The developer’s comparison, as described in the source, also includes tests of other MoE inference approaches using Qwen3.6-35B-A3B on an A10G GPU with 24 GB of memory and ample system RAM. In that setup, Strata reportedly decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures apply to that model and hardware; they are not direct predictions for a laptop or for the proposed Transformers change.
The source characterizes the alternatives as handling offloaded experts differently. It says llama.cpp can run experts on the CPU when they fit in RAM, while performance may fall under pressure on the system page cache. It describes vLLM’s CPU offload as reading offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate as moving an entire layer’s expert bank before processing it. The experimental proposal instead aims to fetch only the experts selected by the router into a bounded cache and retain grouped expert computation.
“34.6 tokens per second, versus 38.1 for vanilla Transformers.”
— The developer, on the revised OLMoE test
The 32 GB Mac Test Is Missing
The source frames running a large MoE model on a 32 GB Mac as a practical goal, but gives no measured result for the proposed implementation on that device or on the cited 180B-model use case. It also does not report storage type, cache size, prompt conditions or repeated runs for the OLMoE speed comparison, limiting what readers can infer about performance across workloads.
The status of pull request #50717 is also unspecified: the source says it was submitted, not whether it was accepted, merged or released. It does not establish how the cache performs when the routed expert set changes rapidly, or provide a same-hardware comparison with Strata. The reported OLMoE result, on its own, does not show how the system performs when it must repeatedly fetch experts from an SSD.
Pull Request and Larger-Model Results
The next milestone is the status of Transformers pull request #50717 and whether the proposed option becomes available in a published version. The linked Hugging Face Space offers a way to inspect the experimental code, according to the source, but code availability is not the same as acceptance into Transformers or validation across devices.
To evaluate the use case that motivated the work, further testing would need to report generation speed and memory use on the intended large MoE model and a 32 GB Mac, along with storage type, cache settings and workload details. Repeated runs and a comparison with Strata under matching conditions would help establish whether the approach can preserve speed when model experts must be streamed from storage.
Key Questions
What did the Transformers experiment change?
It proposes loading routed MoE experts from storage into a bounded device cache, rather than keeping all expert weights in fast memory. A shared preallocated tensor lets the existing grouped matrix multiplication kernel process selected experts together.
How fast was the reported test?
On OLMoE-1B-7B, the developer reported 34.6 tokens per second, compared with 38.1 for the standard Transformers path. The source does not provide full benchmark conditions or repeated-run results.
Has the 180B model been shown running on a 32 GB Mac?
No such result is reported in the source. The 180B-model, 32 GB Mac scenario is presented as a target, not as a completed benchmark.
Is the feature part of a released Transformers version?
The source says the developer submitted the change as pull request #50717. It does not say whether the request was accepted, merged or released.
Primary source: Hugging Face · via ThorstenMeyerAI.com
