180B-Parameter MoE Inference: Working Within 32GB Of RAM With Transformers
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: 180B-Parameter MoE Inference: Working Within 32GB Of RAM With Transformers on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer has proposed an experimental Transformers option that streams selected mixture-of-experts weights from storage into a bounded device cache. In a reported OLMoE test, it reached 34.6 tokens per second versus 38.1 for the standard Transformers path; results for the motivating 180B-model, 32 GB Mac use case have not been reported.

A Hugging Face developer has proposed an experimental SSD-backed expert-streaming option for Transformers, as detailed in the original analysis, aiming to let mixture-of-experts models run when their weights exceed available memory. In a reported test on OLMoE-1B-7B, the method generated 34.6 tokens per second, compared with 38.1 for the standard Transformers path; the source does not report a completed test of a 180B model on a 32 GB Mac.

The approach targets a memory constraint in mixture-of-experts (MoE) models. Each token is routed through a subset of a model’s experts rather than all of them. The developer says experts account for more than 90% of the model weights, creating the possibility of loading only selected experts into faster memory as generation proceeds. The proposed system keeps a bounded cache on the device and fetches other experts from storage when needed.

The developer first tested a simple implementation that ran experts one at a time in a Python loop. Even with experts already cached, that version reached 10.1 tokens per second, compared with about 32 for vanilla Transformers. The revised design stores experts in a shared, preallocated device tensor and uses a table to map expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together, reaching 34.6 tokens per second against the reported 38.1 baseline.

The source says the change was submitted to Transformers’ from_pretrained flow in pull request #50717 and points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work involving Transformers, a model-specific package and Metal kernels, with AI used to help build the code. The developer says their role included selecting experiments, examining profiles and deciding which implementation to keep.

At a glance
reportWhen: Experimental proposal; pull request #50…
The developmentA Hugging Face developer proposed SSD-backed expert streaming for Transformers and reported near-baseline OLMoE speed, while the larger 180B-model test remains unmeasured in the source.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

The Promise of Local MoE Inference

If it works on models that exceed available memory, expert streaming could let people run some larger MoE models on a laptop or a single GPU without keeping every expert in fast memory. That could expand local inference options for developers and researchers who are limited by device memory rather than by whether the model can be loaded in principle.

The reported result is encouraging but narrow: it shows that a particular cache design can approach the baseline speed on the tested OLMoE setup. It does not establish that storage-backed inference will remain fast when experts must be fetched repeatedly, or that the method will work at similar speeds on a 180B model with 32 GB of unified memory. Storage speed, cache size and how often routing changes the required experts could all affect performance.

Products Worth Considering

Apple MacBook Air 13-inch with M5 Chip, 16GB Memory, 512GB SSD, Liquid Retina Display, and 12MP Center Stage Camera – Midnight

Apple MacBook Air 13-inch with M5 Chip, 16GB Memory, 512GB SSD, Liquid Retina Display, and 12MP Center Stage Camera – Midnight

As an affiliate, we earn on qualifying purchases.

How the Cache Design Changed

The developer’s comparison, as described in the source, also includes tests of other MoE inference approaches using Qwen3.6-35B-A3B on an A10G GPU with 24 GB of memory and ample system RAM. In that setup, Strata reportedly decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures apply to that model and hardware; they are not direct predictions for a laptop or for the proposed Transformers change.

The source characterizes the alternatives as handling offloaded experts differently. It says llama.cpp can run experts on the CPU when they fit in RAM, while performance may fall under pressure on the system page cache. It describes vLLM’s CPU offload as reading offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate as moving an entire layer’s expert bank before processing it. The experimental proposal instead aims to fetch only the experts selected by the router into a bounded cache and retain grouped expert computation.

“34.6 tokens per second, versus 38.1 for vanilla Transformers.”

— The developer, on the revised OLMoE test

The 32 GB Mac Test Is Missing

The source frames running a large MoE model on a 32 GB Mac as a practical goal, but gives no measured result for the proposed implementation on that device or on the cited 180B-model use case. It also does not report storage type, cache size, prompt conditions or repeated runs for the OLMoE speed comparison, limiting what readers can infer about performance across workloads.

The status of pull request #50717 is also unspecified: the source says it was submitted, not whether it was accepted, merged or released. It does not establish how the cache performs when the routed expert set changes rapidly, or provide a same-hardware comparison with Strata. The reported OLMoE result, on its own, does not show how the system performs when it must repeatedly fetch experts from an SSD.

Pull Request and Larger-Model Results

The next milestone is the status of Transformers pull request #50717 and whether the proposed option becomes available in a published version. The linked Hugging Face Space offers a way to inspect the experimental code, according to the source, but code availability is not the same as acceptance into Transformers or validation across devices.

To evaluate the use case that motivated the work, further testing would need to report generation speed and memory use on the intended large MoE model and a 32 GB Mac, along with storage type, cache settings and workload details. Repeated runs and a comparison with Strata under matching conditions would help establish whether the approach can preserve speed when model experts must be streamed from storage.

Key Questions

What did the Transformers experiment change?

It proposes loading routed MoE experts from storage into a bounded device cache, rather than keeping all expert weights in fast memory. A shared preallocated tensor lets the existing grouped matrix multiplication kernel process selected experts together.

How fast was the reported test?

On OLMoE-1B-7B, the developer reported 34.6 tokens per second, compared with 38.1 for the standard Transformers path. The source does not provide full benchmark conditions or repeated-run results.

Has the 180B model been shown running on a 32 GB Mac?

No such result is reported in the source. The 180B-model, 32 GB Mac scenario is presented as a target, not as a completed benchmark.

Is the feature part of a released Transformers version?

The source says the developer submitted the change as pull request #50717. It does not say whether the request was accepted, merged or released.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Turn Home Boxing Sessions Into Measurable Rounds

An IdeaNavigator AI product concept proposes phone-camera punch metrics for home boxers, with retention and paid-tier conversion still to be tested.

Anthropic’s Claude Claim: What We Know About Chinese AI Firms

A report attributes an allegation about Chinese AI firms’ use of Claude to Anthropic, but the available headline provides no names or evidence.

AI-Powered Social Media Content Generators: A Halloween Guide

Learn how AI social media generators work, where they save time, and how to check drafts for accuracy, privacy, and a human voice.

What Consultants Can Learn From Tracking Scope Creep

A proposed tool would compare client requests with fixed-price contracts. Its value remains unproven until a three-month test measures results.