TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Alex Wang’s post presents linear layers, multilayer perceptrons and self-attention as forms of associative memory: systems that store key-value pairs and retrieve values from similar queries. Its stated capacity estimates rise from about 64 memories for a linear store to about 2,700 for an MLP and more than one million for self-attention with 64-dimensional keys; these are explanatory estimates, not a report of a new experiment or model release.
Alex Wang has published an explanation of how linear layers, multilayer perceptrons and self-attention can be understood as different kinds of memory store, offering an intuitive route from simple matrix operations to a central mechanism in transformers. The post estimates that, with 64-dimensional keys, these approaches can store roughly 64, 2,700 and more than one million memories, respectively, before interference becomes comparable to the signal being recalled.
Wang frames the problem around associative memory: storing a key-value pair and retrieving its value when given the key, or a similar query. A memory store can accumulate many pairs, but its practical capacity depends on whether it can recall the requested value without other stored pairs interfering. The post uses an interference-to-signal ratio to describe that trade-off.
In the simplest example, a linear layer stores each pair by adding the outer product of its value and key to a weight matrix. Querying the matrix combines the stored values, weighted by how similar each key is to the input. With random, normalized keys and values, Wang estimates that interference grows to the scale of the intended signal when the number of memories is about the key dimension. For 64-dimensional keys, that gives a rough capacity of about 64 memories.
The post then sketches ways to improve recall, moving from a linear store to an MLP and then to self-attention. Wang gives approximate capacities of about 2,700 memories for an MLP and more than one million for self-attention, again for 64-dimensional keys. The supplied source excerpt explains the linear-store calculation but does not include the full derivations behind the later estimates.
Why Memory Explains Attention
The account gives readers a way to connect transformer components to a concrete operation: writing key-value associations and retrieving them by similarity. Rather than treating attention as an isolated formula, it presents it as a more capable memory lookup than a basic linear map. That framing can help explain why transformers can use information from many tokens or stored examples when producing an answer.
The estimates also highlight a practical distinction between storage and reliable recall. A system may encode many associations in its parameters or activations, but those associations can compete when the system retrieves one. Wang’s comparison suggests that improving retrieval, not simply adding more stored information, is part of what makes more advanced mechanisms useful.
These figures should be read as heuristic capacity estimates under assumptions about keys and values, not as measured limits for deployed models. They illuminate the mathematical intuition in the post, but do not establish how many facts a particular language model can reliably remember in real use.
As an affiliate, we earn on qualifying purchases.
From Linear Layers to Attention
Wang begins with a familiar use of language models: asking one to recall rules for a board game. The anecdote introduces a broader question about how models retrieve information, but the post focuses on a simplified mathematical account rather than testing the model in that example. Its central object is a memory store made of key-value pairs, where similarity between a query and a key helps select the corresponding value.
For a linear layer, the weight matrix is formed by summing the outer products of stored values and keys. A query then receives the target value plus contributions from other memories whose keys are also similar. If keys were perfectly orthogonal, those other contributions would vanish, but a key space with dimension 64 cannot provide more than 64 mutually orthogonal directions. With more memories, the post argues, interference becomes harder to avoid.
Wang also notes a connection between learning and writing memories: a gradient-descent update to a linear layer changes its weights by an outer product, which can be interpreted as adding a key-value association. That observation links the memory analogy to training, while the post’s progression to MLPs and attention asks how changes in the memory mechanism can improve recall.
“The key property that characterizes a “good” memory store is its ability to recall what it stored.”
— Alex Wang, author of the post
self-attention neural network components
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Capacity Estimates
The supplied material does not state the post’s publication date, nor does it provide the complete explanation for the MLP and self-attention estimates. Those figures are presented by Wang as approximate capacities; the assumptions and detailed derivations for each are not available in the excerpt. They should not be treated as fixed limits across model designs or tasks.
The estimates also do not show how well a trained language model recalls real-world facts. Actual performance can depend on training data, representation, optimization, query wording and other factors not measured by the simplified random-key analysis. The post’s board-game example is an anecdote, not a controlled evaluation, and the source material does not report a new experiment or an independent validation of the capacity numbers.
As an affiliate, we earn on qualifying purchases.
What the Memory View Clarifies
The post’s next step is conceptual: it uses the linear memory store as a baseline, then develops how an MLP can reduce interference and how self-attention can retrieve from a much larger set of associations. Readers seeking the exact basis for the larger capacity estimates would need the full post, including its omitted sections and accompanying illustrations.
No product launch, benchmark release or follow-up experiment is identified in the source material. The immediate takeaway is the framework itself: compare memory mechanisms by how well they retrieve a stored value as the number of competing memories increases, while treating the stated capacities as estimates tied to the post’s assumptions.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main development?
Alex Wang published an explanatory post describing linear layers, MLPs and self-attention as associative memory stores, and comparing their estimated recall capacity.
What does associative memory mean here?
It means storing a key-value pair and retrieving its value when a query matches, or resembles, the key. The post uses similarity between vectors to model that lookup.
How many memories does the post estimate each method can hold?
For 64-dimensional keys, Wang gives rough estimates of about 64 for a linear store, about 2,700 for an MLP, and more than one million for self-attention. They are estimates, not universal measured limits.
Does the post show that language models can reliably recall a million facts?
No. The figure describes an estimated capacity in the post’s simplified memory framework. It does not establish that a particular model can reliably retrieve that many real-world facts.
What remains unconfirmed?
The provided excerpt does not include the full derivations for the MLP and self-attention figures or independent tests of those estimates. It also gives no publication date or new experimental results.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
