You could have invented transformers from memory
alexwang.ai
October 16, 2026
21 min read
51/100
Summary
Transformer components can be modeled as associative memory stores that retrieve value vectors from similar key vectors, with recall quality limited by interference from other stored memories. For random unit-length keys, a simple outer-product linear store has capacity near the key dimension: 64-dimensional keys can hold about 64 memories before interference matches the desired signal. An optimal linear regression-based store whitens key correlations, recalls up to 64 linearly independent memories exactly, and reaches capacity near 128 memories. Nonlinear MLP activations suppress low-similarity keys and increase estimated capacity. A ReLU MLP reaches about 128 memories with 64-dimensional keys, while squared ReLU and similarly configured SwiGLU MLPs reach roughly 2,700 to 2,900; cubed ReLU reaches about 35,000. Feedforward layers in trained transformers have been found to act like key-value memories, with learned keys responding to recognizable input patterns. Softmax self-attention applies exponential weighting to similarities and uses context tokens rather than trained parameters as its keys and values. Under the stated random-vector model, 64-dimensional self-attention with sharpness β = 8 has estimated capacity of about 1.2 million memories. Increasing attention-logit scaling with context length, including logarithmic schemes used by Qwen and YaRN, can maintain low interference beyond 2 million distinct key-value pairs in the described experiments. These estimates measure random-vector recall in a single layer rather than total information learned from training data.
Key Takeaways
- A basic outer-product associative memory with 64-dimensional keys reaches an interference-to-signal ratio of one at roughly 64 stored random memories.
- An optimal linear memory store can recall linearly independent memories exactly up to the key dimension and reaches estimated capacity near twice that dimension.
- Squared ReLU and SwiGLU-style MLP memory stores have estimated capacities around 2,700 to 2,900 memories for 64-dimensional keys, far above linear stores.
- Self-attention stores key-value pairs from the token context rather than solely in model parameters, so its memory size grows with context length.
- With 64-dimensional random keys and β = 8, softmax self-attention has an estimated capacity of about 1.2 million memories before interference matches the signal.
What the discussion said
The thread mostly debated pedagogy rather than the claim that transformers could be reconstructed from familiar ideas. Several readers praised the article for building intuition from linear algebra and associative memory, arguing that this is exactly the right foundation for understanding the high-dimensional semantic spaces behind LLMs. The connection between transformers and Hopfield networks landed especially well: attention can retrieve a useful stored pattern in a single operation, where classical Hopfield-style recall requires iterative settling. One reader also extended the article’s framing toward a broader question about intelligence as compression, with knowledge packed into weights and relationships. The sharp criticism was not that the technical account was wrong, but that it assumed too much mathematical fluency. Readers who lost the thread wanted concrete worked examples, pseudocode, and clearer definitions of the objects in the notation; one specifically remained unsure how keys and values correspond to an LLM’s context. Others pushed back that the mathematics was already approachable for its intended audience, and that AI fundamentals cannot be made intuitive by stripping away the mathematical language that expresses them. Visual polish, including an animated illustration, was noticed as evidence of unusual care. Overall, commenters saw a valuable explanation whose accessibility depends heavily on prior linear-algebra and communications-theory background.
Where opinion split
The central dispute was whether the mathematical presentation teaches transformers effectively or unnecessarily excludes programmers. Critics argued that unexplained notation and no concrete LLM walkthrough leave essential concepts like attention’s keys and values opaque; defenders argued that linear algebra is the indispensable route to real intuition, and that an article need not flatten itself to fit every software reader.
Community Sentiment
Positives
- Framing attention as one-step associative-memory retrieval gives transformers a crisp conceptual advantage over iterative Hopfield recall, making their effectiveness feel less mysterious.
- Starting from linear algebra was praised as the honest way to build intuition for LLM semantic spaces rather than hiding the machinery behind hand-wavy analogies.
- The article sparked productive links between transformer memory, compression, and stored knowledge in weights, broadening the discussion beyond a recipe for attention.
- Careful visual presentation, including the animated bird, made readers feel the explanation had been crafted rather than dumped from a textbook.
Concerns
- The notation arrives without enough concrete grounding, so readers cannot tell whether symbols denote scalars, vectors, sets, or matrices when the argument becomes important.
- The jump from abstract associative memory to LLM attention leaves a key practical question unanswered: what token-context keys and retrieved values actually represent.
- Calling the mathematics accessible frustrated readers who want an example-driven bridge before being asked to infer transformer behavior from equations alone.
