Story
You could have invented transformers from memory
1 report
1 website
1 discussion
Latest report
Overview
Transformer components can be modeled as associative memory stores that retrieve value vectors from similar key vectors, with recall quality limited by interference from other stored memories. For random unit-length keys, a simple outer-product linear store has capacity near the key dimension: 64-dimensional keys can hold about 64 memories before interference matches the desired signal. An optimal linear regression-based store whitens key correlations, recalls up to 64 linearly independent memories exactly, and reaches capacity near 128 memories. Nonlinear MLP activations suppress low-similarity keys and increase estimated capacity. A ReLU MLP reaches about 128 memories with 64-dimensional keys, while squared ReLU and similarly configured SwiGLU MLPs reach roughly 2,700 to 2,900; cubed ReLU reaches about 35,000. Feedforward layers in trained transformers have been found to act like key-value memories, with learned keys responding to recognizable input patterns. Softmax self-attention applies exponential weighting to similarities and uses context tokens rather than trained parameters as its keys and values. Under the stated random-vector model, 64-dimensional self-attention with sharpness β = 8 has estimated capacity of about 1.2 million memories. Increasing attention-logit scaling with context length, including logarithmic schemes used by Qwen and YaRN, can maintain low interference beyond 2 million distinct key-value pairs in the described experiments. These estimates measure random-vector recall in a single layer rather than total information learned from training data.
Summary from alexwang.ai.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · alexwang.ai
Historical count: 10 · 10 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 10 sampled comments from this thread. Analyzed 2026-10-17.
The thread mostly debated pedagogy rather than the claim that transformers could be reconstructed from familiar ideas. Several readers praised the article for building intuition from linear algebra and associative memory, arguing that this is exactly the right foundation for understanding the high-dimensional semantic spaces behind LLMs. The connection between transformers and Hopfield networks landed especially well: attention can retrieve a useful stored pattern in a single operation, where classical Hopfield-style recall requires iterative settling. One reader also extended the article’s framing toward a broader question about intelligence as compression, with knowledge packed into weights and relationships. The sharp criticism was not that the technical account was wrong, but that it assumed too much mathematical fluency. Readers who lost the thread wanted concrete worked examples, pseudocode, and clearer definitions of the objects in the notation; one specifically remained unsure how keys and values correspond to an LLM’s context. Others pushed back that the mathematics was already approachable for its intended audience, and that AI fundamentals cannot be made intuitive by stripping away the mathematical language that expresses them. Visual polish, including an animated illustration, was noticed as evidence of unusual care. Overall, commenters saw a valuable explanation whose accessibility depends heavily on prior linear-algebra and communications-theory background.
The central dispute was whether the mathematical presentation teaches transformers effectively or unnecessarily excludes programmers. Critics argued that unexplained notation and no concrete LLM walkthrough leave essential concepts like attention’s keys and values opaque; defenders argued that linear algebra is the indispensable route to real intuition, and that an article need not flatten itself to fit every software reader.
- Praise: Framing attention as one-step associative-memory retrieval gives transformers a crisp conceptual advantage over iterative Hopfield recall, making their effectiveness feel less mysterious.
- Praise: Starting from linear algebra was praised as the honest way to build intuition for LLM semantic spaces rather than hiding the machinery behind hand-wavy analogies.
- Praise: The article sparked productive links between transformer memory, compression, and stored knowledge in weights, broadening the discussion beyond a recipe for attention.
- Praise: Careful visual presentation, including the animated bird, made readers feel the explanation had been crafted rather than dumped from a textbook.
- Concern: The notation arrives without enough concrete grounding, so readers cannot tell whether symbols denote scalars, vectors, sets, or matrices when the argument becomes important.
- Concern: The jump from abstract associative memory to LLM attention leaves a key practical question unanswered: what token-context keys and retrieved values actually represent.
- Concern: Calling the mathematics accessible frustrated readers who want an example-driven bridge before being asked to infer transformer behavior from equations alone.