memory-models
Latest stories covering memory-models, with summaries and links to related coverage.
1 articles · page 1 of 1
Connected topics
You could have invented transformers from memory
opinion
21 min read
Transformer components can be modeled as associative memory stores that retrieve value vectors from similar key vectors, with recall quality limited by interference from other stored memories. For random unit-length keys, a simple outer-product linear store has capacity near the key dimension: 64-dimensional keys can hold about 64 memories before interference matches the desired signal. An optimal linear regression-based store whitens key correlations, recalls up to 64 linearly independent memories exactly, and reaches capacity near 128 memories. Nonlinear MLP activations suppress low-similarity keys and increase estimated capacity. A ReLU MLP reaches about 128 memories with 64-dimensional keys, while squared ReLU and similarly configured SwiGLU MLPs reach roughly 2,700 to 2,900; cubed ReLU reaches about 35,000. Feedforward layers in trained transformers have been found to act like key-value memories, with learned keys responding to recognizable input patterns. Softmax self-attention applies exponential weighting to similarities and uses context tokens rather than trained parameters as its keys and values. Under the stated random-vector model, 64-dimensional self-attention with sharpness β = 8 has estimated capacity of about 1.2 million memories. Increasing attention-logit scaling with context length, including logarithmic schemes used by Qwen and YaRN, can maintain low interference beyond 2 million distinct key-value pairs in the described experiments. These estimates measure random-vector recall in a single layer rather than total information learned from training data.