Do transformers need three projections? Systematic study of QKV variants
Transformers utilize a query, key, and value (QKV) attention formulation that is crucial for AI tasks. The study investigates the individual contributions of these three projections and the effects of omitting any of them.
arxiv.org
2 min
6/4/2026
A sleep-like consolidation mechanism for LLMs
Transformer-based large language models struggle with long-context tasks due to poor scaling of their attention mechanism. Implementing a sleep-like consolidation mechanism allows models to convert recent context into persistent fast weights while clearing their key-value cache.
arxiv.org
2 min
5/26/2026
Do transformers need three projections? Systematic study of QKV variants
Transformers utilize a query, key, and value (QKV) attention formulation that is crucial for AI tasks. The study investigates the individual contributions of these three projections and the effects of omitting any of them.
arxiv.org
2 min
6/4/2026
A sleep-like consolidation mechanism for LLMs
Transformer-based large language models struggle with long-context tasks due to poor scaling of their attention mechanism. Implementing a sleep-like consolidation mechanism allows models to convert recent context into persistent fast weights while clearing their key-value cache.
arxiv.org
2 min
5/26/2026
Do transformers need three projections? Systematic study of QKV variants
Transformers utilize a query, key, and value (QKV) attention formulation that is crucial for AI tasks. The study investigates the individual contributions of these three projections and the effects of omitting any of them.
arxiv.org
2 min
6/4/2026
A sleep-like consolidation mechanism for LLMs
Transformer-based large language models struggle with long-context tasks due to poor scaling of their attention mechanism. Implementing a sleep-like consolidation mechanism allows models to convert recent context into persistent fast weights while clearing their key-value cache.
arxiv.org
2 min
5/26/2026
No more articles to load