attention-mechanisms
Latest stories covering attention-mechanisms, with summaries and links to related coverage.
4 articles · page 1 of 1
Connected topics
You Could Have Come Up with Kimi Delta Attention
research
13 min read
Kimi Delta Attention utilizes bra-ket notation for clarity in mathematical derivations, where column vectors represent states, row vectors represent duals, numbers represent scalars, and matrices represent linear transformations. In this notation, vectors are oriented to the right by default, while keys are oriented to the left within the linear attention structure.
Kimi Linear: An Expressive, Efficient Attention Architecture
research
2 min read
Kimi Linear is a hybrid linear attention architecture that outperforms full attention in short-context, long-context, and reinforcement learning scenarios. It utilizes Kimi Delta Attention to achieve this efficiency.
Do transformers need three projections? Systematic study of QKV variants
research
2 min read
Transformers utilize a query, key, and value (QKV) attention formulation that is crucial for AI tasks. The study investigates the individual contributions of these three projections and the effects of omitting any of them.
A sleep-like consolidation mechanism for LLMs
research
2 min read
Transformer-based large language models struggle with long-context tasks due to poor scaling of their attention mechanism. Implementing a sleep-like consolidation mechanism allows models to convert recent context into persistent fast weights while clearing their key-value cache.