You Could Have Come Up with Kimi Delta Attention
blog.doubleword.ai
July 28, 2026
13 min read
🔥🔥🔥🔥🔥
60/100
Summary
Kimi Delta Attention utilizes bra-ket notation for clarity in mathematical derivations, where column vectors represent states, row vectors represent duals, numbers represent scalars, and matrices represent linear transformations. In this notation, vectors are oriented to the right by default, while keys are oriented to the left within the linear attention structure.
Key Takeaways
- Kimi Delta Attention (KDA) is a linear attention variant that simplifies the attention mechanism by removing the softmax normalization, allowing for linear scaling with sequence length.
- DeltaNet introduces a delta-rule correction for memory updates, addressing the interference caused by additive writes in linear attention models.
- The article outlines the progression from traditional softmax attention to advanced linear attention variants, culminating in the development of KDA.
- The derivation of KDA is presented using bra-ket notation to clarify the mathematical relationships involved in the attention mechanism.
Community Sentiment
Positives
- The author’s clear explanation of notation at the start is a breath of fresh air — it's not always the case, and it really helps demystify the content.
- Despite the complexity, some commenters see the underlying math as just adding and multiplying, showing that ML concepts can be accessible.
Concerns
- Many readers are left scratching their heads, with one stating, 'I don't even know most words they used in the paper' — not exactly welcoming for newcomers.
- The complex notation and jargon seem to create a barrier, leading to frustration among those trying to grasp the concepts.
