caching-strategies
Latest stories covering caching-strategies, with summaries and links to related coverage.
1 articles · page 1 of 1
Connected topics
Why harnesses matter for energy (and token) efficiency
research
15 min read
Neuralwatt found that the software harnesses used by AI agents can materially affect GPU energy consumption by determining whether long prompts reuse a model server’s key-value cache. Across customer traffic from September 8 to October 7, 2026, major clients had 1.6% to 11.1% uncached input. A session with 92% cached input requires about 2.7 times as much prompt-processing work as one with 97% cached input, despite both using the same model and conversation. For comparable 50,000-token requests measured October 1–7, mostly uncached prompts used roughly twice the energy of fully cached prompts; GLM-5.3 used about 2.5 times as much. In sessions exceeding 128,000 tokens, about three turns per 1,000 re-read more than half the prompt, and those turns represented 75% of re-read tokens at that depth. Each re-read about 230,000 tokens on average, roughly the prefill work of 600 normal turns. Neuralwatt’s lab tests attributed all observed cache breaks to harnesses editing earlier prompt content. Changes to reasoning effort, tool lists, dates, skills, reminders and compaction can force servers to recompute prompts from the first altered token. Neuralwatt recommends append-only prompt histories, stable system prompts and tool lists, and settling settings before sessions become long.