AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

Why harnesses matter for energy (and token) efficiency

Why harnesses matter for energy efficiency

neuralwatt.com

October 16, 2026

15 min read

🔥🔥🔥🔥🔥

44/100

Summary

Neuralwatt found that the software harnesses used by AI agents can materially affect GPU energy consumption by determining whether long prompts reuse a model server’s key-value cache. Across customer traffic from September 8 to October 7, 2026, major clients had 1.6% to 11.1% uncached input. A session with 92% cached input requires about 2.7 times as much prompt-processing work as one with 97% cached input, despite both using the same model and conversation. For comparable 50,000-token requests measured October 1–7, mostly uncached prompts used roughly twice the energy of fully cached prompts; GLM-5.3 used about 2.5 times as much. In sessions exceeding 128,000 tokens, about three turns per 1,000 re-read more than half the prompt, and those turns represented 75% of re-read tokens at that depth. Each re-read about 230,000 tokens on average, roughly the prefill work of 600 normal turns. Neuralwatt’s lab tests attributed all observed cache breaks to harnesses editing earlier prompt content. Changes to reasoning effort, tool lists, dates, skills, reminders and compaction can force servers to recompute prompts from the first altered token. Neuralwatt recommends append-only prompt histories, stable system prompts and tool lists, and settling settings before sessions become long.

Key Takeaways

Prompt-cache reuse reduces AI inference energy because cached tokens do not need to be recomputed during prompt prefill.
Neuralwatt measured mostly uncached 50,000-token requests using about twice the energy of fully cached requests, and about 2.5 times as much on GLM-5.3.
A small edit near the beginning of a long agent prompt can invalidate most of the cache and cause substantial latency and energy costs.
In Neuralwatt’s tests, OpenCode 2.x avoided a plan-mode reminder cache break found in OpenCode 1.x, while Pi and Kilo Code showed no equivalent automatic breaks.
Neuralwatt routes sessions back to servers holding their caches and said a more prefix-aware router would add only 0.1 to 0.2 percentage points of cache-hit rate beyond its existing routing.

What the discussion said

The thread treated the harness less as incidental agent plumbing and more as the layer that turns a stateless text generator into something that can work through a real codebase. Several commenters argued that tools, iterative control loops, and deliberate context selection let a model break a large task into manageable actions; a raw prompt-response interface simply cannot hold or navigate enough project state. At the same time, they stressed that a basic harness is deceptively small in code but quickly becomes difficult once tool use, context management, and training interactions enter the picture. The article author supplied concrete operational stakes: unstable prompt prefixes can make inference markedly more energy-intensive, and repeatedly rereading giant contexts can become absurdly expensive. Readers were surprised that popular systems still invalidate caches so readily. But the discussion resisted treating cache-hit rate as the north-star metric. Memory injection, compaction, tool-result changes, and fresh summaries may deliberately break a prefix because they remove stale material and give the model a cleaner shot at the task. The emerging consensus was that harness design should optimize task completion first, then use cache stability to reduce cost and latency without preserving irrelevant context merely to win an efficiency number.

Where opinion split

The sharp dispute is whether preserving prompt-cache hits should drive harness design. Efficiency advocates argue that avoidable prefix edits waste large amounts of GPU energy and make capable agents slower and costlier; skeptics argue that a pristine cache is worthless if it traps the model in bloated, distracting context, and that strategic resets can improve answer quality.

Read original article

Community Sentiment

Positive

Positives

Harnesses give code-oriented models an operational exosuit: tools and iterative actions let them search, edit, and manage codebase context that a bare chat interface cannot realistically handle.
Stable prompt prefixes can cut inference energy dramatically, so careful harness design turns the same model capability into faster, cheaper agent runs.
The reported energy measurements make cache behavior tangible: at very long contexts, needless rereads can cost orders of magnitude more than ordinary turns.
Commenters see substantial unrealized room in harness design, especially around context routing, tool loops, and making agents practical on large projects.

Concerns

Popular harnesses appear to invalidate caches through avoidable prompt mutations, turning routine context handling into unnecessary GPU-energy and latency costs.
Chasing cache-hit rates can degrade model output when it preserves irrelevant history; subagents and fresh task-specific context may be more accurate precisely because they discard it.
Harnesses become thorny quickly once memory systems, tool outputs, compaction, and training-aware behavior interact, so a simple agent loop does not guarantee robust AI performance.