Story
Why harnesses matter for energy (and token) efficiency
1 report
1 website
1 discussion
Latest report
Overview
Neuralwatt found that the software harnesses used by AI agents can materially affect GPU energy consumption by determining whether long prompts reuse a model server’s key-value cache. Across customer traffic from September 8 to October 7, 2026, major clients had 1.6% to 11.1% uncached input. A session with 92% cached input requires about 2.7 times as much prompt-processing work as one with 97% cached input, despite both using the same model and conversation. For comparable 50,000-token requests measured October 1–7, mostly uncached prompts used roughly twice the energy of fully cached prompts; GLM-5.3 used about 2.5 times as much. In sessions exceeding 128,000 tokens, about three turns per 1,000 re-read more than half the prompt, and those turns represented 75% of re-read tokens at that depth. Each re-read about 230,000 tokens on average, roughly the prefill work of 600 normal turns. Neuralwatt’s lab tests attributed all observed cache breaks to harnesses editing earlier prompt content. Changes to reasoning effort, tool lists, dates, skills, reminders and compaction can force servers to recompute prompts from the first altered token. Neuralwatt recommends append-only prompt histories, stable system prompts and tool lists, and settling settings before sessions become long.
Summary from neuralwatt.com.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · neuralwatt.com
Historical count: 4 · 4 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 4 sampled comments from this thread. Analyzed 2026-10-17.
The thread treated the harness less as incidental agent plumbing and more as the layer that turns a stateless text generator into something that can work through a real codebase. Several commenters argued that tools, iterative control loops, and deliberate context selection let a model break a large task into manageable actions; a raw prompt-response interface simply cannot hold or navigate enough project state. At the same time, they stressed that a basic harness is deceptively small in code but quickly becomes difficult once tool use, context management, and training interactions enter the picture. The article author supplied concrete operational stakes: unstable prompt prefixes can make inference markedly more energy-intensive, and repeatedly rereading giant contexts can become absurdly expensive. Readers were surprised that popular systems still invalidate caches so readily. But the discussion resisted treating cache-hit rate as the north-star metric. Memory injection, compaction, tool-result changes, and fresh summaries may deliberately break a prefix because they remove stale material and give the model a cleaner shot at the task. The emerging consensus was that harness design should optimize task completion first, then use cache stability to reduce cost and latency without preserving irrelevant context merely to win an efficiency number.
The sharp dispute is whether preserving prompt-cache hits should drive harness design. Efficiency advocates argue that avoidable prefix edits waste large amounts of GPU energy and make capable agents slower and costlier; skeptics argue that a pristine cache is worthless if it traps the model in bloated, distracting context, and that strategic resets can improve answer quality.
- Praise: Harnesses give code-oriented models an operational exosuit: tools and iterative actions let them search, edit, and manage codebase context that a bare chat interface cannot realistically handle.
- Praise: Stable prompt prefixes can cut inference energy dramatically, so careful harness design turns the same model capability into faster, cheaper agent runs.
- Praise: The reported energy measurements make cache behavior tangible: at very long contexts, needless rereads can cost orders of magnitude more than ordinary turns.
- Praise: Commenters see substantial unrealized room in harness design, especially around context routing, tool loops, and making agents practical on large projects.
- Concern: Popular harnesses appear to invalidate caches through avoidable prompt mutations, turning routine context handling into unnecessary GPU-energy and latency costs.
- Concern: Chasing cache-hit rates can degrade model output when it preserves irrelevant history; subagents and fresh task-specific context may be more accurate precisely because they discard it.
- Concern: Harnesses become thorny quickly once memory systems, tool outputs, compaction, and training-aware behavior interact, so a simple agent loop does not guarantee robust AI performance.