llms code-generation developer-tools ai-performance

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

blog.can.ac

February 12, 2026

8 min read

🔥🔥🔥🔥🔥

73/100

Summary

Improving coding performance in 15 language models can be achieved by changing the harness used, rather than the models themselves. The harness affects the efficiency and effectiveness of the models, highlighting its role as a critical factor in AI coding capabilities.

Key Takeaways

The performance of large language models (LLMs) in coding tasks is significantly influenced by the harness used, rather than just the model itself.
Different models exhibit varying patch failure rates due to their inability to conform to specific input structures, with Grok 4 and GLM-4.7 showing failure rates of 50.7% and 46.2%, respectively.
The choice of edit format can drastically affect model performance, as demonstrated by Aider's benchmarks, which showed GPT-4 Turbo's success rate increasing from 26% to 59% based on format choice.
There is no consensus on the best editing solution for LLMs, as evidenced by the Diff-XYZ benchmark, which found no single edit format that dominates across all models and use cases.

Read original article

View story →

Coverage & discussions

Hacker News · blog.can.ac

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Positive

Positives

Improving agent harnesses can significantly enhance the effectiveness of existing models, suggesting that design optimizations may yield greater returns than training new models.
The article highlights the potential for harness-level improvements to reduce token waste, which could lead to more efficient AI applications.
The distinction between the model and the harness emphasizes that effective AI deployment requires careful engineering at the interface, not just advanced model capabilities.
The CORE benchmark example illustrates how switching harnesses can dramatically improve model performance, reinforcing the importance of harness design.