model-performance
Latest stories covering model-performance, with summaries and links to related coverage.
3 stories · 3 reports · page 1 of 1
Connected topics
Why your local LLM feels dumber than it is
research
11 min read
Local LLM output can diverge from a model publisher’s results because hardware, inference engines, CUDA kernels, attention backends, quantization, sampling settings, and chat templates alter next-token calculations. The post recommends workload-representative benchmarks, including long-context, tool-calling, and domain-specific evaluations, rather than a few zero-shot prompts. KL divergence can measure differences between output probability distributions, but requires disclosed reference checkpoints, runtime environments, evaluation data, context lengths, sampling positions, direction, vocabulary handling, and aggregation methods to be interpretable. Tests used Qwen3.6-27B’s official BF16 checkpoint on an RTX PRO 6000 Blackwell GPU with a roughly 100,000-token real-world tool-calling context. FlashAttention 2, FlashInfer, and Triton attention backends produced identical results early in the prompt but later disagreed on top-ranked next tokens; repeated Triton runs were bit-identical, isolating the variation to prefill matrix operations. With BF16 weights, INT8 KV-cache quantization eventually recovered from a reproducible tool-calling error, while INT4 KV-cache quantization did not. In weight-quantization tests with BF16 KV cache, a W8A16 INT8 Qwen variant showed the closest token fidelity to BF16. NVIDIA’s NVFP4 variant reached about 50% top-token flips by 88,000 tokens, and both NVFP4 and AWQ W4A16 generated incorrect Cisco commands instead of the expected "show arp" command.
How much can you delegate to agents?
opinion
7 min read
Trusting AI agents to perform tasks autonomously depends more on the context and the specific use case rather than solely on the model's capabilities. Increased model performance does not automatically justify greater delegation of responsibilities without proper evaluation.
Better Models: Worse Tools
tool
10 min read
Newer Claude models, such as Opus 4.8, sometimes generate extra, invented fields in the nested edits[] array when calling Pi's edit tool. This results in mismatched arguments that cause Pi to reject the tool call and request a retry.