AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

Smaller, faster, safer: running Kimi and GLM at scale

blog.cloudflare.com

August 3, 2026

7 min read

🔥🔥🔥🔥🔥

53/100

Summary

Workers AI runs inference for Moonshot's Kimi K-series and Z.ai's GLM models on GPUs in Cloudflare data centers. These models are large, long-context, and mixture-of-experts types, presenting challenges in efficient serving due to memory constraints.

Key Takeaways

Cloudflare uses quantization techniques to optimize the KV cache for long-context models, increasing memory capacity from 686,000 tokens to 1.37 million tokens without affecting model accuracy.
The KV cache is stored in 8-bit floating point (FP8) instead of 16-bit (BF16), allowing for higher concurrent request handling and reducing costs by approximately 30% per token.
Model weights for GLM 5.2 are compressed from 8-bit floating point to 4-bit integers, reducing the model size by about 40% and decreasing per-GPU memory usage significantly.
Cloudflare benchmarks indicate that performance between FP8 and BF16 KV caches is indistinguishable in terms of model accuracy across various evaluation suites.

Read original article

Community Sentiment

Mixed

Positives

Nice to see a provider being transparent about KV cache quantisation — transparency is key in a field where subtle changes can make or break performance.
The claim that FP8 KV quantisation can deliver meaningful latency and capacity gains with negligible accuracy loss is a promising development in model efficiency.

Concerns

The benchmarks used to assert that FP8 KV quantisation is indistinguishable are lacking, especially since they omit critical coding benchmarks that could reveal serious issues.
Serving quantized models without clearly stating it feels deceptive, especially when the impact on coding agents could be significantly negative.