Two different tricks for fast LLM inference
seangoedecke.com
February 15, 2026
9 min read
🔥🔥🔥🔥🔥
57/100
Summary
Anthropic's fast mode enables up to 2.5 times the token processing speed, achieving around 170 tokens per second. OpenAI's fast mode surpasses this with over 1000 tokens per second, making it six times faster than Anthropic's offering.
Key Takeaways
- Anthropic's fast mode offers up to 2.5 times more tokens per second compared to its previous model, while OpenAI's fast mode exceeds 1000 tokens per second, making it 15 times faster than its previous version.
- Anthropic's fast mode utilizes the actual Opus 4.6 model, whereas OpenAI's fast mode uses a less capable version, GPT-5.3-Codex-Spark.
- OpenAI's fast mode is powered by Cerebras chips, which are significantly larger than standard inference chips, while Anthropic's fast mode is based on low-batch-size inference.
- The tradeoff in AI inference economics involves batching, where smaller batches can lead to faster individual inferences at the cost of overall throughput.
Community Sentiment
Positives
- The potential for the OpenAI-Cerebras partnership to scale chips could revolutionize AI performance, enabling faster models like Codex-5.3 with fewer errors, enhancing user experience.
- In-memory inference using large SRAM chips could significantly reduce latency, making real-time applications like voice AI more natural and responsive, which is crucial for user engagement.
- The innovative approach of parallel distill and refine in Anthropic's fast mode suggests a smarter, more efficient method for handling complex problems while maintaining speed.
Concerns
- The limitation of current chips, such as Cerebras' 44GB SRAM, raises concerns about their ability to handle larger models like GPT-5.3, potentially hindering advancements in AI capabilities.
- The trade-off between speed and accuracy, where increasing speed by 6x could lead to 20% more mistakes, highlights a significant challenge in optimizing AI performance for practical use.
