Story
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
1 report
1 website
1 discussion
Latest report
Overview
Kog AI has launched a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 GPUs using FP16 without speculative decoding. The preview currently supports a 2B model, with plans to add support for large third-party MoE models at similar speeds.
Summary from blog.kog.ai.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · blog.kog.ai
Hacker News discussion ↗
Historical count: 5 · 5 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 5 sampled comments from this thread. Analyzed 2026-05-29.
- Praise: Achieving 3k tokens per second on standard GPUs opens up exciting possibilities for real-time applications, making advanced AI more accessible.
- Praise: The demo showcases the potential for rapid inference, hinting at future advancements in AI capabilities and user experiences.
- Praise: The focus on standard GPUs rather than custom chips suggests a push towards democratizing access to powerful AI tools.
- Concern: The comparison of a 2B model against much larger frontier models raises concerns about the fairness and relevance of the benchmarks presented.
- Concern: Some users feel that the term 'standard GPUs' is misleading, as it primarily refers to high-end data center GPUs rather than consumer-grade hardware.
- Concern: The performance of the small model in the demo has been criticized for lacking depth, indicating limitations in its practical applications.