gpu-inference
Latest stories covering gpu-inference, with summaries and links to related coverage.
2 articles · page 1 of 1
Connected topics
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
tool
18 min read
Kog AI has launched a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 GPUs using FP16 without speculative decoding. The preview currently supports a 2B model, with plans to add support for large third-party MoE models at similar speeds.
Launch HN: IonRouter (YC W26) – High-throughput, low-cost inference
tool
1 min read
Zero-latency API auth and billing for distributed GPU inference.