Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
Kog AI has launched a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 GPUs using FP16 without speculative decoding. The preview currently supports a 2B model, with plans to add support for large third-party MoE models at similar speeds.
blog.kog.ai
18 min
5d ago
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
Kog AI has launched a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 GPUs using FP16 without speculative decoding. The preview currently supports a 2B model, with plans to add support for large third-party MoE models at similar speeds.
blog.kog.ai
18 min
5d ago
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
Kog AI has launched a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens per second on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 GPUs using FP16 without speculative decoding. The preview currently supports a 2B model, with plans to add support for large third-party MoE models at similar speeds.
blog.kog.ai
18 min
5d ago
No more articles to load