AirLLM enables 70B large language models to run on a single 4GB GPU without quantization, distillation, or pruning. Sparse MoE models allow for running larger models, such as the 405B Llama 3.1 on 8GB and the 2.8T Kimi K3 on under 4GB, by streaming one expert at a time.

WASTE is an embeddable C inference engine that allows the Kimi K3 model, which has 2.78 trillion parameters, to run on consumer laptops by streaming activated weights directly from NVMe storage. It operates without third-party runtime dependencies and manages memory by keeping the model trunk in memory while streaming selected experts from disk.

Kimi K3, the first 2.8T parameter open-source model, is now on Telnyx Inference. 1M context window, native vision, configurable reasoning. Input $2.70/1M tokens.

Hetzner is conducting experiments with large language model (LLM) inference, currently offering only one model without billing, service level agreements, or production guarantees. The company aims to assess user interest, system scalability, essential features, and load handling capabilities.

Flash-Moe is a pure C/Metal inference engine that runs the Qwen3.5-397B-A17B model, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB RAM at over 4.4 tokens per second. The 209GB model streams from SSD using a custom Metal compute pipeline without relying on Python or other frameworks.

Taalas has released an ASIC chip that runs Llama 3.1 8B with an inference rate of 17,000 tokens per second, equivalent to writing approximately 30 A4-sized pages in one second. The chip is claimed to be 10 times cheaper in ownership costs and 10 times more energy-efficient than GPU-based inference systems, while also being 10 times faster than current state-of-the-art inference solutions.

Large language models (LLMs) rely on inference engines to process prompts and manage requests efficiently in production environments. Understanding the architecture and scheduling of these engines, such as Nano-vLLM, is essential for optimizing LLM deployment.

Large Language Model (LLM) inference faces significant challenges primarily related to memory and interconnect issues rather than compute power. The autoregressive Decode phase of Transformer models distinguishes LLM inference from training, complicating the process.