AI inference

Source reports covering AI inference. Stories will appear when analysis and community reactions are ready.

8 reports · page 1 of 1

Connected topics are temporarily unavailable. You can still browse stories below.

Source reports

• AirLLM 70B inference with single 4GB GPU

  2026-08-03

  tool

  7 min read

  AirLLM enables 70B large language models to run on a single 4GB GPU without quantization, distillation, or pruning. Sparse MoE models allow for running larger models, such as the 405B Llama 3.1 on 8GB and the 2.8T Kimi K3 on under 4GB, by streaming one expert at a time.

  LLMsDeveloper toolsOpen source models

• Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

  2026-07-31

  tool

  12 min read

  WASTE is an embeddable C inference engine that allows the Kimi K3 model, which has 2.78 trillion parameters, to run on consumer laptops by streaming activated weights directly from NVMe storage. It operates without third-party runtime dependencies and manages memory by keeping the model trunk in memory while streaming selected experts from disk.

  LLMsDeveloper toolsMachine learning models

• Kimi K3 Now Available via Telnyx Inference API

  2026-07-27

  tool

  1 min read

  Kimi K3, the first 2.8T parameter open-source model, is now on Telnyx Inference. 1M context window, native vision, configurable reasoning. Input $2.70/1M tokens.

  Kimi k3Open source modelsComputer vision

• Hetzner is working on LLM Inference

  2026-07-24

  tool

  6 min read

  Hetzner is conducting experiments with large language model (LLM) inference, currently offering only one model without billing, service level agreements, or production guarantees. The company aims to assess user interest, system scalability, essential features, and load handling capabilities.

  HetznerLLMsDeveloper tools

• Flash-MoE: Running a 397B Parameter Model on a Laptop

  2026-03-22

  tool

  6 min read

  Flash-Moe is a pure C/Metal inference engine that runs the Qwen3.5-397B-A17B model, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB RAM at over 4.4 tokens per second. The 209GB model streams from SSD using a custom Metal compute pipeline without relying on Python or other frameworks.

  LLMsDeveloper toolsMacbook pro

• How Taalas “prints” LLM onto a chip?

  2026-02-21

  tool

  4 min read

  Taalas has released an ASIC chip that runs Llama 3.1 8B with an inference rate of 17,000 tokens per second, equivalent to writing approximately 30 A4-sized pages in one second. The chip is claimed to be 10 times cheaper in ownership costs and 10 times more energy-efficient than GPU-based inference systems, while also being 10 times faster than current state-of-the-art inference solutions.

  LLMsTaalasHardware acceleration

• Nano-vLLM: How a vLLM-style inference engine works

  2026-02-02

  9 min read

  Large language models (LLMs) rely on inference engines to process prompts and manage requests efficiently in production environments. Understanding the architecture and scheduling of these engines, such as Nano-vLLM, is essential for optimizing LLM deployment.

  LLMsOpenAIDeveloper tools

• David Patterson: Challenges and Research Directions for LLM Inference Hardware

  2026-01-25

  2 min read

  Large Language Model (LLM) inference faces significant challenges primarily related to memory and interconnect issues rather than compute power. The autoregressive Decode phase of Transformer models distinguishes LLM inference from training, complicating the process.

  LLMsHardware architectureTransformersMemory optimization