Mesh LLM enables distributed AI computing on the iroh platform, allowing teams to run large language models without relying on centralized data centers. This setup provides greater control over model updates, data privacy, and operational costs.

This guide provides instructions for configuring a two-node AMD Strix Halo cluster using Intel E810 (RoCE v2) for distributed vLLM inference with Tensor Parallelism. It covers hardware prerequisites, host configuration for Fedora 43, toolbox installation, network verification, cluster operation, and troubleshooting steps.

Zero-latency API auth and billing for distributed GPU inference.

A small-scale distributed inference cluster can be built using AMD’s Ryzen™ AI Max+ AI PC platform to run a one trillion-parameter Large Language Model. A four-node cluster of Framework Desktop systems demonstrates the local inference of the Kimi K2.5 open-source model.