Technical Setup Guide
Summary
GitHub guide for building a two-node AMD Strix Halo cluster with Intel E810 RoCE v2 to run distributed vLLM inference via Tensor Parallelism. Covers Fedora setup, RDMA networking, container toolbox installation, and cluster launch.
Highlights
- Configures a two-node AMD Strix Halo cluster for distributed vLLM inference with Tensor Parallelism over RoCE v2
- Uses Intel E810 NICs, Fedora 43 host setup, BIOS/kernel tweaks, and RDMA networking with MTU 9000
- Provides an automated toolbox via refresh_toolbox.sh that installs an RDMA-capable container and custom librccl.so patch
- Requires passwordless SSH, static IPs (192.168.100.1/2), trusted firewall interfaces, and HF_TOKEN for gated models
- Launches the cluster interactively using start-vllm-cluster to start Ray and serve models
auto-generated
Context
Audience
ML engineers, AI researchers, and system administrators building AMD Strix Halo based distributed inference clusters
DomainMachine Learning Infrastructure
FormatDocumentation
AccessOpen Source
Related
AMD Strix HalovLLMRayRCCLRoCE v2Intel E810