GitHub - kyuz0/amd-strix-halo-toolboxes
Contribute to kyuz0/amd-strix-halo-toolboxes development by creating an account on GitHub.
Technical Setup Guide
Summary
GitHub guide for building a two-node AMD Strix Halo cluster with Intel E810 RoCE v2 to run distributed vLLM inference via Tensor Parallelism. Covers Fedora setup, RDMA networking, container toolbox installation, and cluster launch.
Highlights
- Configures a two-node AMD Strix Halo cluster for distributed vLLM inference with Tensor Parallelism over RoCE v2
- Uses Intel E810 NICs, Fedora 43 host setup, BIOS/kernel tweaks, and RDMA networking with MTU 9000
- Provides an automated toolbox via refresh_toolbox.sh that installs an RDMA-capable container and custom librccl.so patch
- Requires passwordless SSH, static IPs (192.168.100.1/2), trusted firewall interfaces, and HF_TOKEN for gated models
- Launches the cluster interactively using start-vllm-cluster to start Ray and serve models
auto-generated
Context
Audience
ML engineers, AI researchers, and system administrators building AMD Strix Halo based distributed inference clusters
DomainMachine Learning Infrastructure
FormatDocumentation
AccessOpen Source
Source Overview
Setup guide for a two-node AMD Strix Halo cluster (Framework Desktop, Ryzen AI MAX+, 128GB unified memory) using Intel E810 RoCE v2 for distributed vLLM inference with Tensor Parallelism.
Covers Fedora 43 host configuration: RDMA drivers, static IP networking at 9000 MTU, BIOS/kernel parameters, and a containerized toolbox with custom RCCL patches. Details Ray-based cluster orchestration, vLLM serve launch, and RDMA verification targeting ~5µs inter-node latency.
Topics
Related
AMD Strix HalovLLMRayRCCLRoCE v2Intel E810
