Welcome! Type "help" for available commands.
$
Machine Learning Software
Summary
This GitHub repository is a specialized fork of llama.cpp featuring adaptive speculative decoding and a Vulkan backend optimized for AMD Strix Halo hardware. It provides significant performance enhancements for LLM inference, specifically targeting structured output generation and MoE prefill speeds.
Highlights
- Implements adaptive speculative decoding that adjusts draft length based on measured acceptance rates rather than a fixed value.
- Features a heavily optimized Vulkan backend tuned for AMD Strix Halo (Radeon 8060S) using stock Mesa RADV drivers.
- Delivers massive performance gains, including a 4.7x increase in structured output generation and a 1.9x increase in MoE prefill.
- Includes support for ROCmFPx/FP4 quantization and optimized handling for specific batch widths (3-8).
auto-generated
Context
Audience
Machine Learning Engineers and Developers optimizing LLM performance on AMD hardware
DomainMachine Learning Inference
FormatOpen source repository
AccessOpen source
Topics
Related
llama.cppVulkan APIAMD Strix HaloSpeculative DecodingLLM Quantization