LLM Inference Optimization
Summary
A llama.cpp fork featuring TurboQuant compression and speculative decoding for Gemma 4 and Qwen 3.6 to boost throughput by up to 50%. It optimizes memory usage via low-bit KV caches and supports multimodal inference on various hardware backends.
Highlights
- TurboQuant WHT-rotated quantization provides ~4.3x KV cache compression.
- Gemma 4 MTP speculative decoding increases short-prompt throughput by 30-50%.
- Qwen 3.6 NextN decoding improves speed by 24-36% on MoE models.
- Multimodal support allows image processing alongside text speculative decoding.
- Compatible with Metal, CUDA, Vulkan, and HIP backends.
auto-generated
Context
Audience
AI Engineers and Developers
DomainMachine Learning
Formatopen source software repository
Accessopen source
Topics
Related
llama.cppTurboQuantSpeculative DecodingGemma 4Qwen 3.6GGUF Format