GitHub - QuinsZouls/llama-cpp-turboquant at llama-next
LLM inference in C/C++. Contribute to QuinsZouls/llama-cpp-turboquant development by creating an account on GitHub.
LLM Inference Optimization
Summary
A llama.cpp fork featuring TurboQuant compression and speculative decoding for Gemma 4 and Qwen 3.6 to boost throughput by up to 50%. It optimizes memory usage via low-bit KV caches and supports multimodal inference on various hardware backends.
Highlights
- TurboQuant WHT-rotated quantization provides ~4.3x KV cache compression.
- Gemma 4 MTP speculative decoding increases short-prompt throughput by 30-50%.
- Qwen 3.6 NextN decoding improves speed by 24-36% on MoE models.
- Multimodal support allows image processing alongside text speculative decoding.
- Compatible with Metal, CUDA, Vulkan, and HIP backends.
auto-generated
Context
Audience
AI Engineers and Developers
DomainMachine Learning
Formatopen source software repository
Accessopen source
Source Overview
Fork of llama.cpp for C/C++ LLM inference with TurboQuant low-bit KV cache and weight compression.
Supports Gemma 4 MTP and Qwen 3.6 NextN speculative decoding, with claimed throughput gains of about 30–50%. Includes install, Docker, binary release, and source build instructions.
Topics
Related
llama.cppTurboQuantSpeculative DecodingGemma 4Qwen 3.6GGUF Format
