GitHub - QuinsZouls/llama-cpp-turboquant at llama-next
LLM inference in C/C++. Contribute to QuinsZouls/llama-cpp-turboquant development by creating an account on GitHub.
Welcome! Type "help" for available commands.
$
Machine Learning Inference Engine
Summary
Zinc is a high-performance, Zig-based inference engine designed for running local LLMs on AMD GPUs via Vulkan and Apple Silicon via Metal. It provides a lightweight, single-binary solution that avoids heavy dependencies like ROCm or MLX by using hand-tuned shaders optimized for consumer hardware.
Highlights
- Provides high-performance execution on AMD RDNA3/RDNA3 and Apple Silicon without requiring ROCm or MLX.
- Features hand-tuned shaders specifically optimized for Vulkan (AMD) and native Metal (Apple).
- Offers multiple interaction methods including CLI, an OpenAI-compatible API, and an integrated browser chat UI.
- Includes a managed model workflow for easy downloading, listing, and managing of supported LLMs.
auto-generated
Context
Audience
Developers and AI enthusiasts seeking efficient local LLM deployment on consumer-grade AMD and Apple hardware
DomainMachine Learning Infrastructure
FormatOpen source software repository
AccessOpen source
Topics
Related
Zig Programming LanguageVulkan APIMetal APILocal LLM InferenceLarge Language Models
