GitHub - QuinsZouls/llama-cpp-turboquant at llama-next
LLM inference in C/C++. Contribute to QuinsZouls/llama-cpp-turboquant development by creating an account on GitHub.
Welcome! Type "help" for available commands.
$
Summary
llama.cpp is a plain C/C++ library for LLM inference with state-of-the-art performance across hardware like Apple silicon, x86, RISC-V, NVIDIA GPUs, and Vulkan.
It supports models including LLaMA series, Mistral, Mixtral, DBRX, and PLaMo-13B via GGUF format with quantization from 1.5-bit to 8-bit. Installation options include brew, Docker, pre-built binaries, or source builds, with commands like llama-cli for local models or llama-server for OpenAI-compatible APIs.
