llama.cpp is a plain C/C++ library for LLM inference with state-of-the-art performance across hardware like Apple silicon, x86, RISC-V, NVIDIA GPUs, and Vulkan.

It supports models including LLaMA series, Mistral, Mixtral, DBRX, and PLaMo-13B via GGUF format with quantization from 1.5-bit to 8-bit. Installation options include brew, Docker, pre-built binaries, or source builds, with commands like llama-cli for local models or llama-server for OpenAI-compatible APIs.

Topics

LLM Inference C/C++ Implementation Quantization Techniques GGUF Models Llama.cpp

View on GitHub All Bookmarks

Discover Similar Content

William's Bookmark Library

/**/

port-sweep-bench

github.comSaved March 31, 202611 min

ubergarm · via GitHub

Summary

llama.cpp is a plain C/C++ library for LLM inference with state-of-the-art performance across hardware like Apple silicon, x86, RISC-V, NVIDIA GPUs, and Vulkan.

Topics

LLM Inference C/C++ Implementation Quantization Techniques GGUF Models Llama.cpp

View on GitHub All Bookmarks