Open Source Library
Summary
llama.cpp is a high-performance C/C++ library for local LLM inference, optimized for diverse hardware including Apple Silicon and GPUs. It supports extensive quantization, hybrid CPU+GPU execution, and seamless Hugging Face integration.
Highlights
- Zero-dependency plain C/C++ implementation ensuring broad compatibility.
- Support for quantization from 1.5-bit to 8-bit to reduce memory usage.
- Native optimization for Apple Silicon, x86, and various GPU backends.
- Direct integration with Hugging Face for model caching and GGUF support.
- Features multimodal capabilities and developer plugins for VS Code and Vim.
auto-generated
Context
Audience
Machine Learning Engineers and Developers
DomainMachine Learning Infrastructure
Formatsoftware repository
Accessopen source
Topics
Related
GGUFHugging FaceCUDAMetalLLM Quantizationllama-server