BM42: New Baseline for Hybrid Search - Qdrant
Introducing BM42 - a new sparse embedding approach, which combines the benefits of exact keyword search with the intelligence of transformers.
AI Research Blog Post
Summary
The article presents Contextual Retrieval, an improved RAG technique that preserves contextual signals during embedding and BM25 search, cutting failed retrievals by up to 67% with reranking. It also discusses prompt‑caching for Claude and when a full‑prompt approach is feasible.
Highlights
- Traditional RAG loses context, causing irrelevant or missed results.
- Contextual Retrieval uses Contextual Embeddings and Contextual BM25 to boost accuracy.
- Failed retrievals drop 49% overall and 67% when combined with reranking.
- Claude deployment instructions are provided via a cookbook.
- Prompt caching now makes full‑prompt methods faster and up to 90% cheaper.
auto-generated
Context
Audience
Developers and researchers building AI systems that need reliable knowledge retrieval for customer support, legal analysis, or other domain‑specific assistants
DomainMachine Learning
Formatlong-form article
Accessfree online
Topics
Related
Retrieval‑Augmented GenerationVector database searchClaude prompt cachingBM25 algorithmEmbedding models