~/bookmarks

Welcome! Type "help" for available commands.

$

Welcome! Type "help" for available commands.

$

Welcome! Type "help" for available commands.

$

~/bookmarks

William's Bookmark Library

/**/

so, you want to chunk really fast?

minha.shSaved January 6, 20267 min

Summary

Memchunk is a Rust library for fast text chunking in RAG pipelines, splitting at semantic delimiters like periods, newlines, and question marks to avoid sentence fragments.

It uses the memchr crate with SIMD (AVX2/SSE2) for 1-3 delimiters and lookup tables for more, employing backward search for efficiency. Benchmarks show up to 1TB/s throughput, chunking English Wikipedia in 120ms.

Topics

Algorithm Optimization SIMD Performance Low-Level Programming Text Chunking Optimization RAG Pipelines

Visit Site All Bookmarks

William's Bookmark Library

/**/

so, you want to chunk really fast?

minha.shSaved January 6, 20267 min

Summary

Memchunk is a Rust library for fast text chunking in RAG pipelines, splitting at semantic delimiters like periods, newlines, and question marks to avoid sentence fragments.

It uses the memchr crate with SIMD (AVX2/SSE2) for 1-3 delimiters and lookup tables for more, employing backward search for efficiency. Benchmarks show up to 1TB/s throughput, chunking English Wikipedia in 120ms.

Topics

Algorithm Optimization SIMD Performance Low-Level Programming Text Chunking Optimization RAG Pipelines

Visit Site All Bookmarks

~/bookmarks

~/bookmarks

so, you want to chunk really fast?

Summary

Topics

Discover Similar Content

How We Made 100M Vector Indexing in 20 Minutes Possible on PostgreSQL

tobi/qmd: mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local

GitHub - open-puffer: opensource and fastest vectorDB

Discover Similar Content

so, you want to chunk really fast?

Summary

Topics

Discover Similar Content

How We Made 100M Vector Indexing in 20 Minutes Possible on PostgreSQL

tobi/qmd: mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local

GitHub - open-puffer: opensource and fastest vectorDB