GitHub - open-puffer: opensource and fastest vectorDB
opensource and fastest vectorDB. Contribute to harishsg993010/open-puffer development by creating an account on GitHub.
Dataset / Vector Search Example
Summary
The Hacker News vector search dataset provides 28.74 million Hacker News items with 384‑dimensional embeddings generated by the all-MiniLM-L6-v2 model, delivered as a Parquet file that can be imported into ClickHouse for large‑scale semantic search.
Highlights
- 28.74 M rows with text, metadata and 384‑dimensional embeddings
- Distributed as a single Parquet file on S3, loadable via ClickHouse's `s3` function
- Schema includes fields for filtering (type, author, time, score, etc.) and a vector column
- Instructions to create a MergeTree table, import data, and build an HNSW cosine‑distance index
- Guidance on hyper‑parameter tuning (M, ef_construction) and performance considerations
auto-generated
Context
Audience
Data engineers, machine learning practitioners, and developers building or benchmarking large‑scale semantic search systems
DomainScalable Vector Search / Information Retrieval
FormatParquet dataset + SQL DDL/DML examples (code snippet)
Accessfree online via ClickHouse documentation and S3 bucket
Personal Notes
A discussion on this: https://news.ycombinator.com/item?id=46081053 At the embedding size, this is an 11 billion parameter data set.
Topics
Related
ClickHouse vector index documentationSentence‑Transformers all-MiniLM-L6-v2 modelHNSW algorithm for ANN searchS3 Parquet storage formatHacker News discussion thread (HN)