Vectorize: a vector database for shipping AI-powered applications to production, fast
Vectorize is our brand-new vector database offering, designed to let you build full-stack, AI-powered applications entirely on Cloudflare’s global net...
Personal Notes
A discussion on this: https://news.ycombinator.com/item?id=46081053 At the embedding size, this is an 11 billion parameter data set.
Dataset / Vector Search Example
Summary
The Hacker News vector search dataset provides 28.74 million Hacker News items with 384‑dimensional embeddings generated by the all-MiniLM-L6-v2 model, delivered as a Parquet file that can be imported into ClickHouse for large‑scale semantic search.
Highlights
- 28.74 M rows with text, metadata and 384‑dimensional embeddings
- Distributed as a single Parquet file on S3, loadable via ClickHouse's `s3` function
- Schema includes fields for filtering (type, author, time, score, etc.) and a vector column
- Instructions to create a MergeTree table, import data, and build an HNSW cosine‑distance index
- Guidance on hyper‑parameter tuning (M, ef_construction) and performance considerations
auto-generated
Context
Audience
Data engineers, machine learning practitioners, and developers building or benchmarking large‑scale semantic search systems
DomainScalable Vector Search / Information Retrieval
FormatParquet dataset + SQL DDL/DML examples (code snippet)
Accessfree online via ClickHouse documentation and S3 bucket
Source Overview
The Hacker News vector search dataset contains 28.74 million postings with 384-dimensional vector embeddings generated using the all-MiniLM-L6-v2 model.
The dataset is provided as a Parquet file and can be loaded into ClickHouse for scalable semantic search and similarity queries. Users can build vector similarity indexes and perform efficient semantic searches using cosine distance, with applications ranging from document retrieval to generative AI summarization.
Topics
Related
ClickHouse vector index documentationSentence‑Transformers all-MiniLM-L6-v2 modelHNSW algorithm for ANN searchS3 Parquet storage formatHacker News discussion thread (HN)
