Building a web search engine from scratch in two months with 3 billion neural embeddings
End-to-end deep dive of the project, spanning a large GPU cluster, distributed RocksDB, and terabytes of sharded HNSW.
Wilson Lin’s Vector Embedding Search Engine
search.wilsonl.inSaved September 28, 2025
Summary
Wilson Lin independently built a web search engine in two months using 3 billion SBERT neural embeddings, running on a distributed cluster of 200 GPUs and indexing 280 million pages.
The engine is designed to understand full query intent, not just keywords, and can accurately answer complex, nuanced queries while effectively minimizing SEO spam. Features include low-latency (500ms) searches, embedded AI for quick answers and summarization, and a minimalist interface with tracking for search quality improvements.
