DataChain is a Python library that indexes files in S3, GCS, Azure, and local filesystems into versioned, typed datasets using Pydantic schemas and parallel computation with incremental updates.
It stores schemas, lineage, and file pointers in a local SQLite database, supporting sub-second filter, join, and group_by operations over millions of records plus vector search without a separate store. An optional skill integrates the datasets into Claude Code, Cursor, Codex, GitHub Copilot, and Pi, letting agents query and chain operations on the same versioned data.