datachain-ai/datachain: The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure - datachain-ai/datachain
Python Library
Summary
LOTUS is an open-source Python query engine using LLMs and a Pandas-like API for fast document processing. It optimizes queries for up to 400x speedups and supports declarative AI pipelines for tasks like fact-checking and search ranking.
Highlights
- Pandas-like API with LLM-powered semantic operators
- Up to 400x performance optimization
- Declarative multi-step AI pipeline support
- State-of-the-art accuracy on classification tasks
- Seamless integration with vector databases
auto-generated
Context
Audience
Data Scientists and Software Developers
DomainMachine Learning
Formatsoftware documentation
Accessopen source
Topics
Related
PandasLarge Language ModelsSemantic SearchDocument ProcessingVector DatabasesFEVER Dataset
