inclusionAI/FinFIRST · Datasets at Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Research Paper
Summary
FinFIRST is a specialized financial benchmark designed to evaluate LLM search agents on both their final answers and the traceability of their supporting evidence. It utilizes atomic rubrics to assess information acquisition, source verification, and computation across 123 expert-authored tasks.
Highlights
- Evaluates both final answers and supporting evidence through fine-grained atomic rubrics
- Comprises 123 tasks built from an 18-field taxonomy and 138 registered financial sources
- Identifies computation and answer formation as a consistent performance bottleneck for LLM agents
- Provides a measurable and diagnosable framework for the entire financial research process
auto-generated
Context
Audience
AI Researchers and Machine Learning Engineers
DomainMachine Learning
FormatResearch Paper
AccessOpen source
Source Overview
FINFIRST is a 123-task financial benchmark that scores LLM search agents on both final answers and supporting evidence via atomic rubrics covering information acquisition, source verification, and computation.
Tasks are built from an 18-field taxonomy, 138 registered financial sources, and a 6-stage quality control pipeline. Across 15 model configurations, Claude-Opus-5 achieved the top atomic score (87.59%) while GPT-5.6-Sol led in strict pass rate (71.54%), with computation lagging as a common bottleneck.
Topics
Related
LLM AgentsFinancial Information RetrievalAI BenchmarkingEvidence Grounding
