FinFIRST_paper.pdf · inclusionAI/FinFIRST at main
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
LLM Benchmark Dataset
Summary
FinFIRST is a 123-row Chinese financial QA benchmark on Hugging Face for evaluating LLMs on multi-step data retrieval and arithmetic tasks across stocks, futures, and macro indicators, with detailed rubric-based scoring.
Highlights
- Multi-step tasks requiring both external data retrieval and precise arithmetic such as computing stock purchase costs at minimum lot sizes and GDP year-over-year growth rates
- Granular rubric-based scoring with point allocations per sub-step covering data lookup, calculation, and formatting for fine-grained evaluation of reasoning chains
- Rich metadata taxonomy with over 20 structured fields including relation_type, subject_category, region, time_constraint_type, calculation_type, and source_specification
- Questions span Chinese A-shares, Japanese macroeconomic data, and futures contracts, testing cross-market and cross-asset financial reasoning
- Includes a question_decomposition field that breaks each query into sequential sub-questions, supporting step-by-step evaluation
auto-generated
Context
Audience
NLP researchers evaluating LLM financial reasoning capabilities, developers building financial AI assistants, and benchmark designers interested in rubric-based evaluation methodologies
DomainFinancial NLP and LLM Evaluation
FormatStructured dataset on Hugging Face
AccessFree online
Topics
Related
FinBenCFLEBRubric-based LLM evaluationChinese financial NLPFinancial QA benchmarksMulti-step reasoning evaluation
