UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos

Battle Count: 98

While UniFinEval is primarily a benchmark evaluation paper rather than a trading strategy paper, it is highly relevant to quantitative trading in several ways: (1) It evaluates MLLMs' capabilities in financial risk sensing, asset allocation analysis, and industry trend insights - all critical for quantitative trading decisions. (2) The findings reveal significant gaps between MLLMs and human experts in cross-modal reasoning and decision-making, informing practitioners about current model limitations. (3) The benchmark covers scenarios directly applicable to trading: fundamental analysis, risk identification, and portfolio allocation. (4) However, it does not propose new trading models, strategies, or direct market prediction methods. The relevance is more at the infrastructure/evaluation level for AI-assisted trading systems.

As a benchmark paper, the primary implementation involves: (1) Setting up evaluation pipelines for 10 MLLMs (4 closed-source via API, 6 open-source locally on 8x NVIDIA A800 GPUs using vLLM/LMDeploy). (2) Implementing Zero-Shot and CoT evaluation protocols. (3) Using Qwen-Max for output extraction and evaluation. (4) Conducting systematic error analysis across five dimensions. The benchmark construction itself (manual by 10 financial experts) is labor-intensive but the evaluation framework is relatively straightforward. The complexity lies in the multimodal data handling (text, images, videos) and cross-modal reasoning evaluation.

Data and code are publicly available on GitHub (https://github.com/aifinlab/UniFinEval). The benchmark is manually constructed with detailed quality control procedures documented. However, evaluation of closed-source models (Gemini, GPT, Grok, Claude) depends on API access which may change over time. The dataset is bilingual (Chinese and English) with 3,767 Q&A pairs. Evaluation uses Qwen-Max for output extraction with <1% error rate verified by manual inspection.