LemonadeBench: Evaluating the Economic Intuition of Large Language Models in Simple Markets

Battle Count: 50

The paper focuses on small business economic decision-making (pricing, inventory, scheduling) rather than financial markets or trading. However, the underlying themes of decision-making under uncertainty, long-term planning, profit optimization, and evaluating AI economic reasoning have indirect relevance to quantitative trading. The benchmark methodology of decomposing performance across multiple dimensions could inspire similar frameworks for trading strategy evaluation. The multi-agent competition concept in future work (LemonadeBench 2.0) has parallels to market microstructure.

The simulation itself is relatively simple (single product, 30 days, 4 ingredients), but the evaluation framework with six counterfactual efficiency metrics, tool-based interaction, stateless API design, and demand function modeling requires moderate engineering effort. The theoretical optimal calculation and efficiency decomposition add analytical complexity. Running the benchmark requires API access and computational resources (reasoning models cost 75x more).

Code and evaluation framework available on GitHub. However, only 1 run per model is conducted (no statistical significance), random seeds are used, and results depend on specific API versions and temperature settings. The stateless design and tool schemas are fully specified. Future versions plan 30 runs per model.