Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Battle Count: 179

Directly addresses core challenges in quantitative trading: non-stationary markets, noisy rewards, and the need for stable policy learning. The automatic discovery of auxiliary tasks eliminates manual feature engineering for trading signals. Evaluated on four major global indices with comprehensive risk-adjusted metrics. The meta-gradient approach for credit assignment to auxiliary questions is novel and directly applicable to improving RL-based trading agents. Consistent outperformance across all four markets on Sharpe, Calmar, and Sortino ratios.

Requires implementing a dual-network architecture (main PPO network + question network), GVF formulation with cumulants and state-dependent discounts, TD targets with truncation horizon W, and a non-myopic meta-gradient that differentiates through K inner PPO updates via the chain rule. Hyperparameter tuning involves 12 parameters via Bayesian optimization. The meta-gradient computation through unrolled trajectories adds significant engineering complexity compared to standard PPO.

The paper provides detailed hyperparameter search ranges (Table 1), algorithm pseudocode (Algorithm 1), and mathematical formulations. However, no code repository is mentioned. Data is sourced from Yahoo Finance (publicly available). The meta-gradient implementation details (inner unroll, chain rule through K steps) add complexity. Five independent runs are reported with standard deviations.