Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems

Battle Count: 77

Highly relevant to quantitative trading. The paper directly addresses portfolio optimization under time-inconsistent preferences (mean-variance and non-exponential discounting), which are common in real-world trading. The model-free RL approach eliminates the need for full model knowledge, making it practical for real trading environments. The deterministic policy gradient avoids sampling difficulties of stochastic policies. The algorithm learns equilibrium policies that are dynamically consistent, preventing the common problem of strategy deviation in practice.

High complexity due to: (1) Multiple coupled neural networks (policy, value, advantage rate, auxiliary functions f, g, and their advantage rates - at least 8 networks); (2) Actor-critic structure with inner fixed-point iterations; (3) Martingale orthogonality conditions requiring careful discretization; (4) Structured parameterizations that must match analytical solution forms; (5) Target networks with exponential moving averages; (6) Sampling from multiple distributions (uniform for time, Gaussian for state) for the extended parameters; (7) Re-parameterization tricks to enforce constraints; (8) Multiple learning rates for different network groups.

The paper provides detailed algorithm pseudocode (Algorithms 1-3), explicit parameterizations for both financial examples, and comprehensive hyperparameter settings. However, no code repository is provided. The structured neural network architectures are described analytically, and the theoretical convergence proof (Theorem 4.2) is fully self-contained. Reproduction would require implementing multiple coupled neural networks with martingale-based loss functions, which is non-trivial.