SignalTrace

Reinforcement Learning Trading Policy

John Montgomery6 min read

Reinforcement Learning Trading Policy

A reinforcement learning (RL) trading policy is a decision rule, learned by trial and error, that maps the current market state to a trading action — a trade signal, a target position, a position size, or an order-execution choice. It is the algorithmic answer to the Markov Decision Process Trading Model when the agent does not know the transition and reward functions in advance: rather than solving the MDP analytically by dynamic programming, an RL agent estimates the optimal policy from data by repeatedly acting, observing rewards, and updating. The reward is typically realised profit or a risk-adjusted measure (Sharpe ratio, the Differential Sharpe Ratio, drawdown-penalised return), and transaction costs can be folded directly into it. The foundational work is Moody and Saffell 2001, which introduced recurrent reinforcement learning (Recurrent Reinforcement Learning Trading) and proposed optimising financial performance functions directly instead of forecasting prices first.

Reinforcement Learning Trading Policy [relates] Markov Decision Process Trading Model Moody and Saffell 2001 [defines] Recurrent Reinforcement Learning Trading Reinforcement Learning Trading Policy [part-of] Markov Decision Process Trading Model

The main algorithm families are value-based, policy-based and actor-critic. Value-based methods (Q-learning and its deep extension, the Deep Q-Network or DQN) learn a state-action value function and pick the highest-valued action; they suit discrete action sets such as {short, flat, long}. Policy-based methods — including Moody & Saffell’s recurrent RL and modern policy gradients (PPO) — optimise the policy directly and naturally handle continuous position sizing. Actor-critic methods (A2C, DDPG, TD3, SAC) combine the two, updating a policy “actor” using a value “critic”. Because financial state is noisy and incomplete, many implementations formalise trading as a Partially Observable MDP and use recurrent encoders (LSTM, GRU). All of these target the three concrete trading jobs — signal generation, position sizing and execution — by maximising cumulative reward rather than predicting price, which is the structural appeal of the approach over supervised learning, as argued by Sun Wang An 2021.

Reinforcement Learning Trading Policy [supports] Regime Classification Recurrent Reinforcement Learning Trading [part-of] Reinforcement Learning Trading Policy Sun Wang An 2021 [supports] Reinforcement Learning Trading Policy

On profitability, the academic record is genuinely mixed and must be graded, not taken at face value. The most credible positive result is Zhang Zohren Roberts 2019 from the Oxford-Man Institute: DQN, policy-gradient and A2C agents trading 50 liquid Futures Markets contracts out-of-sample from 2011-2019 beat classical time-series-momentum baselines and stayed profitable at a realistic 25bp cost rate (DQN reached a portfolio Sharpe of 1.288). But even there the edge is regime-dependent — a plain long-only strategy beat the RL agents on equity indices during a strong bull run — and there is no live-trading evidence. The skeptical counterweight is Gort et al. 2022 on the Cryptocurrency Market: their best, least-overfitted PPO agent still lost 34.96% over the May-June 2022 test window. It “won” only by losing less than the S&P crypto benchmark (-50.78%) and the more-overfitted SAC agent (-59.48%). That is loss-minimisation during a crash, not demonstrated profit. The honest reading is that RL trading has many backtest-profit claims but very little substantiated, post-cost, live-tradeable edge.

Zhang Zohren Roberts 2019 [supports] Reinforcement Learning Trading Policy Zhang Zohren Roberts 2019 [trades_market] Futures Markets Gort et al. 2022 [contradicts] Reinforcement Learning Trading Policy Reinforcement Learning Trading Policy [relates] Transaction Costs and Slippage

The failure modes are well documented and structural. RL agents are flagrant overfitters: with thousands of hyperparameter combinations a researcher can get “lucky” and report an over-optimistic backtest that is a false positive — the core warning of Gort et al. 2022, which treats Overfitting in Quantitative Trading and Data-Snooping Bias as a hypothesis-testing problem and rejects agents whose probability of backtest overfitting exceeds a threshold (their SAC agent failed at 21.3%). Markets are non-stationary, so the fixed-MDP assumption underneath the whole method is violated and policies degrade out of sample — Non-Stationarity is the field’s named generalisation problem. Results are extremely sensitive to Reward Design Sensitivity and hyperparameters. Backtests assume no market impact and instant fills, producing a Sim-to-Real Gap that punishes agents in live deployment, especially at institutional scale. And the literature itself is hard to trust: Millea 2021, a critical survey of 152 DRL-trading papers, finds the field so methodologically fragmented — incompatible action spaces, datasets, costs and rewards, with few papers releasing code — that profitability claims cannot be aggregated or replicated, a Replication Crisis in Quantitative Finance in miniature. The companion survey Sun Wang An 2021, aggregating 100+ papers, reaches a convergent verdict: it names data scarcity, non-stationarity / distribution shift and the Sim-to-Real Gap as the field’s unsolved problems and presents RL trading as a research map, not a body of demonstrated profit.

Gort et al. 2022 [supports] Overfitting in Quantitative Trading Overfitting in Quantitative Trading [opposes] Out-of-Sample Backtesting Millea 2021 [contradicts] Reinforcement Learning Trading Policy Millea 2021 [supports] Replication Crisis in Quantitative Finance Non-Stationarity [causes] Sim-to-Real Gap

On balance, RL trading policies are a powerful and theoretically attractive framework — they directly optimise a trading objective, fold in costs, and handle sequential decision making — but as of the current evidence base they are best treated as a research programme, not a substantiated profitable trading approach. The strongest claims are academic backtests; the most rigorous studies either find regime-dependent edges or, after honest overfitting control, find agents that merely lose less than the market. Live evidence is essentially absent. The model family is genuinely useful for studying Regime Classification and adaptive position sizing, but a backtest Sharpe ratio here should be read as a hypothesis, not a result.

Reinforcement Learning Trading Policy [relates] Out-of-Sample Backtesting Reward Design Sensitivity [causes] Overfitting in Quantitative Trading

Connections

Sources

Graph View

Backlinks