AI-driven asset allocation

Artificial intelligence has become central to portfolio design. Reinforcement learning (RL) and deep learning (DL) capture non-linearities, high dimensionality, and regime shifts [1]. Experiments with DDPG, SAC, and PPO suggest that model-free RL adapts well under non-stationarity, while average-reward policy iteration provides horizon-sensitive trade-offs [14]. Yet no universal winner emerges; adaptability depends on context.

Applications are diverse. Deep Q-Networks have been extended to handle microstructure and trading costs [16]. In crypto markets, state spaces enriched with technical and sentiment signals achieved excess returns above 20% [5]. Multi-agent RL (MARL) expands this scope: masked reconstruction for dimensionality reduction [17], game-theoretic MARL for cooperation and competition [18], and modular MARL scaling to high-cardinality portfolios [19]. Still unresolved is whether these designs generalize across asset classes.

Hybrid approaches connect DL forecasts with classical optimizers. Some integrate neural predictors into mean–variance or Omega ratio [20]; others link ML regressors with mean–VaR to address tail risk [21]. Interpretability remains a limitation [8]. Orthogonal reward decomposition has been proposed as a way to balance competing objectives and improve robustness [13].

Language models have also entered finance. BERT transforms news and sentiment into tradable features [22]. MARL has been tested for cross-asset risk, and Pareto-based allocation for multi-objective trade-offs [18]. Alongside, attention maps, SHAP, and LIME provide interpretability tools [23,24]. Still, transparency remains fragmented and case-specific.

A deeper challenge concerns alignment with financial theory. Comparative reviews note methodological heterogeneity [8,25], while heuristic evolutionary optimizers remain competitive [26]. Hansen [27] warns of a “paradox of uncertainty”: AI mitigates risks yet introduces new forms of model risk.

This study differs from prior work that merely inserted factor signals into state definitions. Here, reward functions are tied to time-varying factor exposures, treating not as exogenous input but as the optimization target itself.

Traditional asset allocation theories and factor-based models

Portfolio theory originates with Markowitz’s Modern Portfolio Theory (MPT), showing that diversification reduces idiosyncratic risk by balancing correlations [3]. Sharpe extended this with the CAPM, placing market beta () at the core of systematic risk [28]. Elegant though it was, the single-factor frame proved too restrictive.

Subsequent work broadened the scope. Fama–French added firm size (SMB) and value (HML) to capture beta’s weak explanatory power [4]. Carhart introduced momentum [29], while Campbell distinguished “bad” volatility-driven from “good” long-term beta [9]. These additions refined interpretation, yet the proliferation of factors underscored theoretical instability.

Alternative risk definitions deepened the debate. Sortino advocated downside deviation against a minimum return [30]. Harvey later warned of a “factor zoo,” noting that many claimed factors lack economic grounding [7]. Not all statistical regularities qualify as true risk sources.

Machine learning (ML) further reshaped the field. ML highlights high-dimensional non-linear patterns [8], but Ang [31] stressed that factor models often exaggerate explanatory power. Predictive accuracy alone—without economic rationale—can mislead [10].

Evidence is mixed. Deep networks embedded into factor models show gains [10], while reinforcement learning (RL) reframes allocation as sequential choice. Cornalba (2024) linked multi-objective rewards with portfolio constraints [13], and hybrid RL–quantum methods explored adaptive weighting [32]. Robustness, however, remains contested.

Explainability is a parallel concern. Finance-specific taxonomies for XAI [24] and policy visualization in RL [33] show partial progress. Still unresolved is a core point: very few RL models treat sensitivity itself as a learning target. In shifting regimes, this gap is far from trivial.

Beta estimation and reward design based on technical factors

Ross’s Arbitrage Pricing Theory (APT) framed returns as linear factor combinations [34]. In contrast, this study emphasizes technical indicators as dynamic signals for DRL. Momentum, long central in Carhart’s four-factor model [29], retains relevance under regime shifts [35], making it a natural input for time-varying beta.

Beta () reflects market sensitivity, but fixed-window regressions often diverge—by nearly 40% in high-frequency cases [11]. Dynamic methods address such breaks [12]. Campbell’s notion of “good” and “bad” beta [9] extends here: the former tied to stable exposures, the latter to volatility shocks.

In DRL portfolios, can enter the state vector. Prior studies used Q-learning with moving averages or volume [16], and beta-neutral designs outperformed static OLS by nearly 30% [2]. Other works linked forecasts with mean–VaR [21], or broadened scope through sentiment and graph features [15,22,36]. Multi-agent settings often embed many indicators [18,37].

Reward construction follows similar logic. Some embed directly into returns [1,25]; others penalize drawdown or concentration [19]. Multi-objective rewards improved efficiency, in some reports tripling baseline results [13]. In volatile domains like crypto, Sharpe–drawdown blends guided allocation [5].

Challenges remain. Overfitting and interpretability persist [24]. Still, linking rolling beta estimates with structured rewards bridges factor theory and adaptive RL. Unlike neutral or purely predictive rules, the proposed design embeds in both state and reward, spanning multiple assets and algorithms.

Summary of related works and research gap

RL-based portfolio research has advanced along divergent paths rather than a single trajectory. Park and Sim (2020) combined technical and sentiment inputs, while Lin and Chen (2022) designed multi-agent systems for faster learning. By contrast, Behera (2023) and Ma (2021) imposed explicit risk limits, and Hambly (2023) with Cornalba (2024) highlighted multi-objective rewards, showing that stability under frictions can rival pure return targets. The field lacks consensus.

Factor investing—anchored in size, value, and momentum since Fama (1996)—intersects unevenly with these approaches. Deep models capture shifting exposures (Feng, 2024), but Bagnara (2024) warns that many ML studies privilege fit over economic meaning. Interpretation diverges, and what qualifies as a “factor” remains unsettled.

Explainability further complicates the picture. Graph tools and SHAP improve transparency at prediction level [36], yet Vouros et al. (2022) argue that sequential policy paths remain opaque [33]. Output clarity does not resolve decision ambiguity. Evidence is fragmented.

Two research gaps follow. First, RL rarely embeds classical factors—such as Fama–French variables—into state or reward functions. Second, dynamic factor models seldom connect with RL structures. Both limit theoretical grounding.

This study addresses these issues by integrating rolling-regression betas into factor-aware rewards, while state vectors blend technical and factor signals. The design links three strands: RL allocation (Hambly, 2023), factor modeling (Fama, 1996), and interpretability (Weber, 2024). It balances adaptability with interpretability, aiming to test sensitivity under regime shifts.

Still unresolved is how to encode economic structure without undermining operational performance (Bagnara, 2024). Future work may examine hierarchical factors, new asset classes, and policy-level XAI.

Model design and theoretical framework

Factor integration reshapes a conventional reinforcement learning framework. The cycle of observation (S_t), action (A_t), and reward (R_t) remains, but its meaning is reinterpreted. Here, factors are not exogenous statistics—they are embedded within both state representation and reward design. Three reward alternatives are explored. A Static- version fixes regression sensitivities over time. A Dynamic- form adjusts exposures through rolling regressions of momentum, volatility, moving-average deviations, and volume signals. Finally, Momentum- focuses only on momentum, acting as a minimal benchmark. Each reflects a different trade-off: the first stresses stability, the second adaptability, and the third parsimony.

Rewards do more than register returns. The environment delivers realized gains, but the agent evaluates outcomes through Sharpe, Sortino, or -linked measures. This separation is deliberate. It prevents confusion between what markets generate and how performance is assessed. A small but crucial distinction.

Portfolio allocation is constrained: weights are non-negative and normalized via a softmax transformation. Training relies on Proximal Policy Optimization (PPO). The clipped objective stabilizes learning while allowing smoother updates than many off-policy methods. The choice is pragmatic, not merely conventional.

As illustrated in Fig 1, factor signals feed into both state and reward. The agent makes allocations, the environment responds, and the loop continues. The form looks familiar, yet its essence is different: learning itself is guided by embedded factors. This is not trivial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Factor-Based Deep Reinforcement Learning (FDRL) framework.

https://doi.org/10.1371/journal.pone.0332779.g001

Factor definition and estimation

In this study, we define time-varying factor betas as the sensitivity of asset i’s return to a set of standardized technical indicators. Unlike traditional models such as CAPM [28] or Fama-French [6], which assume static market betas, we estimate a Dynamic- vector using 30-day rolling regressions. This approach is based on the theoretical structure of the Arbitrage Pricing Theory (APT) proposed by Ross (1976) [34], which models asset returns as a linear combination of multiple factor sensitivities and their corresponding risk premia. We extend this framework by treating technical indicators as time-varying factors and proposing a method to dynamically estimate their sensitivities.

The vector of factor sensitivities for asset i at time t is defined in Eq (1):

(1)

Each coefficient is estimated via the linear regression model in Eq (2):

(2)

Here, r_i,t is the log return of asset i at time t, f_i,k,t is the standardized value of the k-th technical factor, and is the estimated sensitivity (beta) to that factor. The factor vector consists of K = 8 elements. Below, we define each technical factor f_i,k,t, along with its economic interpretation:

1. (1) Momentum Factor : Momentum is defined as the 20-day price return, as shown in Eq (3):(3)
   where P_i,t is the closing price of asset i at time t. This factor measures trend-following behavior.
2. (2) Volatility Factor : Volatility is measured as the standard deviation of returns over the 20-day window, as shown in Eq (4):(4)
   This reflects the traditional measure of market risk.
3. (3) Moving Average Deviation : The deviation from the 20-day simple moving average (SMA) is defined in Eq (5):(5)
   This indicator is useful for detecting mean-reversion tendencies and identifying overbought or oversold conditions.
4. (4) Volume Z-score : As illustrated in Eq (6), the volume Z-score normalizes current volume by its historical mean and standard deviation:(6)
   This factor captures volume anomalies and the strength of market reaction.
5. (5) Sharpe Ratio : The Sharpe ratio, shown in Eq (7), quantifies risk-adjusted return:(7)
   where is the mean return, is the standard deviation, and is a small constant for numerical stability.
6. (6) Maximum Drawdown (MDD) : Based on Eqs (8, 9), the drawdown factor is normalized in Eq (10):(8)(9)
   Normalized to:(10)
   This factor penalizes large losses, thus encouraging risk-averse strategies.
7. (7) Equal-Weighted Market Beta : This factor represents the sensitivity of asset i’s return to an equal-weighted market index return. It is estimated through a 30-day rolling OLS regression as shown in Eq (11):(11)
   where is the return of the equal-weighted market index.
8. (8) Policy Portfolio Beta : This factor reflects the asset’s exposure to the return of the agent’s previously executed portfolio. It is defined by the regression in Eq (12):(12)
   where is the portfolio return generated by the agent’s past policies up to time t.

Each estimated coefficient is embedded in the RL agent’s state vector. For each asset i at time t, the complete state vector is constructed as shown in Eq (13):

(13)

This 8-dimensional vector captures trend, risk, liquidity, and exposure-based signals, enabling interpretable and FDRL. For a portfolio of N assets, the full input to the RL agent forms an matrix, facilitating cross-asset comparison and consistent learning across time.

The policy factor was defined as the cumulative return generated by the agent’s own portfolio up to time t–1. No information beyond t–1 is incorporated when computing this measure, which prevents look-ahead bias. At each decision point the order of operations is fixed: market observation, policy action, trade execution, and then the update of the state representation. All subsequent estimates are therefore conditional only on past and present information, never on future outcomes.

Eight technical factors are employed to characterize asset behavior, each computed on a rolling 20-day basis: momentum, volatility, moving-average deviation, volume Z-score, Sharpe ratio, maximum drawdown, equal-weighted return, and equal-weighted policy return. All metrics are standardized using Z-scores to maintain scale comparability.

Factor sensitivities () are estimated via 30-day rolling OLS regression (Eq 14):

(14)

where r_i,t denotes the log return of asset i at time t, f_k,t is the k-th factor value, and is its estimated exposure. The regression employs K = 8 core technical indicators as predictors.

Reward function design

The reinforcement learning agent selects portfolio weights at each decision point. Here, N is the number of tradable assets. The portfolio-level exposure to systematic factors at time t, denoted , is obtained as the weighted aggregation of asset-level betas (Eq 15):

(15)

The design of the reward function is central to shaping the agent’s behavior. Five alternatives were examined, representing distinct views of portfolio efficiency and factor exposure.

(1) Sharpe-based Reward.

Defined in Eq 16, this formulation evaluates expected excess return relative to its total volatility:

(16)

R_t is the realized portfolio return, while R_f denotes the risk-free rate, assumed zero in our baseline. This ratio (Eq 16) has long served as a reference point for risk-adjusted optimization.

(2) Sortino-based Reward.

The Sortino ratio isolates downside risk, replacing the variance term with downside deviation (Eq 17):

(17)

The denominator is defined in Eq 18:

(18)

Only negative returns relative to the threshold contribute. By penalizing shortfall events while ignoring gains, this measure (Eq 18) addresses the asymmetry in investor preferences.

Sharpe and Sortino-based rewards were estimated on rolling windows of past returns (20 days in baseline, 60 and 252 days for robustness). Returns were measured as log differences. Volatility and downside volatility were computed with equally weighted sample estimators, updated online.

(3) Dynamic-beta Reward.

Factor exposures fluctuate with market conditions. To capture this, dynamic betas were estimated from rolling regressions. The reward, given in Eq 19, emphasizes portfolios whose factor exposure exceeds that of a naïve equal-weight allocation:

(19)

where . This specification (Eq 19) rewards strategies that adaptively exploit evolving factor structures.

(4) Static-beta Reward.

Some portfolio objectives require stability. The static reward aligns the portfolio with predefined factor targets as in Eq 20:

(20)

Here, is the exposure of asset i to factor k, and encodes the desired orientation. Eq 20 encourages controlled alignment with theoretical or policy-driven benchmarks.

(5) Momentum-beta Reward.

Factor exposures are not only levels but trajectories. Momentum in factor sensitivity is captured through incremental changes (Eq 21):

(21)

Positive contributions occur when portfolio weights increase exposure to factors showing rising influence. This specification (Eq 21) highlights directional adjustments.

Taken together, the five reward functions form a spectrum. Sharpe and Sortino emphasize efficiency of return relative to risk (Eqs 16–18). Static- and Dynamic- rules (Eqs 19–20) represent anchored versus adaptive alignment. Momentum- (Eq 21) highlights directional change. Their joint evaluation provides a structured comparison of portfolio objectives under reinforcement learning.

To facilitate interpretability and contrast the five reward formulations, we summarize their structural properties in Table 1. Each formulation exhibits a distinct orientation with respect to performance emphasis, volatility sensitivity, and regime adaptation. This overview clarifies the alignment between reward configuration and agent behavior under varying market conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of reward structures.

https://doi.org/10.1371/journal.pone.0332779.t001

MDP construction

Portfolio allocation was cast as a Markov Decision Process within a custom OpenAI Gym setup, enforcing full investment and transaction costs.

The state S_t collects standardized market signals—20-day momentum, volatility, moving-average gap, volume Z-score, Sharpe ratio, and drawdown—plus two factor exposures. One is equal-weighted, the other policy-driven, both estimated by 30-day rolling OLS.

Actions A_t are continuous portfolio weights , normalized through a softmax so that .

Rewards follow five options: Sharpe, Sortino, momentum-, static-, and dynamic-. Each emphasizes a different trade-off between growth, downside control, and factor sensitivity.

Transitions update prices and factors daily. The policy , parameterized by a neural network, was trained with PPO using clipped objectives and entropy regularization. This offered stability while still adapting under frequent rebalancing.

Policy network and optimization algorithm

The policy network is implemented as a multi-layer perceptron (MLP) that maps the state matrix into a portfolio weight vector . A softmax layer enforces the budget constraint, ensuring non-negative weights summing to one:

(22)

Here, denotes the parameterized policy mapping states into raw scores.

Training employs Proximal Policy Optimization (PPO) for stability and efficiency. Its clipped objective is

(23)

where is the likelihood ratio and the estimated advantage.

An entropy term augments the loss to prevent premature convergence:

(24)

Network architecture, optimization hyperparameters, and data inputs remain fixed across experiments. The only variation is the reward specification, isolating the effect of reward shaping on policy learning and portfolio outcomes.

Ethics statement

This study relies exclusively on publicly available, aggregated financial time-series data (daily prices and trading volumes) and does not involve human participants, individual-level records, or any sensitive personal information. In accordance with institutional and journal guidelines, no institutional review board (IRB) approval or informed consent was required.

Dataset description

To empirically evaluate the proposed DRL-based asset allocation framework, we constructed a multi-layered dataset grounded in rigorous financial theory and practical applicability. As summarized in Table 2, the dataset consists of three main asset classes—equity, cryptocurrency, and macroeconomic assets—with an additional multi-asset configuration for robustness testing under heterogeneous conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of asset classes and dataset configuration.

https://doi.org/10.1371/journal.pone.0332779.t002

The equity set covers 22 large-cap U.S. stocks, selected by market capitalization and liquidity. It combines the FANG+ group with major S&P 500 and Dow Jones constituents to secure sectoral diversity. Technology exposure includes Apple (AAPL), Microsoft (MSFT), and NVIDIA (NVDA); non-tech firms include JPMorgan Chase (JPM), ExxonMobil (XOM), and Coca-Cola (KO). Selection follows a multi-factor framework capturing size and value [4], with allocation balance informed by ML-based modeling [21]. The crypto set consists of Bitcoin (BTC-USD), Ethereum (ETH-USD), Binance Coin (BNB-USD), and Solana (SOL-USD). To avoid unstable early data, only post-June 2020 observations were retained [1]. Selection applied a top-market-cap filter [18] and correlation-based screening [36]. Macro coverage includes ETFs tied to interest rates, inflation, and commodities: TLT, IEF (U.S. Treasuries), GLD (gold), SLV (silver), and TIP (TIPS). They emphasize inflation-hedging roles [15]. Preprocessing used Dynamic- estimation [11] and news-response models [12]. A combined 28-asset universe was formed by merging 22 equities, 5 macro ETFs, and BTC-USD. This construction tests cross-market adaptability and diversification of DRL strategies [2].

Daily prices from 2015-01-02 to 2025-06-30 were obtained via Yahoo Finance. Dates were aligned through common indexing, with missing values forward-filled [16]. Data reliability was checked against ML-based forecasting quality protocols [38].

All price and volume series were obtained from Yahoo Finance, using only publicly available, non-identifiable market data. The collection and analysis of these data complied with the terms and conditions of Yahoo Finance and did not involve any intervention or interaction with human participants.

Preprocessing and feature engineering

Data could not be used in raw form. Prices and trading volumes from different assets were first placed on a single calendar to maintain consistency, and gaps were filled forward. This step, though minor, was essential to prevent artificial jumps. Indicators were then standardized, but with two distinct horizons: technical measures employed 20-day rolling z-scores to capture short swings, while values relied on 30-day windows to reveal more persistent exposures. What emerged was the state vector. Rather than a simple stack of variables, it condensed recent market behavior and factor sensitivities into a form that the policy network could interpret for decision-making.

Experimental setup

Yahoo Finance provided daily prices and volumes from Jan 2015 to Jun 2025. Series were aligned to trading days, with gaps forward filled. A small step, yet crucial. Data were split into training (2015–2022) and testing (2022–2025). Learning hinged on PPO. SAC and TD3 were only for comparison. Agents shared one architecture, implemented in Stable-Baselines3, trained for 50,000 steps with seed 42. Features came from two horizons: technical indicators standardized over 20 days, and Dynamic- values from 30-day rolling regressions. These formed an state matrix—not just a pile of signals, but a compressed picture of market conditions. Trading costs were set at 10 bps per dollar traded; returns are reported net of costs. Turnover, , was measured daily and annually. Returns are reported net of frictions.

Baseline models and comparison strategies

The FDRL framework was evaluated against both traditional and learning-based references. As passive control, the Equal-Weighted Portfolio distributes identical shares across assets. The Mean–Variance Portfolio follows the Markowitz paradigm, relying on historical means and covariances to generate static allocations.

Within DRL, five variants were constructed according to reward design: Sharpe- and Sortino-based objectives, and three factor-oriented forms (Dynamic-, Static-, Momentum-). Each variant emphasized a different trade-off, ranging from general risk-adjusted efficiency to explicit sensitivity to market factors. All experiments adopted uniform settings: daily rebalancing, a 0.1% transaction fee, and identical hyperparameters.

Performance comparisons relied on profitability, risk, and turnover metrics, consistent with the evaluation scheme detailed in the previous subsection.

Evaluation metrics

Portfolio performance was assessed across profitability, risk, trading intensity, and interpretability. Profitability was measured by cumulative return and compound annual growth rate (CAGR). Risk efficiency relied primarily on the Sharpe ratio [28], with statistical refinements from Lo [39], Memmel [40], and Ledoit–Wolf [41]. To emphasize downside protection, the Sortino ratio [30] was also considered.

Downside exposure was summarized by maximum drawdown (MDD), the peak-to-trough loss standard in portfolio evaluation [5,16]. Trading activity was captured through turnover, reported daily and annualized via the formula for absolute weight changes, consistent with recent RL-based studies [5,16].

Interpretability was evaluated through the trajectory of time-varying coefficients, enabling detection of regime-dependent shifts in factor sensitivities. This perspective not only clarifies allocation patterns but also links reinforcement learning outcomes to established theories of asset pricing [1,2,9,11,12].

Performance by asset class

This section contrasts portfolio strategies across four domains: Equity, Cryptocurrency, Macroeconomic, and Multi-Asset. Two traditional references—Equal-Weight and Mean-Variance—are evaluated against five DRL extensions of PPO, namely Sharpe-based, Sortino-based, Momentum-, Static-, and Dynamic- rewards. The purpose is to uncover how strategy design interacts with asset-specific features.

Equity asset portfolio performance.

Table 3 summarizes the performance of equity strategies from 2022-05-02 to 2025-06-27 under a 60-day rolling estimation window (w = 60), while Fig 2 illustrates the corresponding cumulative portfolio value trajectories.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Cumulative portfolio value comparison for equity strategies.

https://doi.org/10.1371/journal.pone.0332779.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance metrics of equity-based portfolio strategies (w = 60).

https://doi.org/10.1371/journal.pone.0332779.t003

The factor-oriented variants emerged as the most favorable configurations. Static- and Momentum- achieved Sharpe ratios of approximately 1.28 and Sortino ratios above 2.20, producing cumulative gains near 93 Dynamic- performed comparably while generating the lowest turnover and trading costs among PPO-based models, indicating a more execution-efficient design.

The Equal-Weight portfolio remained a strong and stable baseline. Its Sharpe ratio exceeded 1.13, CAGR reached 22.1 In contrast, the Mean–Variance strategy exhibited structural fragility: its Sharpe ratio collapsed to 0.30, cumulative gains were limited to 9.4

Within PPO-driven benchmarks, Sharpe-PPO and Sortino-PPO formed a tight high-performance cluster (Sharpe 1.28), but both remained slightly below the factor-sensitive variants in cumulative return. Overall, these results reinforce the view that explicit modeling of factor exposures—beyond generic risk-adjusted optimization—is crucial for extracting equity premia.

Crypto Asset Portfolio Performance.

Table 4 reports performance for crypto-asset portfolios under a 60-day window, and Fig 3 illustrates cumulative wealth trajectories. Benchmarks (Equal-Weight, Mean–Variance) are contrasted with PPO-based reward variants and factor-augmented designs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cumulative portfolio value for crypto-asset strategies.

https://doi.org/10.1371/journal.pone.0332779.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance metrics of crypto-based portfolio strategies (w = 60).

https://doi.org/10.1371/journal.pone.0332779.t004

Equal-Weight achieved a Sharpe ratio of 0.93 and a CAGR of 44.69%, although this was accompanied by substantial drawdowns of approximately –43%. Mean–Variance delivered slightly lower cumulative gains but exhibited the strongest downside protection, attaining the highest Sortino ratio (1.74) and limiting maximum drawdown to –33%.

Learning-based strategies showed relatively tight clustering. Sharpe-PPO, Sortino-PPO, and the factor-driven PPO variants (Momentum-, Dynamic-, Static-) recorded Sharpe ratios near 0.80, Sortino values between 1.39 and 1.42, and CAGRs around 3233%. Among these, Sortino-PPO marginally outperformed the other DRL specifications, yet all PPO-based models delivered weaker cumulative growth than the static baselines.

Fig 3 visualizes this divergence. Equal-Weight and Mean–Variance exhibited stronger early appreciation, while PPO-based models followed smoother but slower recovery paths. Despite reduced volatility, factor-sensitivity modeling did not translate into superior long-horizon performance in the crypto domain for the examined period.

Macro Asset Portfolio Performance.

Table 5 reports performance for macro-asset portfolios under a 60-day rebalancing window (w = 60), while Fig 4 illustrates cumulative value trajectories. Benchmarks (Equal-Weight, Mean–Variance) are contrasted with PPO-based reward variants, including factor-oriented specifications.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Cumulative portfolio value for macro-asset strategies.

https://doi.org/10.1371/journal.pone.0332779.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Performance metrics of macro-asset portfolio strategies (w = 60).

https://doi.org/10.1371/journal.pone.0332779.t005

Under w = 60, the Equal-Weight macro portfolio delivered a Sharpe ratio of 0.68 and Sortino of 1.22, with a cumulative gain of 25.1% and an annualized return of 7.9%. This performance, however, came with a maximum drawdown of approximately –13.2% and moderate trading intensity (daily turnover ; annual cost ). Mean–Variance allocation lagged substantially, with Sharpe collapsing to 0.25, cumulative return limited to 4.4%, and only modest risk reduction, despite lower volatility and turnover.

PPO-based designs achieved superior risk-adjusted performance. Sharpe-PPO attained the highest Sharpe ratio (0.84) and Sortino (1.47) among all strategies, while maintaining volatility near 8.5% and a cumulative return of 23.2%. Sortino-PPO showed a similar profile, with slightly lower return but nearly identical downside protection.

Within the factor-sensitive class, Momentum-, Dynamic-, and Static- strategies produced cumulative gains in the 18–21% range, with Sharpe ratios between 0.71 and 0.77 and drawdowns contained near –10%. Although these -aware variants did not dominate Equal-Weight in absolute return, they consistently improved the trade-off between return and risk relative to Mean–Variance, especially when incorporating explicit downside-focused rewards.

Fig 4 visually reflects these patterns: Equal-Weight traces the highest terminal wealth but experiences deeper interim losses, whereas PPO-driven and -aware policies follow smoother trajectories with tighter drawdowns. This suggests that, in macro assets, reinforcement-learning–based allocation can enhance risk efficiency without sacrificing much of the long-run growth offered by simple diversification.

Multi-asset portfolio performance (PPO, SAC, TD3).

Seven strategies were evaluated between 2022-05-02 and 2025-06-27 using a fixed window length of w = 60. Equal-weighting remained a strong benchmark, attaining Sharpe 1.23, Sortino 2.08, and 80.6% cumulative growth (CAGR 21.1%) with moderate volatility (16.7%) and shallow drawdowns (MDD –16.7%). Mean–Variance, by contrast, generated only 7.0% cumulative gain (CAGR 2.3%) with Sharpe 0.36, while incurring very high trading intensity (daily turnover , annual cost ).

PPO-based policies consistently outperformed Mean–Variance on a risk-adjusted basis and slightly improved upon Equal-Weight. As summarized in Table 6, Sharpe-PPO and Sortino-PPO achieved Sharpe ratios around 1.41 with cumulative returns close to 90% and drawdowns contained near –15.8%. -sensitive variants (Momentum-, Dynamic-, and Static-) remained in a similar band: cumulative gains of 86–91%, Sharpe 1.37–1.41, and only marginally higher volatility than Equal-Weight. Daily turnover stayed around 1.6–1.8, implying annual transaction costs on the order of 0.4–0.5%. Fig 5 shows that all PPO paths track each other tightly, with -enhanced versions exhibiting slightly steeper growth after mid-2024.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Cumulative portfolio value paths under PPO-based strategies.

https://doi.org/10.1371/journal.pone.0332779.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Performance metrics of multi-asset portfolios (PPO-based, w = 60).

https://doi.org/10.1371/journal.pone.0332779.t006

SAC-based families shifted the performance distribution upward relative to Mean–Variance while preserving tight risk control. Sharpe-SAC and Sortino-SAC yielded Sharpe ratios of 1.27 and 1.33 with cumulative returns of 81.5% and 79.0%, respectively, and volatility around 15–16%. Dynamic- lowered volatility to 14.4% and reduced drawdown to about –14.7%, trading off some return (73.9% cumulative). Static- delivered the strongest gain among SAC variants (86.9%, CAGR 22.4%), at the cost of slightly higher turnover (2.35) and annual trading cost (0.59%). Fig 6 highlights how factor-aware SAC paths, especially Dynamic- and Static-, separate from the baseline after 2024 while maintaining controlled downside risk.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Cumulative portfolio value trajectories under SAC-based strategies.

https://doi.org/10.1371/journal.pone.0332779.g006

TD3 families posted the steepest climb among all RL agents. Sharpe-TD3 and Sortino-TD3 achieved 91.7% and 87.7% cumulative returns, respectively, with Sharpe ratios above 1.51 and Sortino above 2.65. Volatility compressed to 13.4–14.6% and drawdowns stayed around –14–15%, so improvements in return were not obtained by simply levering up risk. Factor-aware TD3 variants further amplified performance: Dynamic- and Static- delivered 92.1% and 88.8% cumulative gains (CAGR 23–23.5%), while Momentum- remained somewhat more conservative at 71.1% but still comfortably ahead of Mean–Variance. These gains came with higher trading intensity (turnover 2.6–3.2, annual costs 0.65–0.81%), yet the net effect in Fig 7 is a clear domination of TD3 paths over static benchmarks throughout 2024–2025.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Cumulative portfolio value trajectories under TD3-based strategies.

https://doi.org/10.1371/journal.pone.0332779.g007

Statistical validation

To guard against spurious findings, we combined parametric and nonparametric inference. Sharpe-based rewards served as the baseline, and alternative reward functions (Sortino, Momentum-, Dynamic-, Static-) were evaluated using a battery of tests: Newey–West HAC t-statistics, Wilcoxon signed-rank tests, two-sample t-tests, Mann–Whitney U tests, Memmel-adjusted jackknife Sharpe ratio z-statistics, Lo’s heteroskedasticity-consistent Sharpe intervals, and moving-block bootstrap (MBB) resampling. Supporting Information S3 Table report full results by asset group, algorithm, and window length (30, 60, 90, 120 days). Table 7 reports cases where alternative rewards crossed conventional significance levels.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Summary of statistically significant differences relative to Sharpe (SR) baseline.

https://doi.org/10.1371/journal.pone.0332779.t007

Across asset classes, HAC-t and Wilcoxon tests often flagged highly significant differences between Sharpe-based and alternative rewards. However, the block bootstrap was much more conservative: in many cases,the MBB two-sided p-values remained well above conventional thresholds, indicating that differences detected by classical test statistics were not always robust to serial dependence and resampling uncertainty.

In cryptocurrencies, all four PPO variants (Sortino, Momentum-, Dynamic-, Static-) generated sizable mean differences relative to the Sharpe baseline over several window lengths. HAC-t, Wilcoxon, and jackknife Sharpe tests frequently rejected the null of equal performance. Yet, MBB p-values never fell below 0.05 in any window, implying that crypto reward differences are statistically fragile once time-series dependence is accounted for.

Macro portfolios displayed a clearer pattern. Under PPO, Dynamic- (and, to a lesser extent, Static-) achieved the only instances in which HAC-t, Wilcoxon, and MBB all crossed conventional significance levels. In particular, Dynamic- at the 60-day window produced robust rejections of the Sharpe baseline, while Static- showed marginal evidence at the 10% level. These results suggest that modest but persistent reweighting of -exposure improves upon Sharpe-based training for macro assets.

Multi-asset experiments revealed algorithm-specific sensitivities. For SAC, Sortino-, Momentum--, and especially Dynamic--based rewards yielded robust differences from the Sharpe baseline: several window lengths exhibited joint significance under HAC-t, Wilcoxon, and MBB. Static- effects were weaker and did not survive bootstrap correction except marginally.

For TD3, the strongest and most persistent signals appeared in Sortino and Momentum- rewards. These comparators generated MBB p-values below 0.05 across multiple window lengths, whereas Dynamic- and Static- showed only sporadic or window-specific deviations from the Sharpe baseline. Overall, the multi-asset results indicate that -sensitive and downside-aware rewards can materially alter TD3 and SAC allocations, but the strength of the effect depends on both the RL algorithm and the investment horizon.

Robustness checks drew on HAC inference [42], Wilcoxon tests [43], jackknife-adjusted Sharpe ratios [39–41], and moving-block bootstrap resampling [44,45]. Detailed statistics are provided in Supporting Information S2 Table.

Turnover and transaction costs

To assess the economic relevance of the results, we also examined realized turnover and fee drag across domains, algorithms, and window lengths. Supporting Information S4 Table reports mean daily turnover, implied annual transaction costs under a 10 bps linear fee, and the resulting gap between gross and net performance measures (Sharpe ratios and annualized returns).

In equities, equal-weight portfolios exhibited moderate rebalancing intensity: mean daily turnover was close to 1% across windows, implying annualized costs of roughly 0.25% of initial wealth under a 10 bps fee. Mean–variance portfolios, by contrast, required very frequent re-optimization of weights. Their mean daily turnover exceeded 10% in all windows, and the corresponding annualized costs approached 2.5%. As a consequence, gross Sharpe ratios and gross annual returns that looked acceptable ex ante were largely arbitraged away once realistic trading costs were applied. PPO-based policies fell between these two extremes: turnover clustered around 2% per day, leading to annual cost figures in the 0.4–0.5% range and a noticeable, but not catastrophic, erosion of Sharpe and annual returns.

Crypto portfolios showed the strongest sensitivity to turnover. Mean–variance policies generated mean daily turnover on the order of several percentage points (around 6% per day), with annual cost figures in the 140–150% range. Equal-weight portfolios, by contrast, traded only sporadically and incurred essentially zero transaction costs. This cost drag is material relative to the already high volatility of crypto returns, and it explains why net annualized performance measures fall substantially below their gross counterparts in Supporting Information S4 Table despite seemingly attractive raw Sharpe ratios. PPO-based crypto policies reduced turnover somewhat relative to mean–variance allocations, but still incurred annualized cost levels of roughly 120–140% under the assumed fee schedule.

For macro assets, turnover and fee drag were markedly lower. Equal-weight portfolios rebalanced infrequently, with mean daily turnover well below 1%, and annual cost levels around 0.10–0.12%. Even for mean–variance and PPO-based policies, annualized costs remained in the low single-digit percent range rather than the triple-digit levels observed in crypto. This helps explain why macro strategies exhibit a narrower gap between gross and net Sharpe ratios, and why modest improvements in reward design (e.g., Dynamic-) translate more cleanly into net performance.

Multi-asset portfolios occupy an intermediate position. Equal-weight multi-asset benchmarks again show turnover close to 1% per day and annual cost figures near 0.25%. Mean–variance allocations incur substantially higher turnover (around 8% per day), with annualized costs around 2%. RL-based policies (PPO, SAC, and TD3) generally trade more than equal-weight but less than mean–variance: mean daily turnover for these agents lies in the 1.5–3% range, yielding annual cost levels from roughly 0.4% to 0.7%. In several SAC and TD3 configurations, the gross outperformance of alternative reward functions is partially offset by higher trading intensity, underscoring the importance of evaluating reward design jointly with turnover and fee-induced performance erosion. Taken together, Supporting Information S4 Table indicates that (i) naive optimization can generate unrealistically high turnover and fee drag, particularly for mean–variance policies in volatile domains, and (ii) reward shaping that improves gross performance does not automatically improve net performance unless it also controls trading intensity. Our subsequent discussion therefore focuses on configurations where statistically significant gains relative to the Sharpe baseline remain economically meaningful after transaction costs.

Regime-conditional validation

To address reviewer concerns regarding the robustness of beta estimation and its economic validity across market regimes, we complement the statistical tests above with regime-conditional performance analysis. As shown in Supporting Information S6 Table, PPO-based strategies maintain consistent ordering under bull, bear, high-volatility, and low-volatility phases across all asset groups (equity, crypto, macro, multi-asset). This confirms that Dynamic- and Momentum- formulations capture time-varying factor exposure rather than in-sample noise.

Furthermore, S6 Table reports an alternative crypto-specific regime decomposition (BTC-led proxies). The results reinforce that the relative performance of beta-sensitive rewards remains stable even under different definitions of market states, supporting the validity of -driven reward design under regime shifts.