Ethics statement

This study involves analysis of only publicly available de-identified data. As a result, the Committee of Protection of Human Subjects at Dartmouth College determined that the proposed activities are not research involving human subjects, thus institutional review board (IRB) review was not required. The IRB waiver is documented in STUDY00033374.

Data description

To evaluate the reproducibility of deep learning models across diverse populations and contexts, three publicly available datasets were used, namely OhioT1DM [12], DiaTrend [14], and T1DEXI [15]. These datasets comprise high-quality data collected from FDA-approved CGM devices manufactured by Dexcom, Abbott, and Medtronic [16]. Our selected datasets also differ substantially in cohort characteristics, data collection protocols, and glycemic management profiles. Table 1 presents an overview of three clinically validated metrics for assessing CGM data, namely average blood glucose, time in range (TIR), and glycemic variability (GV) [17]. The average daily blood glucose reflects the mean of daily blood glucose levels. Time in Range (TIR) is an indicator of glycemic control within the target euglycemic range of 70–180 mg/dL and has been associated with a reduced risk of complications [18]. Glycemic variability (GV) captures the degree of glucose fluctuations and is linked to hyper- and hypoglycemia risk and metabolic instability [19]. We quantify GV using the coefficient of variation (CV), defined as the standard deviation of glucose values divided by the mean glucose value and expressed as a percentage. All three metrics were computed at the day level for each participant and then averaged across all days on record to obtain participant-level summaries.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Overview of three publicly available datasets used for reproducibility analysis. Our final study subset included 128 participants with over 1.36 million glucose samples¹.

https://doi.org/10.1371/journal.pdig.0001633.t001

OhioT1DM is the most widely used benchmark in prior studies [20–27], however, it consists of a small but intensively monitored cohort (n = 12). This dataset contains multimodal data collected from wearable devices and manual user logs, including CGM measurements recorded at 5-minute intervals, basal and bolus insulin delivery records from insulin pumps, self-reported meal times, meal types, and estimated carbohydrate content, step count data from the Basis Peak sensor band, as well as additional physiological and behavioral information, such as sleep quality, stress levels, and heart rate. With respect to glycemic control, participants in this dataset exhibit moderate glucose management, with an average daily TIR of 63% (clinical target: TIR > 70% [30]).

DiaTrend comprises data from 54 individuals with type 1 diabetes (T1D) who demonstrate more heterogeneous and suboptimal glucose control, with an average daily TIR of 55% (clinical target: TIR > 70% [30]). The DiaTrend dataset includes data from CGMs and insulin pumps, comprising CGM measurements sampled at 5-minute intervals. However, basal and bolus insulin delivery records are only available for the final 1–2 months of the data collection period.

In contrast, T1DEXI features a larger, better-managed, and more physically active cohort, with an average daily TIR of 73%, (clinical target: TIR > 70% [30]). This dataset includes 5-minute interval CGM time-series data, step count, and physical activity information recorded by the Verily Study Watch, basal and bolus insulin delivery records from insulin pumps or multiple daily injections (MDI), and self-reported meal intake information. Fig 2 shows that these datasets collectively span a wide spectrum of glycemic control and further highlight participant-level differences across average daily BG, daily TIR, and daily glycemic variability. Therefore, including these three datasets offers an opportunity to examine model reproducibility and generalization across varying diabetes management conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of key metrics for assessing blood glucose management in three publicly available datasets - OhioT1DM [12], DiaTrend [14], and T1DEXI [15].

A) Daily average blood glucose per participant, B) Daily time in range ratio per participant, and C) Daily glycemic variability per participant.

https://doi.org/10.1371/journal.pdig.0001633.g002

Although these three datasets contain multiple data modalities from different sources, this study mainly used the CGM time-series data. The CGM devices differed across datasets: the OhioT1DM dataset was collected with Medtronic Enlite CGM sensors, DiaTrend was collected with CGMs from Dexcom, Abbott, and Medtronic, while T1DEXI primarily included Dexcom G6 CGMs. All CGMs here comply with ISO 15197:2013 and FDA accuracy standards, requiring 95% of measurements to fall within 15 mg/dL (for glucose <100 mg/dL) or 15% (for glucose 100 mg/dL) [26]. These systems are widely used clinical standard for managing type 1 diabetes; therefore, the impact of device-specific variability is expected to be minimal.

To account for differences in dataset structure, annotation availability, and recording duration across the three datasets, we applied dataset-specific inclusion and harmonization criteria. The goal was to ensure that each included participant had sufficient CGM continuity and relevant event annotations for the intended analyses, while also preventing any single dataset or participant from disproportionately influencing the results. For OhioT1DM, we retained the original 12 participants and preserved the official train–test split provided by the dataset creators. For DiaTrend, we included 53 of the 54 available participants, excluding one participant (#52) with fewer than 28 days of contiguous CGM data, and restricted each participant to the most recent eight weeks of data to reduce imbalance caused by participants with years of recordings and to prioritize periods with more consistent meal and bolus annotations. For T1DEXI, we selected participants with good data consistency–at least seven consecutive days of 70% CGM coverage and at least one meal and bolus record per day; this yielded a subset of 63 participants, providing sufficient event-annotated data while maintaining a sample size comparable to DiaTrend and OhioT1DM. Across all datasets, CGM data were harmonized to a 5-minute sampling interval when needed. The selected participants’ IDs were listed in S1 Table.

Model selection and characteristics

In this study, we sought to investigate the technical, statistical, and conceptual reproducibility of several SOTA deep learning methods for blood glucose prediction. To select high-quality methods from prior work for reproducibility evaluation, we considered the following four criteria: (1) code availability and/or detailed model reporting, (2) evaluation on the publicly available OhioT1DM dataset, (3) strong performance for short-term prediction (30 min), and (4) diversity of model architectures. More specifically, we prioritized studies that provided either publicly accessible code or sufficiently detailed methodological descriptions to support faithful reimplementation. We also restricted model selection to studies that evaluated performance on the OhioT1DM dataset to enable direct comparison between the originally reported results and our reproduced results. As the most widely used publicly available benchmark in glucose forecasting research, OhioT1DM serves as a common reference point for methodological comparison (Fig 1), providing a consistent basis for assessing technical, statistical, and conceptual reproducibility. To ensure consideration of high-quality methods for short-term glucose forecasting, we also required strong reported performance at the 30-minute prediction horizon on OhioT1DM (i.e., RMSE < 20 mg/dL). Finally, to capture the dominant modeling paradigms in prior literature, we selected model architectures that are LSTM-based, CNN-based, and Transformer-based, which account for 43%, 13%, and 12%, respectively, of model architectures used in prior work based on our literature review (see S1 File).

Leveraging the above criteria, we selected and experimentally replicated eight representative studies: three LSTM-based models—Martinsson et al. (2019) [20], van Doorn et al. (2021) [21], and Rabby et al. (2021) [24]; two CNN-based models—Li et al. (2020) [22] and Deng et al. (2021) [23]; and three Transformer-based models—Lee et al. (2023) [25], Zhu et al. (2025) [26], and the PatchTST implementation in Karagoz et al. (2025) [27]. Table 2 summarizes the eight selected studies in terms of their datasets used for training and evaluation, input features, code availability, and model type. Although these models share certain methodological and implementation similarities, they also exhibit notable differences across datasets, input configurations, and evaluation criteria. With respect to datasets, all eight studies evaluated their proposed models on the OhioT1DM dataset. However, there were some differences in their evaluation strategy. For example, some studies trained and tested their models on only a subset of the dataset, specifically the first six participants released in 2018 [20–22, 24], while others used the full set of twelve participants [23]. In contrast, some other studies trained their models on external datasets and used OhioT1DM solely for evaluation [25–27]. Furthermore, some studies incorporated additional datasets during model training. For example, Li et al. [22] used the ABC4D dataset to incorporate more participants with type 1 diabetes (T1D), while van Doorn et al. [21], Deng et al. [21], and Lee et al. [25] included individuals with type 2 diabetes (T2D) during pretraining to enhance data diversity. However, when evaluating on the OhioT1DM dataset, these studies applied transfer learning or retraining procedures to adapt the models specifically to T1D data. Therefore, despite differences in pretraining data composition, all selected models were ultimately evaluated under a T1D setting and remained appropriate for our reproducibility analysis.

Regarding input features, five studies relied exclusively on historical CGM data [20,21,23,25,26], whereas others incorporated additional contextual data such as insulin delivery, carbohydrate intake, or step count [22,24,27] as shown in Table 2. Differences also emerged in evaluation metrics. While most studies reported performance across multiple metrics and prediction horizons, the most common evaluation metric used was the Root Mean Squared Error (RMSE) between predicted and actual glucose values at a 30-minute prediction horizon - see Eq 1. Exceptions include Lee et al. (2023) [25], which reported only mean absolute percentage error (MAPE), and Karagoz et al. (2025) [27], which computed RMSE over the entire prediction sequence rather than at the fixed 30-minute prediction horizon, as shown in Eq 2. This sequence-level RMSE can yield lower reported errors by averaging performance across earlier and typically easier prediction steps. By contrast, RMSE₃₀ reflects the error at the clinically relevant 30-minute prediction horizon and is consistent with best-practice guidelines and recent literature reviews [11, 31]. Therefore, in our reproducibility evaluation, we selected the sampling horizon (i.e., the duration of historical input data) with the best reported performance in each prior study for replication, and used RMSE₃₀ as the primary evaluation metric. More detailed model specifications and implementation information are provided in S2 Table and S2 File.

(1)

(2)

where N is the number of test samples and L is the prediction sequence length (L = 30 minutes in this study).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Summary of the eight deep learning studies selected for replication. The table lists each study’s input features, training and evaluation datasets, code availability, sampling horizons, model architecture, and major reported metrics.

https://doi.org/10.1371/journal.pdig.0001633.t002

In addition to the differences summarized in Table 2, differences in the input feature set, model architecture, and input horizon also influenced how missing data was handled. For LSTM- and CNN-based models or models using shorter input horizons with CGM-only inputs [20,21,23,25], predictions were limited to available continuous CGM segments to preserve temporal consistency. In contrast, models that incorporated additional lifestyle features (e.g., insulin, carbohydrate, or activity) [22,24], as well as Transformer-based models with longer input horizons [26,27], addressed missing data through linear or spline interpolation within sequences and extrapolation for terminal gaps to maintain sequence continuity. Therefore, when replicating these works, the input data were interpolated as the original studies did.

Technical reproducibility evaluation

Technical reproducibility refers to the ability of a model’s result to be fully replicated under identical experimental conditions [13]. To evaluate the technical reproducibility of the eight deep learning models, we sought to reproduce them using the same experimental settings, including the same training and evaluation dataset, exact input features, missing data handling process, model architecture, training, validation, and test logic, and evaluation metric reported in the original publication. In other words, regardless of the methodological differences summarized in Table 2, we reproduced each study’s reported results by following either the authors’ released code or the implementation details described in the paper. However, the inconsistencies existing throughout the entire pipeline make direct comparison between studies difficult, highlighting the necessity for more standardized and aligned evaluation protocols.

Statistical reproducibility evaluation

Statistical reproducibility refers to the ability of a model’s result to be “upheld under resampled conditions that may yield mildly different numerical results but should not statistically affect the claimed result” [13]. In this study, rather than resampling, which would further reduce the already limited OhioT1DM dataset, we trained all eight models using the complete training portion of the OhioT1DM dataset and evaluated them on its test portion. For some studies [20–24], this adjustment effectively acted as an “upsampling,” incorporating a broader range of training instances from the same dataset. However, for studies originally trained on external datasets [25–27], the change primarily served to maintain consistency between training and evaluation setups by preserving a comparable data distribution. This unified experimental setting ensures that observed performance differences reflect variations in model design and implementation rather than inconsistencies in data usage. While this approach differs from the resampling-based definition, it serves as a practical adaptation of statistical reproducibility for a small, real-world biomedical dataset. In addition, we refined the input configurations of models that originally incorporated multiple input factors [22,24,27] by comparing their technical replication results using either a single CGM input channel or multiple channels integrating additional lifestyle features, as shown in S3 Table. For the statistical reproducibility evaluation, the implementation of Karagoz et al. (2025) remained multi-channel, whereas Li et al. (2020) and Rabby et al. (2021) were optimized to use a single CGM input channel due to better performance during the technical reproducibility evaluation.

In addition to unifying and optimizing the model inputs, the statistical reproducibility evaluation also standardized the output metrics to the conventional RMSE at a fixed 30-minute prediction horizon, as defined in Eq 1. Overall, the implementation for statistical reproducibility evaluation unifies the input and evaluation metrics for the eight methods, while maintaining the consistency and integrality of the blood glucose prediction pipeline. This standardization further ensured comparability across methods and allowed observed performance differences to be attributed primarily to variations in model architecture and learning dynamics.

Conceptual reproducibility evaluation

Conceptual reproducibility describes whether the intended effect of a model can be reproduced under conditions that align with its high-level purpose [13]. To assess conceptual reproducibility, we evaluated the same eight methods on two additional publicly available datasets—DiaTrend [14] and T1DEXI [15]. Both datasets include individuals with type 1 diabetes but differ substantially from OhioT1DM in cohort size and overall glycemic control (Table 1, Fig 2). All models were independently retrained on the DiaTrend and T1DEXI datasets, respectively, to ensure dataset-specific learning rather than transfer from OhioT1DM. For the DiaTrend and T1DEXI datasets, we performed 5-fold cross-validation, training on four folds and evaluating on the remaining fold, iterating until all participants were tested as unseen subjects. This setup ensured a fair comparison across models and directly assessed each model’s ability to generalize to new individuals. Apart from replacing the training and evaluation datasets (OhioT1DM) with DiaTrend and T1DEXI, and adopting a 5-fold cross-validation scheme, all other experimental settings were identical to those used in the statistical reproducibility evaluation described in the previous subsection.

Technical reproducibility

The technical reproducibility evaluation examined whether the reported performance of each deep learning model for blood glucose prediction could be replicated under the same experimental settings as described in their original studies. As summarized in Table 3, the RMSE values reported across the seven studies (excluding Lee et al., 2023) ranged from 15.96 to 19.28 mg/dL, while our exact replication results ranged from 15.45 to 19.41 mg/dL. The results of Martinsson (2019), Deng (2021), Zhu (2025), and Karagoz (2025) were closely reproduced with minimal deviation likely attributable to inherent training stochasticity, whereas Li (2020), van Doorn (2021), and Rabby (2021) yielded slightly higher RMSEs but still within of the reported values, indicating good overall reproducibility.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Overview of technical and statistical reproducibility evaluation of eight deep learning papers on 30-minute blood glucose prediction. Paper reported RMSE (mg/dL) is the authors’ reported RMSE for future 30-min prediction with standard deviation (SD) in parentheses.

https://doi.org/10.1371/journal.pdig.0001633.t003

Statistical reproducibility

Table 3 presents our results from statistical reproducibility evaluation. We can observe minimal differences between the paper-reported RMSEs, technical and statistical reproducibility evaluation results, thereby showing stability across most models when retrained and evaluated under standardized conditions. Our statistical reproducibility evaluation facilitated a fair comparison of the eight blood glucose prediction pipelines by retraining and evaluating them on the standardized OhioT1DM dataset using unified evaluation metrics, as described in the Methods section. The five traditional LSTM- and CNN-based pipelines showed minimal performance variation (within ±1.21 mg/dL of their technical reproducibility results), suggesting strong robustness to unified data and metric conditions. Meanwhile, the Transformer-based model from Zhu et al. (2025) exhibited a slight performance degradation, as can be seen from the RMSE increase by 2.5 mg/dL under statistical reproducibility evaluation. However, we can observe that all methods are within 1 SD of the paper-reported RMSE and the technical reproducibility result. In contrast, from Table 3, we can observe that a discrepancy of 6.11 mg/dL between Karagoz et al. (2025)’s custom and the standardized RMSE_30min, thereby underscoring how variations in metric definitions can substantially affect reported outcomes. These findings collectively suggest that while most models retain statistical reproducibility under controlled conditions, evaluation standardization is critical to enable fair comparisons across studies.

Conceptual reproducibility

Fig 3 presents the results obtained from our evaluation of statistical (Fig 3A) and conceptual reproducibility (Fig 3B&C) for the eight deep learning papers evaluated in this study. From Fig 3B & C, we can observe varying levels of increased error in the prediction performance across all eight deep learning methods for blood glucose prediction when evaluated on DiaTrend and T1DEXI. More specifically, we observe differential performance with an increase in prediction error by 4.89 to 5.97 mg/dL on the DiaTrend dataset and -0.34 to 2.51 mg/dL on the T1DEXI dataset - see Fig 3D & E. Furthermore, we observe increased variability in prediction performance across all eight methods on DiaTrend and T1DEXI compared to OhioT1DM - see Fig 3A-C and S5 Table. In particular, the prediction performance across all eight methods ranged from 14.0 - 25.6 mg/dL RMSE per individual on OhioT1DM, compared to 15.1 - 39.7 mg/dL on DiaTrend, and 9.9 - 38.5 mg/dL on T1DEXI. We compared the difference in prediction performance for all eight methods on OhioT1DM vs. DiaTrend and OhioT1DM vs. T1DEXI using t-tests. Our results showed statistically significant differences in the prediction performance of all eight methods on the DiaTrend dataset compared to performance on the OhioT1DM dataset - see Fig 3D and S7 Table. These results indicate that while all deep learning models exhibited reasonable technical and statistical reproducibility, their conceptual reproducibility was limited. This limitation, however, may not arise solely from model design or prediction pipelines, but rather from intrinsic differences in glucose management behaviors across datasets—a factor further examined in the following section. Detailed RMSE and MAE values at 30-min prediction horizon can be found in S5 Table.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Reproducibility evaluation of eight deep learning methods from prior literature for the task of blood glucose prediction 30-minutes in advance.

Evaluation results using three publicly available datasets, including A) OhioT1DM (n = 12), B) DiaTrend (subset: n = 53), C) T1DEXI (subset: n = 63). D & E) Present the differential performance of each blood glucose prediction method on the DiaTrend and T1DEXI datasets compared to the performance on the OhioT1DM dataset.

https://doi.org/10.1371/journal.pdig.0001633.g003

To examine cross-dataset robustness, we also evaluated OhioT1DM-pretrained models directly on DiaTrend and T1DEXI without retraining (S6 Table). We observed that most models exhibited modest performance degradation when using pretrained models on OhioT1DM, hence retraining each model provided the best performance.

Prediction performance differs based on glucose management

As shown in Fig 3, replication RMSEs were consistently higher on the DiaTrend dataset than on OhioT1DM and T1DEXI. This pattern aligns with participants’ glycemic management profiles (Fig 2), where higher mean glucose and lower time-in-range are indicative of below target glycemic control. These observations suggest that prediction performance may be influenced by differences in individual glucose management.

To further evaluate potential associations between prediction performance and individual glycemic control, we quantified diabetes management of each person in all three datasets using four clinically-validated metrics, namely, mean glucose, time in range (TIR), glycemic variability, and HbA1C [17]. Each CGM metric (i.e., all metrics except HbA1C) was calculated on a daily basis, then an average across all days was calculated for each participant. Fig 4 illustrates the association between blood glucose prediction performance and individual-level diabetes management metrics. Across datasets and models, lower RMSE values (i.e., less prediction errors) were consistently observed in individuals with higher glycemic control (i.e., lower mean glucose, higher TIR, lower glycemic variability, and lower HbA1C). Additionally, higher prediction errors occurred amongst individuals with lower glycemic control (higher mean glucose, lower TIR, and higher glycemic variability). The Pearson correlation coefficients ranged from 0.38 to 0.63 (mean: 0.55; p < 0.001), indicating moderate-to-strong associations, with up to 40% of performance variance attributable to glycemic management.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Association between blood glucose prediction and individual diabetes management.

This figure uses the deep learning model proposed by Martinsson et al. [18] for illustration due to its best prediction performance across the eight models. (A-D) Blood glucose prediction RMSE vs. the daily average blood glucose per person, TIR per person, Glycemic variability per person, and HbA1C per person, respectively. Higher RMSE equates to greater error in the prediction performance.

https://doi.org/10.1371/journal.pdig.0001633.g004

These findings suggest that deep learning models perform substantially better in metabolically stable individuals characterized by smoother and more predictable glucose trajectories. In contrast, individuals with higher glycemic variability and lower TIR exhibit more abrupt fluctuations and nonlinear dynamics, which increase forecasting difficulty. Importantly, these findings suggest that current models may implicitly favor well-controlled patients, potentially overestimating robustness when evaluated with datasets from a stable cohort. The reduced accuracy observed in individuals with poorer glycemic control raises concerns, as these patients may benefit most from reliable glucose prediction. Overall, the results highlight that glycemic control is a critical determinant of predictive performance and should be explicitly considered in reproducibility and generalization assessments.

As supplementary assessments, we also examined whether demographic factors—including sex/gender, age, and insulin delivery mode—are associated with differences in blood glucose prediction performance across the eight deep learning methods. Specifically, we assessed the relationship between the listed demographic variables and prediction RMSE (S9 Table, S10 Table, and S11 Table) and conducted stratified analyses by diabetes management level to better isolate the sources of performance variability (S1 Fig). These analyses suggest that differences in glucose management, rather than demographic characteristics, account for most observed performance disparities. However, it is important to note our analysis is limited because there was a significant imbalance across sex/gender (33% male vs 67% female), age (48% < 30 years), and insulin delivery mode (82% insulin pump/closed loop). This imbalance limits the strength and generalizability of this finding. Therefore, until these patterns are validated in a larger and more demographically balanced cohort, we remain cautious about drawing strong conclusions regarding demographic effects.

Challenges to reproducibility in blood glucose prediction

The experimental replication provides a concrete way to identify and understand the practical challenges that hinder reproducibility in blood glucose prediction research. Although we observed the consistency between the paper-reported results and the technical replication results from Table 3, several reproducibility challenges still can not be overlooked. These challenges can be broadly grouped into three categories: (1) data-related constraints, (2) implementation transparency challenges, and (3) inconsistent evaluation metrics. With regard to data-related constraints, many studies relied heavily on the OhioT1DM dataset, which includes only 12 participants, thereby limiting the generalizability of their results. Also, Lee et al. 2023 [25] used private datasets for model training, thereby making reproducibility evaluations by independent researchers infeasible. In addition to data-related limitations, incomplete or missing code further hampered our reproducibility evaluation studies. For example, four of the eight studies did not openly release their source code, and even when code was available, such as Zhu et al. 2025 [26] and Karagoz et al. 2025 [27], critical implementation details were missing, including data filtering procedures, preprocessing scripts, and specifications for input data construction. We also observed that for complex model architectures there were sometimes numerous unreported hyperparameters which further challenged our reproducibility studies. These missing pieces introduced challenges for replication and possibly introduced errors in the replicated result. Finally, inconsistency in evaluation introduced additional challenges when comparing the performance of results across SOTA methods. Some studies, such as Lee et al. 2023 [25] did not report standard metrics such as RMSE, while some studies, such as Karagoz et al. 2025 [27] customized their RMSE calculation, thereby making it difficult to compare performance across methods without experimental replication. These issues highlight the pressing need for greater transparency, standardized evaluation practices, and broader data accessibility to strengthen reproducibility and comparability in deep learning models for blood glucose prediction.

In addition to the findings from the technical reproducibility evaluation, the statistical reproducibility evaluation results shown in Table 3 reflect a fair comparison result between eight models, based on unified input and standardized evaluation metrics. As shown in Table 3, we observed that the three Transformer-based models achieved lower predictive performance compared with LSTM and CNN-based architectures. Similar findings have also been reported in some prior studies on time-series forecasting [32,33]. This discrepancy in prediction performance likely stems from multiple reasons. One possible reason is that those Transformer-based models rely on longer temporal windows, especially Zhu et al, 2025 [26] and Karagoz et al. 2025 [27], which have a higher requirement on data completeness and quality. Another possible reason is that the three replicated Transformer architectures have higher model complexity as shown in S2 Table, and usually require larger training datasets to achieve stable convergence [33]. Given the limited sample size of the OhioT1DM dataset (n = 12), these requirements may not be fully met, which may explain their suboptimal performance relative to the more data-efficient RNN- and CNN-based models.

Compared with the relatively stable results from the technical and statistical reproducibility evaluations, the conceptual reproducibility analysis revealed substantially lower generalizability when models were retrained on new datasets (Fig 3). The consistently higher RMSEs and variance observed on the DiaTrend and T1DEXI datasets across almost all eight methods suggest that differences in glycemic management strongly influence prediction performance. In particular, individual management heterogeneity (Fig 4) plays a decisive role: models tend to perform better for individuals with well-controlled glucose trends and worse for individuals with poorer glycemic control.

While the correlation analysis reported that demographic factors such as age and insulin delivery mode are highly relevant to prediction accuracy, a further stratified analysis (S1 Fig) did not find consistent evidence supporting strong demographic effects. Glucose management could play a more prominent role than demographics in shaping prediction performance, but further validation in larger and more balanced cohorts is needed to confirm this relationship. However, it is still worthwhile for future studies on blood glucose prediction to prioritize recruiting participants with diverse levels of glycemic control, not merely to improve model performance, but to enable fairer and more generalizable evaluations that better reflect real-world variability.

Recommendations for future work

Given the replication challenges and the differential performance observed across each individual, across management subgroups, and across datasets, we present three core recommendations toward the goal of developing robust machine learning algorithms for blood glucose prediction.

• Management-Aware Model Development A central finding of this study is that individual glycemic management characteristics are highly correlated with models’ prediction performance. Future glucose prediction research should therefore explicitly incorporate management-aware modeling strategies. This may include conditioning models on metrics such as average glucose level, TIR, and/or glucose variability; designing adaptive architecture for participants with different management profiles; and routinely reporting performance stratified by glucose management levels. Such practices would help ensure that model performance is not overstated when evaluated on datasets from individuals with less stable glucose profiles and would support the development of models robust to the challenging patterns in the real world.
• Dataset Transparency and Reporting Standards Our results highlight the extent to which dataset characteristics can influence prediction performance. To ensure fair comparison across studies, we recommend adopting minimum dataset reporting standards tailored to glucose prediction tasks. These include reporting glycemic management metrics (e.g., mean glucose, TIR/TAR/TBR, glucose variability), CGM quality indicators (e.g., percentage of missing data, interpolation ratios, device type), and relevant behavioral context (e.g., meal regularity or insulin delivery mode when available). Standardized dataset characterization would not only improve transparency but also contextualize performance differences and aid reproducibility studies across research groups.
• Data, Code, and Evaluation Transparency To support fair comparison and reproducibility, future glucose prediction studies should adopt transparent and standardized practices in dataset selection, code release, and model evaluation. Whenever possible, models should be evaluated on publicly available datasets, with clear reporting of glycemic management characteristics to contextualize performance. Complete and well-organized code releases, including detailing model architectures, hyperparameters, and implementation choices, as these are essential for technical reproducibility evaluation. Studies should also report standardized metrics (e.g., RMSE) at benchmark prediction horizons such as 30 minutes ahead to enable meaningful cross-study comparison. Although these practices have become increasingly common in the SOTA, reiterating them is still worthwhile, as they remain foundational for transparent and rigorous methodological evaluation.

Code Availability and Environment

The source code used for this study is publicly available at https://github.com/Augmented-Health-Lab/ReproducibilityStudy_DL_BGPrediction. Detailed instructions for running the experiments and replicating the analysis are provided in the associated README file.

All code developed and used in this study is based on Python 3.9.20. For all machine learning tasks, PyTorch 2.2.1 + CU118, TensorFlow 2.10.1, and scikit-learn 1.5.2 were used. Data analysis and statistics were conducted with SciPy 1.11.4, NumPy 1.26.2 and Pandas 2.1.3.