https://orcid.org/0000-0001-7915-8772
Figures
Abstract
Background
Public synthetic datasets enable transparent model comparison but cannot establish performance in clinical populations. We asked how four fixed classifiers compare under one leakage-controlled validation framework on a large synthetic heart-disease competition dataset.
Methods
The official training set contained 630,000 observations (44.83% outcome prevalence) and 13 predictors; the 270,000-row test set had no outcome labels. L2-regularized logistic regression, HistGradientBoosting, CatBoost, and LightGBM were evaluated using identical stratified five-fold outer splits. Logistic-regression preprocessing was fitted within outer-training data, and early stopping used outer-training data only. The primary measure was pooled out-of-fold (OOF) AUROC; secondary measures were average precision, Brier score, calibration intercept and slope, fold variability, and train–validation optimism. Duplicate/overlap, identifier-only, and permuted-outcome controls assessed leakage.
Results
Pooled OOF AUROC was 0.955443 for CatBoost, 0.955324 for LightGBM, 0.955066 for HistGradientBoosting, and 0.952874 for logistic regression. Corresponding average precision ranged from 0.945912 to 0.948828 and Brier scores from 0.081116 to 0.083518. Calibration intercepts ranged from −0.001299 to 0.002153 and slopes from 0.998896 to 1.004388. Mean optimism was 0.000017–0.002686. No duplicated identifiers, duplicated feature profiles, or train–test feature-profile overlap were found; identifier-only AUROC was 0.500121 and mean permuted-outcome AUROC was 0.500615.
Conclusions
All four models discriminated strongly and showed close internal calibration, while gradient-boosting models had only marginally higher AUROC than logistic regression. The principal contribution is a fully rerunnable, controlled benchmark rather than a novel algorithm or a clinically validated prediction model. External performance in real clinical cohorts remains unknown.
Citation: Ennab RM (2026) Reproducible benchmarking of four machine-learning classifiers on a large synthetic heart-disease competition dataset. PLoS One 21(9): e0354304. https://doi.org/10.1371/journal.pone.0354304
Editor: Suyan Tian, The First Hospital of Jilin University, CHINA
Received: July 4, 2026; Accepted: September 6, 2026; Published: September 21, 2026
Copyright: © 2026 Raed M. Ennab. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The official third-party competition files (train.csv, test.csv, and sample_submission.csv) are available from the Kaggle Playground Series Season 6 Episode 2 competition page under Kaggle’s access terms: https://www.kaggle.com/competitions/playground-series-s6e2. The author-generated canonical code is provided without access restriction as S1 File. Core generated outputs—including fold metrics, calibration values, data-quality checks, hyperparameters, software versions, figures, and manifests—are provided as S2 File. Combined out-of-fold predictions are divided between S1 Data and S2 Data to comply with PLOS’s per-file size limit; combined unlabeled test predictions are supplied as S3 Data; and a concise successful-run record is supplied as S3 File. The supporting files contain no patient or confidential data and will be published with the article.
Funding: The author(s) received no specific funding for this work.
Competing interests: The author has declared that no competing interests exist.
1. Introduction
Cardiovascular prediction is a prominent application of machine learning, but performance estimates depend as much on the data-generating setting and validation design as on the algorithm itself [1–3]. Public synthetic datasets are useful for transparent, low-barrier benchmarking because investigators can share complete workflows without patient privacy concerns. They do not, however, recreate every dependency, measurement process, subgroup, or prevalence pattern of a clinical cohort [4,5].
Gradient-boosted decision trees are strong candidates for mixed-type tabular data. LightGBM emphasizes histogram-based computational efficiency, whereas CatBoost provides dedicated categorical-feature handling [6,7]. A regularized logistic model remains an important comparator because it establishes whether the additional nonlinearity and computational cost of boosting produce a material gain. Evaluation should extend beyond a single discrimination statistic: calibration, probability accuracy, fold variability, and transparent overfitting checks are central to reliable prediction-model reporting [8–10].
This study does not propose a new classifier and is not presented as clinical model development. Its contribution is an audit-oriented comparison generated by one fully rerunnable notebook with shared saved folds, fold-contained fitted preprocessing, out-of-fold predictions, calibration measures, training–validation optimism, and negative controls. The research question was: under a single leakage-controlled five-fold validation framework applied to the 13 official predictors, how do logistic regression, HistGradientBoosting, CatBoost, and LightGBM compare in discrimination, calibration, apparent overfitting, and computational cost?
2. Methods
2.1. Study design and data source
We conducted a retrospective computational benchmarking study using the official Kaggle Playground Series Season 6 Episode 2 ‘Predicting Heart Disease’ files [11]. The organizers describe the data as synthetic and generated from a model trained on a heart-disease dataset. The training file contained 630,000 rows and 15 columns (identifier, 13 predictors, and outcome); the test file contained 270,000 rows and 14 columns (identifier and the same 13 predictors). Test outcomes were unavailable. Heart Disease was encoded Presence = 1 and Absence = 0. The training set contained 282,454 positive observations (44.83%) and 347,546 negative observations (55.17%). No resampling or class weighting was used because this benchmark was moderately balanced. Table 1 summarizes the training data.
2.2. Predictors and preprocessing
Every model used the same 13 supplied predictors; the identifier was excluded. Eight low-cardinality variables were modeled as categorical: sex, chest pain type, fasting blood sugar over 120, electrocardiographic results, exercise angina, ST-segment slope, fluoroscopic vessel count, and thallium category. Age, blood pressure, cholesterol, maximal heart rate, and ST depression were continuous. No derived predictors were created. No external data, pseudo-labels, model blending, or leaderboard-derived post-processing entered the analysis. Because no predictor value was missing or non-finite, no imputation was performed. Table 2 provides descriptive outcome prevalence for selected categorical predictors.
2.3. Data-integrity and split-comparability checks
Before modeling, we counted missing and non-finite values; duplicate identifiers; repeated full rows; repeated feature profiles; cross-file identifier overlap; and exact cross-file feature-profile overlap using row hashes. Continuous train–test differences were summarized using absolute standardized mean differences and Kolmogorov–Smirnov statistics. Categorical differences were summarized by total variation distance. These test-file comparisons assessed structural similarity only; they did not estimate predictive performance because test outcomes were unavailable.
2.4. Shared validation design
A single set of five stratified, shuffled outer folds (random state 42) was saved and reused for every workflow. Each outer fold contained 126,000 validation observations and one out-of-fold prediction was generated for every training row. All fitted preprocessing components and all model-selection steps used the corresponding outer-training data. For LightGBM, categorical dtype levels were assigned once from the observed predictor codes without using outcomes; every code occurred in every outer-training fold. For CatBoost and LightGBM, a stratified 10% inner holdout from each outer-training fold selected the early-stopping iteration; the chosen count was then fixed and a new model was fitted on all 504,000 outer-training observations before prediction of the untouched outer fold. HistGradientBoosting used its internal 10% early-stopping subset drawn only from outer-training data. The competition test predictions were retained for reproducibility but were not used to estimate performance.
2.5. Model specifications
Logistic regression used L2 regularization (C = 1.0), the lbfgs solver, and a maximum of 1,000 iterations. Within each outer fold, continuous predictors were standardized and categorical predictors were one-hot encoded with unknown categories ignored.
HistGradientBoostingClassifier used a maximum of 1,000 iterations, learning rate 0.06, 31 leaf nodes, minimum 20 observations per leaf, L2 regularization 0.01, internal early stopping with a 10% validation fraction, and 30 non-improving iterations. The eight categorical predictors were identified to the estimator and the random state was 42.
CatBoost used binary log loss with AUROC monitoring, native categorical handling, at most 5,000 iterations, learning rate 0.02, depth 6, bagging temperature 0.2, CPU execution, and 200-round early stopping. Deterministic fold-specific seeds 43–47 were used for inner selection and outer refitting.
LightGBM used a binary objective with AUROC monitoring, native categorical handling, at most 5,000 estimators, learning rate 0.02, maximum depth 8, 31 leaves, minimum 400 observations per leaf, feature and row subsampling of 0.8, subsampling frequency 1, L1 regularization 0, L2 regularization 2, and deterministic column-wise CPU training. Deterministic fold-specific seeds 43–47 were used, with the same inner-selection/refit design and 200-round early stopping.
2.6. Single-predictor analysis
To describe standalone signal without imposing ordinal scores on categorical codes, each predictor was also evaluated in a separate logistic-regression workflow using the same five outer folds. Continuous predictors were standardized and categorical predictors one-hot encoded within each training fold. Pooled out-of-fold AUROC was calculated from the resulting probabilities. This analysis is descriptive and is neither a causal importance analysis nor a SHAP analysis.
2.7. Performance and calibration measures
The primary measure was pooled AUROC calculated from the complete out-of-fold prediction vector. Secondary measures were mean and sample standard deviation of the five fold AUROCs, average precision (AP), Brier score, calibration intercept and slope, mean training AUROC, and mean optimism (training minus validation AUROC averaged across folds) [8,12]. Calibration intercept and slope were estimated by logistic regression (C = 106; lbfgs; maximum 1,000 iterations) of the observed outcome on the logit of out-of-fold probabilities clipped to [10−6, 1 − 10−6]. Calibration curves used ten equal-frequency bins. Because no paired inferential comparison was planned for the canonical analysis, between-model differences are described numerically rather than as statistically significant.
2.8. Leakage and negative-control analyses
Leakage safeguards comprised fold-contained fitted preprocessing, inner iteration selection isolated from each outer-validation fold, duplicate and cross-file feature-profile screening, an identifier-only AUROC, and five logistic-regression runs after independent permutations of the training outcome (seeds 20260815–20260819). Training–validation optimism provided a model-specific check for apparent overfitting.
2.9. Software and reproducibility
The successful canonical run used Python 3.12.13, NumPy 2.0.2, pandas 2.3.3, SciPy 1.16.3, scikit-learn 1.6.1, Matplotlib 3.10.0, Pillow 11.3.0, CatBoost 1.2.10, and LightGBM 4.6.0 on CPU without an accelerator. The notebook saves fold assignments, individual and combined predictions, fold metrics, calibration-bin values, model settings, software versions, figures, input hashes, and a SHA-256 output manifest. The run completed in 10,629.1 seconds. The canonical source notebook is supplied as S1 File; core outputs and combined predictions are supplied as S2 File, S1–S3 Data, and a concise successful-run record is supplied as S3 File.
2.10. Use of generative artificial intelligence tools
During study development and manuscript revision, the author used ChatGPT and Codex (OpenAI) and Gemini (Google) to assist with drafting and debugging Python code, organizing and checking the canonical workflow, language editing and restructuring, assembling the revision documents, and packaging and splitting supporting files. These tools were not treated as authors or autonomous analysts. The author specified the research question and methods and inspected and edited the code and documents. The canonical notebook was executed from a fresh environment; its numerical outputs and hash manifest were verified; and the analytical results, figures, manuscript tables, and supporting files were cross-checked against the source data and executed outputs. The author reviewed and approved the analysis, outputs, interpretation, and final files and accepts responsibility for all content. AI assistance affected portions of the Python workflow, code comments, manuscript and response wording, Word-document assembly, and supporting-file packaging. The source data, executed model outputs, and scientific conclusions were not autonomously generated by AI. No confidential or patient data were supplied to the tools because the study used public synthetic data.
3. Results
3.1. Dataset integrity and comparability
The prevalence of Presence was 44.83% (282,454/630,000). No missing or non-finite feature values, duplicate identifiers, cross-file identifier overlap, duplicated full training rows, duplicated feature profiles, or train–test feature-profile overlap were found. Train and test distributions were very similar: the largest absolute standardized mean difference among continuous predictors was 0.003846 (blood pressure), and the largest total variation distance among categorical predictors was 0.003089 (ST-segment slope). These findings support a clean same-generator benchmark split but do not constitute external validation.
3.2. Standalone predictor discrimination
Single-predictor pooled out-of-fold AUROC was highest for thallium category (0.800037), followed by chest pain type (0.767724), maximal heart rate (0.755862), ST depression (0.733585), and fluoroscopic vessel count (0.714529) (Fig 1). Blood pressure was near chance (0.499550), while fasting blood sugar over 120 yielded 0.508857. These estimates use cross-validated probabilities and do not represent multivariable feature importance.
Points show pooled out-of-fold AUROC from separate single-predictor logistic-regression models using the shared five-fold assignments. Continuous predictors were standardized and categorical predictors were one-hot encoded within each training fold. The dashed vertical line denotes chance discrimination (AUROC 0.50). BP, blood pressure; ECG, electrocardiogram; HR, heart rate.
3.3. Comparative model performance
CatBoost had the numerically highest pooled out-of-fold AUROC (0.955443), followed by LightGBM (0.955324), HistGradientBoosting (0.955066), and logistic regression (0.952874) (Table 3). The CatBoost–LightGBM difference was 0.000119 and the CatBoost–logistic-regression difference was 0.002569. Mean fold AUROCs agreed closely with the pooled estimates, and fold standard deviations were 0.000448–0.000496. Average precision ranged from 0.945912 to 0.948828. No paired inferential comparison was performed, so these small numerical differences should not be interpreted as evidence of statistical or clinical superiority.
3.4. Calibration, optimism, and negative controls
Across models, Brier scores ranged from 0.081116 to 0.083518, calibration intercepts from −0.001299 to 0.002153, and slopes from 0.998896 to 1.004388 (Table 3; S1 Fig), indicating close internal agreement between predicted and observed probabilities. Mean train–validation optimism ranged from 0.000017 for logistic regression to 0.002686 for HistGradientBoosting. These results characterize internal out-of-fold calibration only.
The identifier-only AUROC was 0.500121. Five permuted-outcome logistic-regression controls yielded AUROCs from 0.498726 to 0.502126, with a mean of 0.500615. Together with the overlap screens and fold-contained model fitting and early-stopping selection, these near-chance controls provided no evidence of a simple identifier, duplication, or preprocessing leakage mechanism.
3.5. Workflow characteristics and computational cost
Table 4 recapitulates the four workflows. In the successful CPU run, recorded model time was 0.357 minutes for logistic regression, 3.742 minutes for HistGradientBoosting, 10.607 minutes for LightGBM, and 159.190 minutes for CatBoost. LightGBM was within 0.000119 AUROC of CatBoost with approximately one-fifteenth of its recorded runtime. Runtime is environment-specific and was not a clinical or inferential endpoint.
4. Discussion
4.1. Principal findings
All four fixed classifiers achieved high and stable internal discrimination in this large synthetic benchmark. The ordering favored CatBoost numerically, but the entire AUROC range was only 0.002569, and logistic regression approached the boosted models at a fraction of their computational cost. The single-predictor analysis also showed that several supplied variables carried substantial standalone signal. The practical result is therefore not that one complex model was demonstrably superior, but that model choice altered performance only slightly under a shared, tightly controlled validation design.
Calibration intercepts and slopes were close to their ideal values in out-of-fold data, and Brier scores were similar across workflows. Identifier and label-permutation controls remained at chance, while duplicate and cross-split overlap screens were negative. These checks increase confidence that the internal estimates were not driven by an obvious leakage mechanism. They cannot establish calibration or discrimination after transport to a different clinical population, where prevalence, measurement, case mix, and missingness may differ [4,5,8].
4.2. Implications for reproducible benchmarking and future validation
The comparison illustrates why a common validation scaffold matters. Every workflow used the same saved outer folds, all fitted preprocessing components used outer-training data, and the boosting procedures that needed explicit iteration selection were refitted on the complete outer-training data before validation. Full out-of-fold predictions allow the reported discrimination and calibration statistics to be recomputed directly. The small performance differences also make computational cost relevant: LightGBM approached CatBoost’s AUROC with markedly lower CPU time, whereas logistic regression provided a particularly efficient baseline.
The analysis intentionally did not include SHAP, causal attribution, subgroup-fairness analysis, or a post hoc rule-based score. Fig 1 describes single-predictor discrimination only. A clinically useful next study would require temporal and geographic validation in real cohorts, reassessment of average precision and calibration under realistic prevalence, prospectively specified subgroup analyses, and decision-curve evaluation at clinically meaningful thresholds. TRIPOD+AI and PROBAST+AI provide appropriate reporting and appraisal frameworks for that work [9,10].
4.3. Strengths and limitations
The study’s main strengths are a single locked and fully rerun notebook, identical saved outer folds for all classifiers, fold-contained fitted preprocessing, inner early stopping with full outer-training refitting for CatBoost and LightGBM, complete out-of-fold predictions, multiple discrimination and calibration measures, explicit train–validation optimism, leakage audits, permuted-outcome controls, and cryptographic input and output manifests. Several limitations remain. The data are synthetic; their 44.83% outcome prevalence and feature relationships may not reflect any clinical population. Cross-validation estimates internal performance only, and the unlabeled competition test set derives from the same data-generation process; neither provides external or prospective validation. The model and hyperparameter set was fixed before the canonical rerun but was not chosen under a registered protocol and was not exhaustive; no causal, subgroup-fairness, decision-curve, or SHAP analysis was performed; and close internal calibration may not persist under prevalence or case-mix shift. These limitations constrain interpretation to comparative reproducibility within this benchmark.
5. Conclusions
Within this synthetic benchmark, all four classifiers achieved high discrimination and close internal calibration, with only small gains for gradient boosting over L2-regularized logistic regression. The study’s contribution is a fully rerunnable comparison using shared folds, fold-contained iteration selection, saved out-of-fold outputs, and explicit leakage controls. These results provide a reproducible internal benchmark comparison, while performance in real clinical cohorts remains unknown.
Supporting information
S1 Fig. Internal out-of-fold calibration curves.
Observed outcome proportions are plotted against mean predicted probabilities in ten equal-frequency bins for each model; the diagonal is ideal calibration.
https://doi.org/10.1371/journal.pone.0354304.s001
(TIF)
S1 File. Canonical reproducible analysis source notebook.
The notebook defines the complete four-model benchmark, data-quality and leakage checks, calibration analysis, figures, predictions, and manifests from the official Kaggle inputs.
https://doi.org/10.1371/journal.pone.0354304.s002
(IPYNB)
S2 File. Core canonical analysis outputs.
The ZIP archive contains fold assignments, fold and pooled performance metrics, calibration-bin values, data-quality and train–test comparability checks, negative controls, model settings, software versions, figure-source values, run completion record, and SHA-256 manifests.
https://doi.org/10.1371/journal.pone.0354304.s003
(ZIP)
S3 File. Successful canonical-run record.
Plain-text record of the CPU-only Kaggle environment and runtime, authoritative outputs, input and output hashes, and completion marker.
https://doi.org/10.1371/journal.pone.0354304.s004
(TXT)
S1 Data. Combined out-of-fold predictions, part 1.
Gzip-compressed CSV containing the first half of the 630,000 training observations, outcomes, saved outer folds, and predictions from all four models.
https://doi.org/10.1371/journal.pone.0354304.s005
(GZ)
S2 Data. Combined out-of-fold predictions, part 2.
Gzip-compressed CSV containing the second half of the 630,000 training observations, outcomes, saved outer folds, and predictions from all four models.
https://doi.org/10.1371/journal.pone.0354304.s006
(GZ)
S3 Data. Combined test predictions.
Gzip-compressed CSV containing identifiers and predictions from all four models for the 270,000-row unlabeled competition test set. These predictions were not used to estimate manuscript performance.
https://doi.org/10.1371/journal.pone.0354304.s007
(GZ)
References
- 1. Lüscher TF, Wenzl FA, D’Ascenzo F, Friedman PA, Antoniades C. Artificial intelligence in cardiovascular medicine: Clinical applications. Eur Heart J. 2024;45(40):4291–304. pmid:39158472
- 2. Krittanawong C, Virk HUH, Bangalore S, Wang Z, Johnson KW, Pinotti R, et al. Machine learning prediction in cardiovascular diseases: A meta-analysis. Sci Rep. 2020;10(1):16057. pmid:32994452
- 3. Liu T, Krentz A, Lu L, Curcin V. Machine learning based prediction models for cardiovascular disease risk using electronic health records data: Systematic review and meta-analysis. Eur Heart J Digit Health. 2025;6(1):7–22. pmid:39846062
- 4. Gonzales A, Guruswamy G, Smith SR. Synthetic data in health care: A narrative review. PLOS Digit Health. 2023;2(1):e0000082. pmid:36812604
- 5. Kokosi T, Harron K. Synthetic data in medical research. BMJ Med. 2022;1(1):e000167. pmid:36936569
- 6. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, et al. LightGBM: A highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst. 2017;30:3146–54. https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html
- 7. Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A. CatBoost: Unbiased boosting with categorical features. Adv Neural Inf Process Syst. 2018;31:6638–48. https://papers.neurips.cc/paper_files/paper/2018/hash/14491b756b3a51daac41c24863285549-Abstract.html
- 8. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: The Achilles heel of predictive analytics. BMC Med. 2019;17:230.
- 9. Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. Erratum in: 2024;385:q902. 10.1136/bmj.q902 pmid:38636956
- 10. Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, Dhiman P, et al. PROBAST+AI: An updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. pmid:40127903
- 11. Kaggle. Predicting heart disease: Playground Series, Season 6 Episode 2. 2026 [cited 2026 Aug 16]. Available from: https://www.kaggle.com/competitions/playground-series-s6e2
- 12. Huang C, Li S-X, Caraballo C, Masoudi FA, Rumsfeld JS, Spertus JA, et al. Performance metrics for the comparative analysis of clinical risk prediction models employing machine learning. Circ Cardiovasc Qual Outcomes. 2021;14(10):e007526. pmid:34601947