Research design and population

This study employed a cross-sectional design to analyze the data collected during the PoCOsteo study, a prospective cohort conducted in Austria (Graz Medical University) and Iran (Tehran University of Medical Sciences) [18]. The PoCOsteo study was an extension to phase II of the Bushehr Elderly Health (BEH) cohort [19]. The BEH cohort, conducted jointly by the Tehran University of Medical Sciences (Endocrinology and Metabolism Research Institute) and Bushehr University of Medical Sciences (Persian Gulf Marine Biotechnology Research Centre), aimed to investigate the prevalence of non-communicable diseases and their risk factors in the elderly. The PoCOsteo study, started in 2018, collected comprehensive baseline data on various health metrics and risk factors, with ongoing assessments and follow-ups for 12 months. The data analyzed in this study has been extracted from only one timepoint of the mentioned larger prospective cohort investigation. Therefore, the current study is designed as a cross-sectional investigation. The sample size included n = 2000 participants with a mean age of 62.15 years (± 8.07), and 57.1% females. We accessed the data on May 22, 2024.

The Iranian part of this cohort included community-dwelling men and women aged 50 years and older residing in Bushehr, Iran, who had not planned to leave this city for at least five years after enrollment. Upon enrollment, the subjects (or an accompanying individual, typically a relative) provided the informed consent. The cohort excluded older individuals residing in care facilities or those not physically or mentally capable of participating in the study. The participants were included through a multistage cluster random sampling method from 75 municipal blocks.

Following the granting of informed written consent, all individuals were interviewed and examined by qualified nurses to collect information on demographic status, lifestyle factors, general health, medical history, medication use, and mental and functional health. The enrollment process has previously been presented in more detail [19,20].

Data collection

This study employed a relatively large set of characteristics from the study population, including data on demographics, family history, past medical history, medication, potentially related symptoms, blood pressure measures, physical examination, and laboratory assessment. The authors did not have access to any information that would reveal the identities of individual participants during or following the data collection process. Family history has been focused on diabetes, hypertension, stroke, and myocardial infarction. Medical history included questions regarding cerebrovascular accidents, transient ischemic attacks, other neurologic conditions, heart failure, previous cardiac infarction, renal failure, malignancies, hypo/hyperthyroidism, hepatic disorders, pulmonary disorders, and the history of hospital admission in the preceding year.

This study has used the Rose Angina Questionnaire (RAQ) to indicate the presence of angina. This questionnaire is a standardized tool designed to assess the presence and severity of angina pectoris and myocardial infarction and is widely used in epidemiologic studies [21,22]. It consists of a series of questions that evaluate the frequency, duration, and intensity of chest pain episodes, as well as the circumstances under which they occur, such as physical exertion or emotional stress [23]. This questionnaire defines angina as chest pain that restricts physical activity, located over the sternum or in the left arm and chest, and which resolves after ten minutes of rest [23]. This questionnaire has been translated into Persian and approved as a reliable [24] and valid [25,26] tool in the Iranian population. To train the ML models, we dichotomized the participants using this questionnaire into either the “Angina” group (definite angina) or the “No Angina” group (probable or no angina) [22]. Patients with all the following characteristics were classified as “Angina” (definite angina):

1. I. Chest pain when walking at an ordinary pace on the level or uphill or when hurrying
2. II. Stopped walking or slowed down due to chest pain
3. III. Chest pain is relieved in less than 10 minutes

The participants’ blood pressure (BP), including systolic and diastolic BP (SBP and DBP, respectively), were measured using a standard mercury sphygmomanometer after a 15-minute rest while seated. BP was measured for each patient twice on the right arm with a ten-minute interval. The average of the two readings was recorded as either SBP or DBP. Pulse pressure was calculated as the difference between SBP and DBP (PP = SBP – DBP) and proportional pulse pressure as PP divided by SBP [27].

For biochemical measurements, a qualified nurse collected 25 milliliters of venous blood from each subject after fasting for 8–10 hours. The laboratory work-up included complete blood count (white blood cell count, red blood cell count, hemoglobin, mean corpuscular volume, hematocrit, platelet count), serum levels of fasting blood sugar, hemoglobin A1c, lipid profile (total cholesterol, high-density lipoprotein, low-density lipoprotein, triglyceride), creatinine, urea, parathyroid hormone, and 25-OH vitamin D.

To measure the anthropometric indices, participants put on light clothing and had their shoes removed. Using a digital scale and a fixed stadiometer, height (in centimeters) and weight (in kilograms) were measured by the standard procedure. Hereby, the body mass index was calculated as equal to body weight (kilograms) divided by height (meters) squared. Additionally, right leg and arm circumference and length were measured along with neck and hip circumference (all in centimeters) using a flexible non-stretching tape measure. Body mass composition was also measured using a dual x-ray absorptiometry (DXA, Discovery WI, Hologic, Bedford, Virginia, USA), from which we have utilized whole body fat mass, lean mass, and total mass [28].

Handgrip strength was measured using a standardized digital dynamometer. Participants were seated comfortably with the shoulder adducted and neutrally rotated, the elbow flexed at a 90° angle, and the forearm and wrist in a neutral position. Each participant was instructed to squeeze maximally for three seconds during each trial, with standardized verbal encouragement provided across all measurements. Three trials were conducted for each hand, with a resting period of at least 60 seconds between attempts to avoid muscle fatigue. The highest value from the trials was recorded as the participant’s maximum grip strength. Previous studies have demonstrated grip strength assessment to be a valid and reliable procedure among healthy and various clinical populations [29].

Collecting the data described above, we intentionally chose to forgo the use of automated methods for feature selection, a decision reached after consultations with experts in the field. We assert that while ML algorithms can offer automated selection techniques, they may not capture the full complexity and nuances of the data at hand.

Descriptive statistics

Upon collection, the data were delineated using descriptive statistics stratified by the RAQ result (no angina, probable angina, and definite angina). Then, basic analytical statistics were used to compare the outcomes of RAQ in each variable using the Kruskal-Wallis test for the quantitative variables and Chi-square for the qualitative variables. These steps were carried out in Statistical Package for the Social Sciences (SPSS) v22. P-values less than 0.05 were considered statistically significant.

Preprocessing

In order to feed the data to the ML models, they were first cleaned and prepared as follows. All the steps henceforth were carried out in Python programming language v3.11.8. Fig 1 demonstrates the major steps taken throughout the study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Study flow diagram.

https://doi.org/10.1371/journal.pone.0329023.g001

Given the sensitivity of the data format for the employed ML models, data cleaning was an essential process. To address the missing data in the dataset an imputation method was used. When presenting data to an ML model for training or testing, the missing values for some features may disrupt the model’s performance. Here, two approaches can be adopted. One is to completely exclude the data rows that do not have all features available, which would result in a significant portion of the data being wasted in this way. The other approach is to fill in the missing values using the imputation method, which is an approximate method [30]. Implementing the second approach, we aimed for a method that balances simplicity and effectiveness. More complex methods such as KNN imputation or multiple imputation techniques, while potentially more accurate, also have certain drawbacks, e.g., sensitivity to the choice of distance metrics and the number of neighbors or demanding more complex implementation and computation [31,32]. Ultimately, the function used in the present analysis was SimpleImputer from the scikit-learn library [33], due to its straightforward implementation, computational efficiency, and the adequacy of its performance for our specific dataset. This choice strikes a balance between maintaining data integrity and ensuring the reliability of our analysis, while maintaining the overall distribution of the data and minimizing the introduction of bias that could arise from more complex imputation methods. This technique performs imputation using descriptive statistics, replacing the missing values with average of each column (strategy = ’mean’) for quantitative variables and most frequent values for the qualitative variables (strategy = ’most_frequent’).

Next, due to the different scales of the features, it was necessary to normalize the data to prevent the distortion of model performance and to make the training process shorter and more efficient. The StandardScaler function was applied to the predictor variables, accessible from the preprocessing module in the scikit-learn library [33]. This function uses the following formula for normalization:

Where Z is the normalized value, x is the original value, u is the arithmetic mean of the feature, and s is the standard deviation. The training and test data were normalized separately to prevent information leakage from the test data to the training data.

Finally, all the qualitative variables were encoded as categorical variables using the function “to_categorical” from the utils module of the Keras library.

Machine learning models

Different algorithms have varying strengths and weaknesses, and their performance can significantly depend on the nature of the data and the specific problem being addressed [34]. By exploring a diverse set of algorithms, we can identify which model best captures the underlying patterns in the data, leading to improved accuracy and generalization. Additionally, using multiple models allows for a more robust evaluation of the results, as it helps to mitigate the risk of overfitting a particular algorithm. This approach also facilitates the comparison of model performance metrics. Hereby, we have developed models of logistic regression (LR), artificial neural network (ANN), support vector machine (SVM), k-nearest neighbors (KNN), linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), decision tree (DT), random forest (RF), and adaptive boosting (AdaBoost).

• Logistic Regression: LR applies the logistic function to a linear combination of the input features, transforming the output into a value between 0 and 1, which can be interpreted as a probability. The model estimates the relationship between the independent variables and the dependent binary outcome, allowing for the identification of significant predictors [35]. Logistic regression is favored for its simplicity, interpretability, and efficiency, making it a popular choice in various fields. Hyperparameter tuning for this model was performed through randomized grid search optimizing for penalty, solver, C, max_iter, and l1_ratio.
• Artificial neural network: This model was designed as a Multi-Layer Perceptron (MLP). MLPs are widely used in various applications, including classification, regression, and pattern recognition tasks, making them a foundational model in deep learning [36,37]. Hyperparameter tuning for MLP was performed through randomized search using keras-tuner engine, testing the performance (AUC) of models with different number of layers and units in each layer.
• Support Vector Machine: A potent supervised ML technique, SVM is mainly utilized for classification, while it can also be used for regression issues. The core idea behind SVM is to find the optimal hyperplane that best separates data points of different classes in a high-dimensional space. We have tuned, trained, and tested this model using the scikit-learn library [33]. Hyperparameter tuning for this model was performed through randomized grid search optimizing for C, gamma, kernel, degree, and coef0.
• Linear and Quadratic Discriminant Analysis: By maximizing the ratio of between-class variance to within-class variance, LDA identifies the optimal projection that enhances class separability. This makes LDA particularly effective for problems where the classes are linearly separable [34]. QDA, on the other hand, is a classification technique that extends LDA by allowing each class to have its own covariance matrix. This means that QDA can model more complex decision boundaries, as it does not assume that the classes share the same variance structure. Like LDA, QDA also relies on the assumption that the data follows a Gaussian distribution, but it uses quadratic functions to define the decision boundaries. This flexibility allows QDA to perform better than LDA in situations where the classes are not linearly separable, making it suitable for more complex datasets [34]. Tuned hyperparameters were solver and shrinkage for LDA, and reg_param and tol for QDA.
• Decision Tree: A decision tree aims to create a model that predicts the target variable by learning simple decision rules inferred from the data. However, they can be prone to overfitting, especially when the tree is deep and complex [38]. To overcome this challenge, we have used two ensemble models that utilize decision trees: random forest and AdaBoost. Randomized grid search was performed to tune decision tree hyperparameters criterion, splitter, max_depth, min_samples_split, min_samples_leaf, max_features, and class_weight.
• Random forest: It is an ensemble learning method that builds upon the concept of decision trees to improve predictive accuracy and control overfitting. It constructs a multitude of decision trees during training and outputs the mode of their predictions (for classification) or the average (for regression). Each tree in a random forest is trained on a random subset of the data and a random subset of features, which introduces diversity among the trees and helps to reduce variance. This ensemble approach enhances the model’s robustness and generalization capabilities, making random forests highly effective for a broad variety of applications [39]. Hyperparameter tuning for this model was performed through randomized grid search optimizing for n_estimators, max_features, max_depth, min_samples_split, min_samples_leaf, and bootstrap.
• Adaptive Boosting: AdaBoost is another ensemble learning technique that combines multiple weak classifiers to create a strong classifier. When used with decision trees, AdaBoost typically employs shallow trees, often referred to as “stumps,” as the base learners. AdaBoost is particularly effective when the data is noisy, or the model needs to adapt to complex patterns [40]. AdaBoost was tuned for hyperparameters n_estimators, learning_rate, estimator__max_depth, estimator__min_samples_split, and estimator__min_samples_leaf.
• K-Nearest Neighbors: The core idea behind KNN is to make predictions based on the ‘k’ closest data points in the feature space to a given input instance. One of the key advantages of KNN is its non-parametric nature, meaning it makes no assumptions about the underlying data distribution [41]. KNN was tuned for hyperparameters n_neighbors, weights, and metric.

All models were developed using the scikit-learn library v 1.5.0 [33] except ANN, which was developed by the Keras library v 3.3.3 [36]. Similarly, the hyperparameters were tuned for all these models via the function RandomizedSearchCV from the scikit-learn library and the keras-tuner library v1.4.7 for ANN, which is integrated with both Keras and scikit-learn libraries [42]. While scikit-learn is chiefly dedicated to traditional ML algorithms, Keras provides several advantages over it regarding the development of ANNs, of which flexibility, customization, scalability, support for production deployment, and integration with deep learning frameworks are the most prominent [36]. Table 1 Presents the optimized settings for the hyperparameters used in developing each model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The hyperparameters for each model based on the random hyperparameter search.

https://doi.org/10.1371/journal.pone.0329023.t001

Model development and evaluation

All models have been trained and evaluated through the commonly used method of 10-fold cross-validation [43]. It is a robust statistical method used to evaluate the performance of ML models and ensure their generalizability to unseen data. This technique randomly divided the dataset into ten equal-sized subsets, or “folds.” The model is then trained and validated on nine of these folds, while the remaining fold is used as the test set to assess the model’s performance. Therefore, the data were divided to train (64%), validation (16%), and test (20%) sets. Early stopping method has been used while monitoring validation set loss to prevent overfitting. This procedure is carried out ten times, with each fold being used as the test set one time. The final performance metric is obtained by averaging the results from all ten iterations. This approach helps to mitigate the risk of overfitting, as it provides a more comprehensive evaluation of the model’s ability to generalize across different subsets of the data [43].

To provide a basis for comparing the models’ performance, the following metrics were calculated for each model in each fold, and the average values were reported: area under the curve (AUC), Youden index, accuracy, sensitivity, and specificity. AUC is derived from the receiver operating characteristic (ROC) curve and quantifies the model’s ability to discriminate between positive and negative classes across various threshold settings. The Youden index, calculated as the sum of sensitivity and specificity minus one, provides a single measure that captures the model’s overall effectiveness in distinguishing between classes, with higher values indicating better performance [44]. We have measured this index at the J point, the optimal threshold for classification. We have also measured the models’ accuracy (the proportion of true positive and true negative predictions among the total predictions), sensitivity (the model’s ability to correctly identify positive instances), and specificity (the model’s ability to correctly identify negative instances).

To determine each feature’s importance as a measure of its contribution to the models’ performance, we conducted the permutation feature importance technique. Through this technique, the values of a specific feature are randomly shuffled while measuring the change in a model’s performance metric (AUC in our study), thereby assessing the contribution of that feature to the model’s predictive power. We then ranked features based on their importance for each model. Using the results from top three models, the numeric values of the rank of each feature were summed across the three models, and eventually, the features were sorted from the smallest summed indices (most important feature) to the largest summed indices (least important feature), where the results were tabulated and presented accordingly. This technique has been adopted previously [45,46].

Ethics statement

This study is carried out in accordance with the Declaration of Helsinki and the relevant guidelines. Due to the retrospective nature of the study (with the approval ID of IR.BPUMS.REC.1403.024), the Research Ethics Committees of Bushehr University of Medical Sciences waived the need for informed consent.

Models’ performance

The performance metrics of the ML algorithms were evaluated through a 10-fold cross-validation process (Table 3). Among the algorithms assessed, LDA achieved the highest mean AUC of 77.2% (95% CI: 72.7% − 81.7%), indicating robust performance, followed by RF (mean AUC of 77.0%, 95% CI: 74.3% − 79.7%) and LR (mean AUC of 76.4%, 95% CI: 71.3% − 81.5%). All models presented acceptable discrimination, except for KNN, DT, and ANN, where poor discrimination was achieved (below 70%). Regarding accuracy, LR outperformed other models with a mean accuracy of 73.9% (± 8.5%). Sensitivity was notably highest in the SVM model at 79.2% (± 18.0%), while QDA exhibited the best specificity at 77.3% (± 12.0%). Conversely, the DT algorithm demonstrated the lowest performance across all metrics, particularly in AUC (64.4% ± 9.5%) and Youden’s index (0.324 ± 0.133). These results highlight the varying effectiveness of different algorithms in predictive modeling. Based on the objectives of this study, we have chosen the top three best models (highest AUCs) to analyze further and interpret: LDA, RF, and LR (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance metrices of the trained models across cross-validation folds (n = 10).

https://doi.org/10.1371/journal.pone.0329023.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Receiver operating characteristic curve for the models with the highest mean area under curve (AUC) across ten folds of cross validation.

https://doi.org/10.1371/journal.pone.0329023.g002

Feature importance

Table 4 highlights the ranking and description of each feature, emphasizing their significance across the models. Notably, “Uneven heartbeat” consistently ranked first in all three models, indicating its critical role in predicting outcomes related to the studied condition. Other prominent features included “Normal activity dyspnea” and “Congestive heart failure,” underscoring their relevance in the predictive framework. “Paresthesia” showed variability in importance, ranking second in Random Forest, third in Logistic Regression, and ninth in Linear Discriminant Analysis, suggesting its varying impact depending on the model used. Additionally, features such as “Hospital admissions” and “Any dyspnea” maintained high rankings, further supporting their importance in clinical assessments. The inclusion of laboratory metrics like “Red blood cell count” and “Creatinine” also reflected the models’ reliance on both clinical symptoms and laboratory values. Overall, the consistent ranking of these features across all three models highlighted their potential as key indicators in the predictive analysis, providing useful insights for clinical decision-making and further research.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. The intersection of the most contributing features ranked among the top 20 important features present in all the three best performing models.

https://doi.org/10.1371/journal.pone.0329023.t004

Limitations

The most significant limitation of our study, given its epidemiological design, was the lack of access to the gold standard for diagnosing coronary heart disease, which is coronary angiography. This limitation might affect the definitive classification of participants concerning their CHD status. However, it is important to note that we have substantial evidence supporting the validity of the Rose Angina Questionnaire as a reliable tool for predicting CHD in similar settings [92–94]. The RAQ has been widely utilized in epidemiological studies, making it a suitable alternative for our analysis. Consequently, we adopted the RAQ as the primary outcome measure for our ML-based models. While this approach did not provide the same level of diagnostic certainty as coronary angiography, it allowed us to leverage existing data effectively and draw meaningful conclusions regarding the predictive factors associated with angina pectoris in our cohort. It is also important to highlight the retrospective nature of the study as another limitation, which might introduce biases inherent in the data collection process. These biases might affect the validity of the findings and their applicability to current clinical practices. Furthermore, the reliance on historical data restricted the generalizability of the results. As such, caution should be exercised when interpreting the outcomes, and further prospective studies are warranted to validate these findings and enhance their relevance in contemporary settings. Additionally, the lack of external validation raised concerns about the robustness and applicability of the results to broader populations or different settings. Another limitation of the employed methodology concerned the potential risk of overfitting, as while ensemble models were employed with cross-validation techniques to enhance predictive performance, there remained a potential risk of overfitting. This risk could compromise the models’ ability to generalize to unseen data, thereby affecting the reliability of the conclusions drawn from the analysis.