Study participants

This cross-sectional study was conducted at the Medical Outpatient Clinic at one of the tertiary hospitals in Malaysia. A total of 270 patients were recruited over a five-month period from February 2017 to June 2017. The target population comprised individuals diagnosed with type 2 diabetes mellitus, hypertension, and/or dyslipidaemia. Participants were selected using a convenience sampling method. Inclusion criteria were as follows: patients aged 18 years and above at the time of recruitment; those under hospital follow-up for at least one of the aforementioned chronic medical conditions; individuals who had been on prescribed medication therapy for the condition(s) for a minimum of 12 months; and those able to read and/or understand either English or Malay. Exclusion criteria included pregnant individuals and those with psychotic disorders, cognitive impairments, or significant visual and/or hearing impairments.

This study received ethical approval from the Universiti Kebangsaan Malaysia Research Ethics Committee (Ethics Committee Approval Reference: UKM PPI/111/8/JEP-2017–057). Potential respondents at the outpatient clinic were invited to participate in the study. The researcher approached patients or their caregivers and explained the purpose of the study using the patient information leaflet. Patients who agreed to participate were provided with an explanatory statement and asked to sign an informed consent form. During the recruitment phase, approximately 332 individuals were approached, with 62 declining to participate primarily due to time constraints, resulting in the final sample of 270 participants. To ensure data integrity and quality control, a real-time verification process was implemented during data collection. Researchers reviewed each questionnaire immediately upon completion in the presence of the participant to identify and resolve any ambiguous responses or skipped items. This active oversight ensured that all 270 collected responses were complete cases, thereby eliminating the need for post-hoc data imputation due to missing data. Additionally, incomplete survey responses were omitted, as missing answers, particularly to key questions, could compromise data quality and lead to inaccurate conclusions. Individuals unable to understand either English or the native language were excluded from the survey.

Informed consent

Potential respondents at the outpatient clinic were invited to participate in the study. The researcher approached patients or their caregivers and explained the purpose of the study using the patient information leaflet. Patients who agreed to participate were provided with an explanatory statement and asked to sign an informed consent form. Individuals unable to understand either English or the native language, were excluded from the survey.

Study instrument

Data collection for this study was carried out using a set of validated questionnaires available in both English and Malay languages. For instruments that were originally in English and lacked a Malay version, particularly those assessing health belief factors, a standard forward-backward translation method was employed to ensure linguistic and conceptual equivalence. The structured questionnaires were administered via face-to-face interviews conducted by the research team. Researchers actively assisted participants in reading and completing the survey to ensure full comprehension of the items, minimise missing data, and accommodate elderly patients or those with visual impairments. The questionnaire comprised both categorical and continuous variables and was divided into four distinct sections.

Section A gathered information on participants’ demographic and clinical characteristics, including age, gender, race, religion, marital status, educational level, employment status, household income (per month), status of type 2 diabetes mellitus, hypertension, and/or dyslipidaemia, number of medications prescribed over the past 12 months, number of daily doses, duration of treatment (in months), and self-reported health status. Self-reported health status was adapted from the general health perception item of the SF-36 Health Survey. The SF-36 is a rigorously validated instrument demonstrating high test-retest reliability (r > 0.80) and strong construct validity across diverse populations [15]. This section provided essential baseline information for understanding the study population.

Section B focused on information regarding history of CAM use. In this study, CAM was defined as “a practice of medicine that is other than the practice of medicine or dental practices utilised by registered medical or dental practitioners” [16]. A CAM-user was considered to have used one or more of the CAM modalities for the stated medical disease in the past 12 months at the time of the study. This section comprised of a pre-set list of CAM modalities as described in the questionnaire. Vitamin and supplements were added as an extra variable to distinguish its use from herbal products. Lastly, an empty column was provided in this section to allow for patients to state any use of CAM which was not specified in the questionnaire.

Section C assessed health belief factors consisting of 17-item scale based on the Complementary and Alternative Medicine Beliefs Inventory (CAMBI) [14]. It measured three domains: – belief in natural treatment (Question 1–6), belief in participation in treatment (Question 7–11) and belief in holistic health (Question 12–17). The CAMBI survey has been previously validated and demonstrated satisfactory internal consistency. The reported Cronbach’s alpha for the overall scale was 0.81, with individual domain subscales also showing acceptable reliability where belief in natural treatment (α = 0.75), belief in participation in treatment (α = 0.68), and belief in holistic health (α = 0.73). Furthermore, reliability analysis conducted on this current study cohort confirmed the instrument’s robustness, yielding a highly satisfactory overall Cronbach’s alpha of 0.918. The internal consistency for the specific domains within this study’s sample was equally strong: belief in natural treatment (α = 0.895), belief in participation in treatment (α = 0.705), and belief in holistic health (α = 0.836). Each item in this section was scored on a seven-point Likert scale ranging from one (strongly disagree) to seven (strongly agree). After reversing the codes of the negatively worded items (item 9, 11, 14, 17), item scores were summed to provide the total sub-scaled scores of each domain, i.e., belief in natural treatment (ranged 1–42), belief in participation in treatment (ranged 1–35) and belief in holistic health (ranged 1–42). Median cut-offs were obtained for each sub-scale score. A median cut-off points of ≥32 suggests positive belief in natural treatment; a median cut-off point of ≥23 suggests positive belief in participation in treatment and a median cut-off point of ≥29 suggests positive belief in holistic health. Scores equal or less than these cut off points were considered negative belief.

The MALMAS is a rigorously validated instrument specifically designed for the Malaysian population, demonstrating high internal reliability with a reported Cronbach’s alpha of 0.88, alongside strong concurrent validity [13,17]. The first item of the MALMAS consisted of five responses: (1) All the time, (2) Often, (3) Sometimes, (4) Rarely and (5) Never. The remaining seven items consisted of dichotomous response of “Yes” or “No”. The responses in the MALMAS had a total score which ranged from 0 to 8. Patients were considered adherent if they had a total score of six and above (≥ 6), and non-adherent if they had a total score of below six (< 6) [13,17]. Medication adherence, as measured by the MALMAS, served as the primary outcome of this study and was used as the target variable for prediction.

Data pre-processing

The study utilised a complete dataset of 270 responses, requiring no imputation, meaning there were no missing values in either the predictor variables or the outcome variable. Variables were recoded for numerical processing [18], and those exhibiting near-zero variance were removed to prevent instability [19]. Following this, 19 predictor variables (7 continuous and 12 categorical) were retained. Continuous variables were standardised using Min-Max Normalisation to prevent scale-driven bias [20].

To ensure robust evaluation and prevent data leakage, all preprocessing steps were performed independently on training and validation subsets [21]. The data was partitioned into a 70% training set (for model development) and a 30% untouched validation set (for evaluation) using stratified random sampling to maintain proportional representation of adherence groups. The validation set was kept completely separate during model training and was only used to assess final model performance, ensuring an unbiased evaluation of predictive accuracy.

To evaluate the impact of class imbalance, models were trained on both the original unbalanced dataset (17:13 ratio) and a balanced dataset generated via Synthetic Minority Oversampling Technique-Tomek Links (SMOTE-Tomek) [22]. SMOTE generates synthetic samples of the minority class (in this case, non-adherent patients) by interpolating between existing minority class example. Tomek links, on the other hand, help clean the dataset by identifying and removing ambiguous samples that are close to the decision boundary between classes. By combining both techniques, SMOTE-Tomek not only balances the class distribution but also improves the overall quality of the training data, enhancing the model’s ability to generalise and reducing the likelihood of overfitting to noise. Model stability was ensured through stratified 5-fold cross-validation, with hyperparameters optimised via grid search. Specific hyperparameter configurations are detailed in S1 Table.

Feature selection

Feature selection plays a crucial role in reducing data dimensionality and simplifying model complexity, ultimately leading to better model performance and interpretability. By identifying and retaining only the most relevant variables, this process offers several advantages such as, improved prediction accuracy, reduced risk of overfitting, enhanced model transparency, lower storage requirements, and importantly, decreased time and cost associated with administering questionnaires during clinical consultations [23].

In this study, the sequential backward elimination technique was employed to identify the most significant predictors. This method begins with the full set of input variables and iteratively removes those that contribute the least to model performance. The process is repeated until no further variables can be eliminated without negatively affecting model accuracy (or in this study, Area Under the Receiver Operating Characteristic Curve (AUC) values was use as the determinant), resulting in a leaner and more efficient feature set. From the original 19 variables this method helped to eliminate irrelevant or redundant variables while preserving predictive strength [24].

Three machine learning algorithms, Random Forest (RF), Logistic Regression (LR), and Support Vector Machine (SVM) were used to guide the feature selection process. Initially, models were trained using the complete set of variables, and feature importance scores were generated. Variables were then ranked based on these scores, and the least important variable determined by the lowest contribution to the AUC was removed. The model was retrained using the reduced set of variables, and performance was re-evaluated using AUC. This cycle of elimination and retraining continued until no substantial improvement in model performance was observed compared to the previous iteration. The final subset of variables retained through this process was considered the optimal feature set and was used in subsequent stages for model training, evaluation, and interpretation.

Model development

Following data pre-processing and feature selection, predictive models were developed using both the original unbalanced dataset and a balanced dataset generated via SMOTE-Tomek. Models were trained using two feature sets: the full 19-variable set and subsets identified through sequential backward elimination. To develop predictive models for medication non-adherence, this study employed twelve machine learning algorithms comprising linear, non-linear, probabilistic, and tree-based methods. Linear models included Logistic Regression (LR) [25] and Support Vector Machine (SVM) with a linear kernel [26]. Non-linear and distance-based methods included SVM with a radial basis function (RBF) kernel [26], k-Nearest Neighbours (k-NN) [27], and Gaussian Process [28]. Probabilistic approaches consisted of Gaussian Naïve Bayes [29] and Bernoulli Naïve Bayes [30], while tree-based and ensemble methods included Decision Tree [31], Random Forest (RF) [32], AdaBoost [33], Gradient Boosting [34], and Bagging [35]. These algorithms were selected to capture diverse learning paradigms and predictive capabilities for medication non-adherence classification.

To further improve predictive performance and model robustness, stacked ensemble learning was implemented by combining the outputs of all twelve base learners through a meta-learning framework [36]. Stacked ensemble learning enables the integration of heterogeneous models to reduce generalisation error, mitigate overfitting, and improve prediction stability and accuracy [37]. Three stacked ensemble configurations were developed using Generalised Linear Model (GLM) [37], Random Forest (RF) [38], and Gradient Boosting Machine (GBM) [39] as meta-learners. The Ensemble GLM prioritised interpretability, Ensemble RF emphasised robustness and variance reduction, while Ensemble GBM focused on iterative optimisation and predictive accuracy. The systematic combination of these various data balancing techniques, feature subsets, algorithm families, and ensemble strategies ultimately yielded 120 distinct model configurations. Evaluating this extensive array of models served as a sensitivity analysis, ensuring that the final selection was robust, stable, and not overly dependent on a single methodological approach.

Model evaluation, validation, and performance measures

Model evaluation and validation were conducted using descriptive statistics, statistical hypothesis testing, and standard machine learning performance metrics. Frequencies were reported for categorical variables, while means and standard deviations were used for continuous variables. Univariate analysis was performed using chi-square tests for categorical variables and two-sided independent Student’s t-tests for continuous variables, with significance set at p < 0.001. This stringent threshold was specifically selected to control for multiple testing across the 19 predictor variables and to reduce the risk of Type I errors, ensuring that only variables with a highly robust statistical association were prioritised for the subsequent machine learning feature selection process.

Model performance was evaluated using area under the receiver operating characteristic curve (AUC), balanced accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). AUC was selected as the primary metric due to its robustness in handling imbalanced datasets. To statistically compare model performance, a paired corrected resampled t-test was applied to account for dependencies arising from repeated training-validation splits.

Following identification of the best-performing model, SHapley Additive exPlanations (SHAP) [40] were used to interpret variable contributions toward medication non-adherence prediction. Variables were ranked using mean absolute SHAP values and visualised using SHAP feature importance and beeswarm plots to illustrate the direction and magnitude of each variable’s effect on prediction outcomes.

To determine the optimal classification threshold, both the Youden Index [41] and Decision Curve Analysis (DCA) [42] were employed. The Youden Index was used to identify thresholds that maximised sensitivity and specificity, while DCA evaluated the net clinical benefit across varying probability thresholds [43,44]. In addition, model calibration was assessed using Platt scaling, with comparisons between original and calibrated models performed using log loss, Brier score, χ² goodness-of-fit, and associated p-values. All analyses, including preprocessing, statistical testing, model evaluation, calibration, and SHAP analysis, were conducted using Python libraries including scikit-learn, scipy, and numpy.

Together, these methods ensured a balance between predictive accuracy and practical clinical relevance, supporting informed decision-making based on the model’s output. The complete workflow of this process is summarised in the flowchart in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overall methodological flow of the study.

https://doi.org/10.1371/journal.pone.0354682.g001

Feature selection

To identify the most significant variables for predicting medication non-adherence among patients with chronic diseases, sequential backward elimination approach was applied. This method was conducted using three different machine learning algorithms, RF, SVM and LR to ensure robustness in feature selection and the selected variables are as seen on Table 2 below. Through this process, three variables consistently emerged as important across all three methods: race, hypertension status, and belief in natural treatment. These variables represent the common set of predictors most strongly associated with medication non-adherence. Subsequently, predictive models were constructed in two ways: first, using the full set of available variables (19 variables), and second, using only the subset of variables identified by the sequential backward elimination approach (RF-selected = 8 variables, SVM-selected = 6 variables and LR-selected = 9 variables) as seen in Table 2 below. This comparison allowed us to evaluate whether models trained on a reduced, more targeted feature set could achieve comparable or improved predictive performance relative to those trained on the complete feature set.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Variables selected using sequential backward elimination with three different machine learning algorithms; SVM, RF, and LR.

https://doi.org/10.1371/journal.pone.0354682.t002

Model performance

To comprehensively evaluate the models’ performances, a wide range of model configurations were developed and assessed in this study. The models were constructed using different data settings, including both balanced and unbalanced datasets, and were trained on two types of feature sets: (i) the full set of variables and (ii) the selected sets of variables selected through sequential backward elimination using RF, SVM and LR. A total of 120 predictive models were evaluated across various data and feature configurations (full results available in S2 Table). Table 3 presents the best-performing models for each combination of dataset setting and feature selection method.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Best performing models from each combination of dataset setting and feature selection method.

https://doi.org/10.1371/journal.pone.0354682.t003

Overall, the top models demonstrated strong predictive capabilities, with Area Under the Curve (AUC) values ranging from 0.775 to 0.816. Key findings from the performance evaluation include:

• Top Individual Model: Gaussian Naïve Bayes (all variables, unbalanced dataset) achieved the highest performance among individual algorithms with an AUC of 0.812 evaluated on the 30% untouched validation dataset, significantly outperforming other individual configurations (p < 0.001).
• Top Ensemble Model: The Ensemble GLM trained on LR-selected variables with the unbalanced dataset achieved the highest overall AUC of 0.816 evaluated on the 30% untouched validation dataset. While it matched the AUC of the Ensemble GLM trained on SVM-selected variables, it was designated as the overall best-performing model due to a higher balanced accuracy (0.785 vs. 0.781). Differences in performance among the top ensemble models were not statistically significant (p > 0.001).

The overall best-performing model was derived from the original unbalanced dataset utilising targeted variable selection (LR), rather than all available variables or synthetic balancing. This suggests that targeted feature selection effectively reduced noise and optimised model learning, yielding superior performance over data balancing techniques in this cohort.

.

Model interpretation

SHAP analysis was performed on the optimal model (Ensemble GLM trained on LR-selected variables, unbalanced dataset) to interpret the influence of each predictor. Fig 2 presents the SHAP beeswarm plot, which ranks the nine selected variables by their overall importance to the model’s predictions. Positive SHAP values indicate a higher likelihood of predicting medication non-adherence.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. SHAP summary plot of the best performing model.

https://doi.org/10.1371/journal.pone.0354682.g002

The directional impacts of these nine predictors are summarised as follows:

1. Belief in Natural Treatment: The most influential variable where stronger positive beliefs in natural remedies were significantly associated with a higher likelihood of non-adherence.
2. Race: Indigenous patients demonstrated a stronger tendency to adhere to prescribed treatments compared to other racial groups.
3. Treatment Burden: Higher numbers of prescribed daily doses (3rd) and total medications (4th) directly increased the probability of non-adherence, underscoring the negative impact of treatment complexity.
4. Religion: Identified as the fifth most important predictor contributing to the model’s decision-making process.
5. Educational Level: Higher educational attainment demonstrated a modest protective effect, associating with greater medication adherence.
6. Belief in Holistic Health: Similar to natural treatments, stronger positive attitudes toward holistic health practices were linked to increased non-adherence.
7. Duration of Treatment: Patients with longer treatment histories exhibited higher adherence rates compared to those at earlier stages of treatment.
8. Hypertension Status: A diagnosis of hypertension was slightly associated with a greater tendency to adhere compared to non-hypertensive patients.

Risk stratification

To establish the optimal classification threshold for clinical use, the Youden Index was evaluated. The Youden Index is a metric that balances sensitivity and specificity to identify the threshold that maximises the model’s overall discriminative ability, ensuring the best possible trade-off between correctly identifying non-adherent patients and minimising false positives. As illustrated in Fig 3, the balanced accuracy curve (blue line) and the Youden Index curve (green line) both demonstrated clear peaks within the 0.4 to 0.5 probability range. Specifically, a cutoff threshold of 0.4 was identified as optimal, yielding the highest balanced accuracy of 0.788 and a Youden Index of 0.576, alongside a sensitivity of 0.804 and specificity of 0.771. This threshold of 0.4 was selected as the final cutoff point, as it reflects the model’s ability to maintain strong diagnostic accuracy while minimising classification bias toward either adherence or non-adherence.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Graph of different cutoff points versus balanced accuracy of the Youden Index.

https://doi.org/10.1371/journal.pone.0354682.g003

To further evaluate clinical usefulness, Decision Curve Analysis (DCA) was employed. Unlike the Youden Index, DCA assesses the net clinical benefit across a continuous range of threshold probabilities, demonstrating whether the model adds practical value compared to default “treat all” or “treat none” strategies. In Fig 4, the x-axis represents the threshold probability for classifying a patient as non-adherent (triggering intervention), while the y-axis reflects the net benefit, which balances the gains of correctly identifying non-adherence (true positives) against the harms of unnecessary intervention (false positives).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Decision Curve Analysis (DCA) of the original model (best performing model) and calibrated model.

https://doi.org/10.1371/journal.pone.0354682.g004

The analysis revealed that the model calibrated via Platt Scaling (orange line) consistently demonstrated a higher net benefit than the original model (blue line) across most thresholds. The original model achieved its maximum net benefit of 0.98 at a threshold of approximately 0.73, whereas the calibrated model shifted to a threshold of 0.80 with a superior net benefit of 1.46. Both models significantly outperformed the “treat all” strategy (green line), which maintained a low baseline net benefit of 0.43, confirming that indiscriminate intervention is suboptimal. As expected, model performance converged with the “treat all” approach at near-zero thresholds and declined toward zero at extremely high thresholds due to missed cases. Ultimately, the DCA confirms that applying the calibrated model, particularly at the 0.80 threshold, provides the greatest overall clinical utility for guiding targeted patient interventions.

To further evaluate the robustness of the prediction model, a comparison between the original and calibrated models was conducted using calibration curve analysis (Fig 5) and several calibration metrics. As can be seen from Table 4, both models achieved the same AUC (0.816), indicating comparable discriminatory ability. However, the original model demonstrated slightly better probability estimation, with lower Log Loss (0.52 vs. 0.528) and Brier Score (0.17 vs. 0.171). In terms of calibration, the original model also performed better, as reflected by its lower X² statistic (6.407 vs. 11.44) and higher p-value (0.602 vs. 0.178), suggesting closer alignment to the ideal calibration line. These findings indicate that, despite calibration, the original model maintained more reliable predicted probabilities and overall performance. While the differences are small, the original model appears more suitable for clinical application.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Calibration comparison measures between original and calibrated models.

https://doi.org/10.1371/journal.pone.0354682.t004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Calibration curve between original and calibrated model.

https://doi.org/10.1371/journal.pone.0354682.g005

To complement the calibration and DCA results, this study also compared the optimal cutoff values derived from the Youden Index and DCA as seen in Table 5 below. Both cutoffs yielded the same AUC of 0.816 (95% CI: 0.731–0.900), confirming that the model’s overall discriminatory ability was unaffected by the choice of threshold. However, the performance profiles differed. At a cutoff of 0.4, the model achieved a more balanced sensitivity (80.4%) and specificity (77.1%), resulting in higher balanced accuracy and Youden’s Index. In contrast, the higher cutoff of 0.73 led to markedly increased sensitivity (97.8%) but much lower specificity (34.3%), favouring the identification of nearly all non-adherent patients at the expense of more false positives. While the Youden Index pointed to 0.4 as the statistically optimal threshold and the DCA suggested benefit at higher cutoffs, this study aimed to identify the most suitable balance between false negatives and false positives. Therefore, both indices were considered and selected 0.4 as the final cutoff, with further justification provided in the discussion part.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Comparison between Youden index and DCA.

https://doi.org/10.1371/journal.pone.0354682.t005