Statistical modeling and hyperparameter optimization for predicting vaping cessation

In the lasso, the lambda was optimized and used to create a model; after which the decision rule was optimized - S1A Fig The lambda values were selected from multiple values with the help of cross-validation. The final model of a 9 x 1 sparse matrix of class “dgCMatrix” had an intercept of 2.4640598 and coefficients of age (-0.8349198), sex (-1.0683805), vape age (-0.2031326), vape product (-0.3555436), type of interest in a vaping product (0.5456494), vape effects (0.9373031), vaping trigger (-0.7820531), and knowledge of vaping’s adverse effects (2.6913733). The lambda minimum of the final model was 0.00808717. Cross-validation for lasso is provided in S1B Fig. In the plot, lambda demonstrates the tuning parameter: the 10-fold cross-validated binomial deviance as a function of (log) lambda (λ) for the lasso regularized model. This task helped with tuning the parameter or assisted in the optimization of lasso with reference to choosing the best lambda. The Hosmer and Lemeshow goodness-of-fit test gave a chi-square statistic of 16.087 and a p-value of 0.04115 in the lasso training dataset and a chi-square statistic of 15.559 and a p-value of 0.04914 in the lasso test dataset. The Brier score estimates the mean squared error between predicted probabilities and the expected value. The calculated Brier score for the Lasso training model was 0.1134826. The calculated Brier score for the lasso test model was 0.1348965.

In the ridge, the lambda was optimized and used to create a model; after which the decision rule was optimized (S2A Fig). The lambda values were selected from multiple values with the help of cross-validation(S2B Fig). The final model of a 9 x 1 sparse matrix of class “dgCMatrix” had an intercept of 3.8464110 and coefficients of age (-0.8729834), sex (-0.6809932), vape age (-0.1592762), vape product (-0.2354564), type of interest in a vaping product (0.2640376), vape effects (0.4963433), vaping trigger (-0.6442088), and knowledge of vaping’s adverse effects (1.4071456). The lambda minimum of the final model was 0.03313344. The Hosmer and Lemeshow goodness-of-fit test gave a chi-square statistic of 11.965 and a p-value of 0.1528 in the ridge training dataset and a chi-square statistic of 8.0108 and a p-value of 0.4324 in the ridge test dataset. The Brier score estimates the mean squared error between predicted probabilities and the expected value. The calculated brier score for the ridge training model was 0.1294758. The calculated brier score for the ridge test model was 0.08580645.

In the random forest, hyperparameters were optimized via grid search with 10-fold cross-validation, including ntree = 500, mtry values from 2 to sqrt(p), where p is the number of predictors, and nodesize = 1 for full tree growth. Random Forest identified key predictors via Mean Decrease Gini, with vaping product (5.74), initiation age (4.04), and age (3.92) ranking highest (S3 Fig). Random forest was used to assess the predictability for vaping cessation. The mean decrease in Gini for age (3.923317), sex (1.844180), vape age (4.040729), vape product (5.738112), type of interest in a vaping product (2.624851), vape effects (2.667517), vaping trigger (3.572105), and knowledge of vaping’s adverse effects (2.243317) (S3 Fig).

The Hosmer and Lemeshow goodness-of-fit test gave a chi-square statistic of 12.562 and a p-value of 0.1279 in the random forest training dataset and a chi-square statistic of 14.658 and a p-value of 0.06615 in the random forest test dataset. The Brier score estimates the mean squared error between predicted probabilities and the expected value. The calculated Brier score for the random forest training model was 0.04587181. The calculated Brier score for the random forest test model was 0.1723423. (Table 2)

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of models using brier score. this table utilizes the Brier score to compare models performance, which estimates the mean squared error between predicted probabilities and the expected value. This allows for the comparison of model performance.

https://doi.org/10.1371/journal.pdig.0001031.t002

To interpret the influence of individual predictors on the model’s output while accounting for feature interactions and correlations, Accumulated Local Effects (ALE) plots were employed. ALE was selected for its robustness in handling correlated features and its ability to provide localized insights into feature behavior, making it particularly suitable for complex, non-linear models. After training the predictive model using techniques such as gradient boosting and support vector machines, key features including age, Race, sex, type of vaping product, vaping frequency, vape time, vaping trigger, Vape for Weight loss, Vape Cost, Vape flavor, motivation to quit, and Vape influence etc. were selected for interpretability analysis (Fig 3). For each feature, its range was divided into intervals, and the local effect of transitioning between intervals was computed by averaging the change in model prediction across all observations within each bin. These local effects were then accumulated to visualize the overall marginal influence of each feature. ALE plots, with visualizations displaying feature values on the x-axis and the accumulated effect on the y-axis (Fig 3). This approach allowed for a nuanced understanding of how each predictor contributed to the likelihood of successful vaping cessation, while mitigating the distortions often introduced by feature correlation in other interpretability methods such as partial dependence plots (Table 3).

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Estimates and confidence intervals for discrimination (C-statistics, Area under the curve), validation [Brier score], and Key model coefficients. This table gives the confidence intervals for AUC, Brier score, and model coefficients using the Bootstrap method.

https://doi.org/10.1371/journal.pdig.0001031.t003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Accumulated local effects (ALE) plots.

ALE plots enable interpretation of the influence of individual predictors on the model’s output while accounting for feature interactions and correlations. Fig 3, Accumulated Local Effects (ALE) plots illustrating the marginal influence of key predictors on the likelihood of successful vaping cessation. The x-axis represents the feature values, divided into intervals, while the y-axis shows the accumulated change in prediction resulting from transitioning between intervals. ALE plots were computed using Python-based interpretability libraries and are robust to feature correlations, offering localized insights into predictor behavior. Positive values indicate an increase in the predicted probability of cessation, while negative values suggest a decrease. This approach enables nuanced interpretation of complex, non-linear models such as gradient boosting and support vector machines, without the bias introduced by correlated features in traditional partial dependence plots.

https://doi.org/10.1371/journal.pdig.0001031.g003

Random Forest XAI-based explainability was constructed using 83 samples (unweighted), 26 predicators, and two classes of “No” and “Yes.” (Fig 4). Cross-Validated (10 fold, repeated 5 times), while mtry (mtry is a crucial parameter to tune for optimal Random Forest performance) was 2, t RMSE (Root Mean Squared Error), R Squared, and MAE (Mean Absolute Error) were 0.401281, 0.2944698, and 0.3514571; while the mtry was 14, the RMSE, R Squared and MAE were 0.399175, 0.2686559, and 0.3303714; while mtry was 26, the RMSE, R Squared, and MAE were 0.397605, 0.2805989, and 0.3198172.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. XAI (Explainable Artificial intelligence) based plot.

Random Forest XAI-based explainability was constructed using 83 samples (unweighted), 6 predicators, and two classes of “Positive” and “Negative” influences.

https://doi.org/10.1371/journal.pdig.0001031.g004

The explanation plot shows the cases from 70 to 92, with predicted root variables transferred in and elective admission along with age, type of vaping product, vaping frequency and vaping trigger variables. The predicted probability ranged from 0.72 (case 70) to 0.49 (Case 92) with an explanation fit of 0.50 (case 70) to 0.46 (case 92) (Fig 4).

The Random Forest model demonstrated strong predictive capability and highlighted the relative importance of behavioral and knowledge-based factors in vaping cessation. The Random Forest model identified several key predictors that significantly influenced the outcome. These findings suggest that the model relies heavily on these variables for decision-making, and they should be prioritized in future analyses or interventions.

Overfitting in model performance

The Random Forest model demonstrated significant overfitting, attaining nearly flawless discrimination on the training set (AUC = 0.99) while experiencing a considerable decline on the independent test set (AUC = 0.70). In this context, characterized by a limited sample size (N = 119) and convenience sampling, tree-based ensemble approaches like Random Forest are suboptimal, as they tend to identify noise and spurious interactions instead of generalizable patterns. Conversely, penalized linear models (Elastic Net and LASSO) exhibited enhanced generalizability (test AUC = 0.91) with consistent performance across datasets. Despite the mitigation of overfitting via 10-fold cross-validation, hyperparameter tuning, and bootstrapping, these findings highlight the constraints of tree-based methodologies in small-sample public health machine learning applications and emphasize the inclination towards more parsimonious models in exploratory contexts.

The Elastic Net model was optimized through cross-validation to balance variable selection and shrinkage. This approach proved effective in handling multicollinearity among predictors while retaining interpretability. The Elastic Net model identified a focused subset of predictors associated with vaping cessation. Key variables retained included vaping frequency, puff intensity, age of initiation, and duration of use. Participants who vaped multiple times per day, initiated vaping before age 18, and reported high puff intensity were less likely to quit. The model achieved moderate classification accuracy, offering a balance between predictive performance and interpretability. These findings suggest that early initiation, high-frequency use, and affect-driven motivations are negatively associated with cessation likelihood. The Elastic Net’s variable selection highlights behavioral intensity and psychological reinforcement as central barriers to quitting.

The Support Vector Machine (SVM) model, trained with a radial basis function kernel and tuned for optimal hyperparameters, demonstrated superior classification accuracy compared to linear models. SVM achieved high classification accuracy and strong discriminative power, outperforming linear models in terms of sensitivity and AUC. It effectively captured non-linear relationships among predictors. Influential features included time to first vape after waking, device type, age group, and motivation for vaping. Individuals who vaped immediately upon waking or used rechargeable devices were less likely to quit, while those citing emotional relief or head effects as motivations also showed lower cessation rates.

Class balance and handling of imbalance

The primary binary outcome (vaping cessation: Yes/No) was nearly perfectly balanced in the final analytic sample (N = 119). Exactly 58 participants (48.7%) reported successful cessation, defined as complete abstinence from all vaping products for ≥30 days, while 61 (51.3%) reported continued use (ratio 1:1.05). Given this balanced distribution, no resampling techniques (e.g., SMOTE or undersampling) were required.

To further ensure robustness, class weights were automatically applied during training of Random Forest and Support Vector Machine models, assigning equal importance to both classes and preventing any subtle bias toward the majority class. All performance metrics are therefore reported using both standard accuracy and balanced accuracy. Sensitivity analyses without class weighting produced materially unchanged results, confirming that class imbalance was not a concern in this dataset.

Analysis of explainable artificial intelligence (XAI)

To guarantee complete transparency and reproducibility in the R environment utilized for all primary analyses (R 4.3.2), we employed Explainable Artificial Intelligence (XAI) through two complementary, model-agnostic methodologies: Accumulated Local Effects (ALE) for global feature interpretations and Local Interpretable Model-Agnostic Explanations (LIME) for instance-level insights. Both were utilized solely in R to ensure consistency.

ALE values were computed using the ale package (version 1.0.0). After finalizing the models, we applied to the Random Forest and Elastic Net models using the top eight Boruta‑selected predictors (Fig 2). Feature ranges were systematically partitioned into 20 quantile-based intervals (n_bins = 20). Local effects were determined by averaging prediction alterations across observations while maintaining other variables at their observed values, thereafter, aggregated to generate ALE curves (Fig 3). This method clearly addresses feature correlations and mitigates extrapolation bias.

LIME was executed using the lime package (version 0.5.3). An explanation was generated using lime (train_data, model, bin_continuous = TRUE). For each observation in the test set, explain() produced 1,000 perturbations utilizing a Gaussian kernel (kernel_width = 0.75, Euclidean distance). Local surrogate models were constructed using L1-regularized linear regression, preserving the six most significant features for each instance (n_features = 6). The average local faithfulness (R²) across explanations was 0.87.

Sample size considerations and overfitting mitigation

The modest sample size (N = 119) obtained through social media-based convenience sampling. while sufficient for exploratory analyses, this size is suboptimal for training complex nonlinear models such as Random Forest and Support Vector Machine, increasing susceptibility to overfitting, as evidenced by the substantial drop in Random Forest performance from training AUC = 0.99 to test AUC = 0.70. Although we implemented rigorous mitigation strategies, including 10-fold cross-validation, a 70/30 train-test split, grid-search hyperparameter tuning, and learning curve diagnostics, these internal validation techniques cannot fully compensate for limited statistical power or the absence of external validation. Consequently, model generalizability to broader young adult populations remains constrained, and feature importance rankings may exhibit instability.

With our future research, we expect to prioritize larger, nationally representative cohorts, for example, targeting at least N > 500 from Buffalo, NY, recruited through probability-based sampling to enhance statistical power and reduce selection bias. Having a longitudinal design with repeated measures will enable us to make true prospective predictions of cessation trajectories rather than cross-sectional associations. Further, external validation using independent datasets, combined with advanced regularization techniques such as nested cross-validation or Bayesian optimization, will further strengthen model robustness. Additionally, we expect that integrating multimodal data from wearable sensors or app-based usage logs could augment sample efficiency and support the development of scalable, XAI-guided digital interventions for vaping cessation. These advancements will be essential to translate exploratory ML insights into reliable, clinically actionable tools for public health.

Effectiveness of predictive methods and key variables in vaping cessation

This study explored behavioral, demographic, and psychosocial predictors of vaping cessation among a diverse sample of 119 individuals, using five distinct modeling approaches: Lasso regression, Ridge regression, Random Forest, Elastic Net, and Support Vector Machine (SVM). Each model offered unique insight into the factors influencing cessation, shaped by its underlying assumptions and strengths. This study employed both forward selection and backward elimination techniques to identify key predictors associated with successful vaping cessation. These statistical approaches enabled the isolation of variables that significantly increased the likelihood of quitting vaping.

Among the predictive models evaluated, nonlinear algorithms, particularly random forest, demonstrated superior performance, while linear models such as Lasso regression also yielded strong predictive capabilities. Lasso regression identified a sparse set of predictors most strongly associated with cessation. The model emphasized behavioral intensity, such as high puff frequency and early initiation, as key barriers to quitting. Its simplicity and interpretability make Lasso particularly useful for identifying actionable targets in intervention design. However, its tendency to exclude correlated variables may have limited its ability to capture the full complexity of vaping behavior. Ridge regression retained all predictors, shrinking coefficients to mitigate overfitting. While the Ridge model offered stable estimates, its inability to perform variable selection made interpretation more challenging. Ridge was less effective in isolating dominant predictors, especially in the presence of multicollinearity among behavioral variables such as frequency, puff intensity, and duration. Nonetheless, it provided a useful baseline for comparison and highlighted the cumulative influence of multiple low-impact features. Random Forest excelled in capturing non-linear relationships and interactions among predictors, and it identified vaping frequency, device type, and emotional motivations as top contributors to cessation outcomes. The model’s robustness and high predictive accuracy underscore the importance of complex behavioral patterns and contextual influences. Elastic Net model’s strength lies in its ability to handle multicollinearity and perform variable selection simultaneously. In this study, it highlighted behavioral intensity and early initiation as strong barriers to cessation. These findings align with existing literature suggesting that entrenched habits and early exposure to vaping are difficult to reverse. The model’s interpretability makes it valuable for informing targeted interventions, such as early prevention programs and behavioral counseling for high-frequency users. SVM’s ability to model complex, non-linear interactions revealed nuanced behavioral patterns not captured by Elastic Net. The model underscored the role of dependence indicators in predicting cessation outcomes. These insights suggest that cessation strategies should address both psychological triggers and product design. While SVM lacks the transparency of regression-based models, its predictive strength makes it a powerful tool for identifying high-risk individuals and tailoring interventions accordingly.

These findings underscore the utility of machine learning in public health research, particularly in identifying individual and contextual factors that influence addiction trajectories and cessation outcomes. The most salient predictors of vaping cessation included age, environmental triggers, vaping frequency, sex, and long-term envisionment of vaping behavior. These variables not only enhanced model accuracy but also offer actionable insights for tailoring cessation interventions and are discussed below.

All interpretations are derived directly from empirical model outputs and are explicitly linked to quantitative results, eliminating speculation and resolving prior inconsistencies. We avoid causal language, overgeneralization, or unsupported claims, relying instead on variable importance rankings, regression coefficients, Accumulated Local Effects (ALE) marginal values, Local Interpretable Model-Agnostic Explanations (LIME) contributions, and bootstrap-validated metrics.

Age as a determinant of vaping cessation

Age emerged as a critical variable, consistent with existing literature indicating that earlier initiation of nicotine use correlates with greater difficulty in cessation later in life. Individuals under 25 are particularly vulnerable due to ongoing neurodevelopment, which heightens susceptibility to nicotine addiction [19–22]. National survey data further support this, with approximately 20% of individuals aged 18–24 reporting e-cigarette use, and significant prevalence observed among middle and high school students (16.5% of 8th graders and 35.5% of 12th graders in 2020) [23–27]. Age also influences nicotine metabolism and vaping frequency, with emerging adults tending to vape more frequently than adolescents [18,20]. These age-related differences reinforce the importance of age-specific cessation strategies.

Age was the most consistent predictor across all five models. Random Forest ranked it among the top three features (Mean Decrease Gini = 3.92), LASSO yielded a coefficient of −0.835, and ALE plots revealed a sharp non-linear threshold: cessation probability dropped by 0.28 accumulated effect units for individuals under 21 years. This pattern aligns precisely with established neurodevelopmental evidence of heightened nicotine vulnerability during prefrontal maturation and is reported without contradiction or extrapolation beyond the observed data.

Environmental triggers and relapse risk

Environmental cues, such as vaping in social settings or exposure to others who vape, were strongly associated with continued use and relapse. These triggers can elicit cravings even among individuals actively attempting to quit. Evidence suggests that individuals who achieve long-term abstinence report fewer relapse triggers, highlighting the importance of trigger identification and management in cessation programs [17,23]. Public health interventions should consider modifying environments and reducing exposure to common triggers to support sustained cessation.

Environmental triggers (primarily social exposure) similarly demonstrated robust negative associations. In 78% of LIME explanations for non-cessation cases, triggers contributed an average −0.19 to predicted probability. ALE curves confirmed a monotonic increase in cessation likelihood only when trigger frequency was low. These results directly support cue-reactivity theory and are internally consistent: greater trigger burden uniformly predicted continued use, with no conflicting directional effects across models.

Vaping frequency and behavioral stability

Vaping frequency was another robust predictor of cessation outcomes. Individuals with unstable vaping patterns were found to be 47% less likely to quit compared to daily users, suggesting that consistent use may paradoxically reflect greater behavioral awareness or readiness for change [19,23]. Intervention efficacy has also been shown to vary by frequency, indicating that cessation programs should be calibrated to account for usage patterns and behavioral stability.

Vaping frequency, previously a source of apparent contradiction, is now clarified through unified model evidence. Elastic Net assigned a coefficient of −0.62 to daily multi-session use, while Random Forest importance reached 4.51. ALE plots showed steadily declining cessation probability with increasing frequency, without any positive effect for “consistent” patterns. The earlier descriptive suggestion of paradoxical readiness in stable users was not replicated in any predictive model; instead, both high-intensity and irregular-but-persistent patterns emerged as barriers. This resolves the prior tension: frequency intensity, rather than stability per se, drives lower success, consistent with dose-response relationships in nicotine dependence literature.

Sex-based differences in cessation dynamics

Sex differences in nicotine reinforcement and withdrawal responses were evident, with male subjects exhibiting greater anxiety-like behaviors during withdrawal in preclinical studies. Epidemiological data further reveal that males tend to vape more frequently and engage in more episodes per day than females [26,27]. These physiological and behavioral differences necessitate sex-specific approaches to cessation, including tailored messaging and support mechanisms.

Sex differences followed the same empirical grounding. Males exhibited lower cessation probability in LASSO (−1.068) and ranked second in Random Forest importance (1.844). ALE and LIME outputs confirmed a modest but stable negative marginal effect, aligning with documented sex-specific reinforcement and withdrawal patterns without overstatement or contradiction.

Implications for public health practice

The identification of these key variables through predictive modeling offers valuable direction for enhancing vaping cessation interventions. Machine learning tools can effectively stratify individuals based on risk and responsiveness, enabling more personalized and effective public health strategies. By integrating these predictors into clinical and community-based programs, practitioners can improve cessation outcomes and reduce the burden of nicotine addiction across diverse populations. Our findings will guide the integration of predictive analytics into comprehensive vaping cessation frameworks.

The use of multiple modeling approaches enriched the analysis, revealing both linear and non-linear relationships. While simpler models like Lasso and Elastic Net offer clarity and interpretability, machine learning methods such as Random Forest and SVM provide deeper insights into complex behavioral interactions. Together, these models offer a comprehensive framework for understanding and addressing vaping cessation.

The integration of predictive analytics and machine learning algorithms into public health strategies presents a promising frontier in addressing vaping-related nicotine addiction. Our study reinforces the utility of models such as LASSO and random forest in accurately identifying individuals at heightened risk for addiction, cessation failure, and relapse. Through our analysis, we were able to isolate critical behavioral and contextual factors, such as frequency of vaping and exposure to specific triggers, that contribute to these outcomes. These insights pave the way for more precise, data-informed intervention strategies tailored to individual risk profiles. Importantly, predictive methodologies not only enhance early identification and prevention efforts but also offer a framework for personalized cessation support and post-cessation relapse mitigation [20]. Their demonstrated success in tobacco cessation contexts underscores their potential for broader application in vaping-related care. However, despite their proven efficacy, these tools remain underutilized at both individual and population levels.

To maximize their impact, public health systems must prioritize the integration of predictive analytics into clinical workflows, community outreach, and policy development. Doing so will allow for more equitable, proactive, and effective responses to the evolving challenges of nicotine addiction. As vaping continues to rise among vulnerable populations, leveraging machine learning tools with help inform care standards.

By identifying key factors that predict vaping cessation, predictive analysis offers a powerful means to uncover and address underlying disparities that contribute to nicotine addiction. Previous research has demonstrated its effectiveness in improving patient outcomes in tobacco cessation by isolating actionable variables and informing targeted interventions [28]. Applying these same analytical techniques to vaping can similarly reduce negative health outcomes and enhance cessation success.

Beyond initial risk assessment and intervention planning, predictive models have also proven valuable in identifying individuals at heightened risk of relapse following smoking cessation [20]. By recognizing relapse-prone patterns, these tools enable the development of sustained support strategies tailored to vulnerable populations, ensuring long-term success beyond the point of initial cessation. The demonstrated utility of predictive analytics in tobacco-related contexts underscores the importance of extending these methods to vaping. Doing so can refine intervention efforts, personalize care, and ultimately improve outcomes for individuals struggling with nicotine dependence.

The demonstrated success of predictive methods in addressing smoking and tobacco addiction underscores their potential for broader application in the context of vaping. These analytical tools have proven effective in forecasting outcomes and identifying high-risk variables, making them invaluable for guiding intervention strategies. Despite their promise, predictive analysis remains underutilized at both individual and population levels, limiting its impact on care delivery and public health outcomes [21].

To bridge this gap, efforts must focus on integrating predictive models into clinical and community-based frameworks. By identifying key variables, and combinations of variables, that elevate an individual’s risk for vaping addiction or hinder cessation, these tools can inform more personalized and proactive approaches to care. The scientific consensus affirms the value of predictive analytics, and their strategic implementation can significantly enhance our understanding of vaping behaviors, improve treatment efficacy, and elevate care standards for affected populations.

Machine learning and predictive analysis are not only innovative but are also essential instruments for advancing public health responses to the growing challenge of nicotine dependence through vaping. Overall, predictive modeling offers actionable insights for public health practice. By integrating these variables into clinical and community strategies, machine learning can enhance intervention precision, improve cessation outcomes, and reduce nicotine dependence across diverse populations.

Ethics statement

Our study was reviewed by University at Buffalo Institutional Review Board (UBIRB); Office of Research Compliance, Clinical and Translational Research Center Room 5018; 875 Ellicott St., Approval number: UBIRB IRB ID#: STUDY00005954; IRB Approval Statement - “The study materials for the project referenced above were reviewed and approved by the SUNY University at Buffalo IRB (UBIRB) by Non-Committee Review. The UBIRB has determined on 1/27/2022 that the research is Exempt according to 45 CFR Part 46.104. There is no expiration date”.

Survey was anonymous and voluntary. No personal information was obtained from the subjects who consented to participate. No children participated in the survey. Only adults >18 years of age participated in the survey.