2.1. Design and pre-registration

This protocol builds on and extends prior work by our group [11,15,29–31]. This IPD-NMA will be pre-registered in the PROSPERO International prospective register of systematic reviews after acceptance of this protocol for publication in a peer-reviewed journal [29–31]. Additional important protocol amendments will be updated in this register [29–31]. Extraction of data from the primary datasets into the database for this project will start after PROSPERO registration. The Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) [32] and the Transparent Reporting of Multivariable Prediction Models Developed or Validated using Clustered Data (TRIPOD-Cluster) statements [33] guide the writing of this article (see S1 and S2 Tables), and a detailed analytic plan for predictive modelling will be pre-registered in the Open Science Framework.

2.2. Eligibility criteria

Eligible studies are randomized clinical trials for adult acute-phase depression comparing two or more of the following interventions: ADM, CT, BA, IPT, STPP. No restrictions will be placed concerning the years when the study was conducted, or with regard to publication language, date, or status [29–31].

2.3. Outcome

Depressive symptom level at treatment completion will be the primary outcome of this study, as symptom reduction is considered to be the main aim of acute-phase depression treatments [29,31]. We chose a continuous measure over categorical outcomes (e.g., remission) as our primary target because dichotomizing continuous variables reduces statistical power for estimating interactions with treatment [25,29–31].

Depressive symptom level at treatment completion is operationalized as a participant’s score on the Beck Depression Inventory-II (BDI-II) [47] at the primary post-treatment time point as defined by the study’s authors [30,31]. We chose the BDI-II as the main outcome because it is a self-reported patient-centered instrument that was designed to measure depression severity [47], is frequently applied in depression treatment research, and is sensitive to change [48]. If collected study data does not include BDI-II, we will convert scores on the primary (secondary, third, fourth, etc. in this order) continuous depression scale, as defined by the study’s authors, into BDI-II scores using existing conversion algorithms [49–51]. If none of the measures assessed in the study is included in conversion algorithms, we aim to develop one ourselves based on the collected IPD. When none of the used outcome scales can be justifiably converted into the BDI-II, we will exclude the study from analysis. If this results in three or more excluded studies, we will consider transforming primary continuous depression outcomes to a 0–100 scale for all studies.

2.4. Predictors

Moderators of treatment effect (elsewhere also referred to as “prescriptive predictors”, “effect modifiers” or “treatment-covariate interactions”) affect the direction or magnitude of differences in outcome between two treatments [52], and thus can help predict whether a patient will benefit more from one treatment than another [18]. Variables that moderate individuals’ response to treatment can also have prognostic (or “main”) effects (see Cohen & DeRubeis [18] for additional discussion), which will also be included in the model.

Putative predictor variables include demographic (e.g., age, gender), clinical (e.g., depression severity, depressive episode duration, comorbid anxiety disorder), and psychological (e.g., personality, coping style) participant characteristics assessed before the start of treatment [29–31]. Candidate variables will be selected for inclusion in the model based on a recent systematic review on predictors of treatment response in depression and sufficient availability across the studies, following the approach described by Vale and colleagues [53].

2.5. Systematic literature search

To identify eligible studies, we will use a search strategy that has been described previously [30,31]. We will search a database of randomized clinical trials examining the efficacy and effectiveness of psychological treatments for depression that has been used in a series of published meta-analyses (www.metapsy.org) [54]. This database was developed through comprehensive literature searches in the bibliographic databases PubMed, PsycINFO, Embase.com, and the Cochrane Library and is updated every four months. The search strings use a combination of index terms and free-text words indicative of depression and psychotherapies, with filters for randomized clinical trials. The exact search terms can be found at https://osf.io/nv3ea/. Two raters independently screen all records, assess full-text papers for METAPSY database eligibility, and categorize the treatment comparison(s) of included studies, with disagreements being resolved in consensus in each phase [30,31].

Two other raters will independently assess all full-text papers of studies marked as comparing a psychotherapy mono-treatment condition against another active (i.e., psychotherapy or ADM) mono-treatment condition for meeting the inclusion criteria for this work [30,31]. Disagreements will be resolved through consensus and if consensus cannot be reached, a third rater will be consulted [29–31]. To identify studies that might have been missed, we will check the references of prior reviews and meta-analyses [30,31]. We will also contact psychotherapy listservs to request ongoing or unpublished studies, and studies that were missed [29,30].

2.6. IPD collection

We will invite authors of eligible studies to participate in this project using a strategy that has been successful in soliciting participation in previous depression treatment IPD meta-analyses [11,29–31]. We will invite representatives of each team that shares IPD to join as authors on all publications resulting from the use of these data, inasmuch as they meet internationally accepted criteria for authoring scientific articles (www.icmje.org) [29–31]. In addition, we will make the combined database available to investigators sharing IPD to examine other research questions, provided that the authors of the original studies approve the use of their data for this purpose [11,29–31,55].

We will apply a multi-step protocol to contact authors. We have described this protocol in more detail previously [29–31] and it has proven to be successful in reaching authors for depression treatment IPD meta-analyses. In short, we will contact the authors (starting with the corresponding author and proceeding to the other co-authors) by email with three reminders. In case of no response to email, we will first mail a letter to the corresponding author (again with three attempts) and then contact the corresponding author by telephone. This is followed by contacting the other authors by letter and telephone, and other ways to contact one of the authors (e.g., via colleagues who might know them) [29–31]. A study’s data will be considered unavailable only if all these attempts fail, or an author either indicates that the IPD were not retained or declines to share these data [29–31].

We will request the following anonymized participant-level data items: randomized treatment condition, all outcome variables assessed prior to, during, and after treatment (with item-level data for depression outcome measures), and all potential predictor variables assessed in the study [30,31]. We will also ask for the study protocol and the assessment batteries used in the study.

The following study-level characteristics are extracted from the publication by two independent raters upon inclusion in the METAPSY database: country, recruitment method (e.g., community, clinical), target group (e.g., adults in general, students), depression inclusion criteria, number of treatment sessions, treatment format (e.g., individual, group), and assessment time points [30,31]. We will examine whether the ADM treatment protocols meet minimum adequate clinical best practice guidelines by assessing studies for the use of a therapeutic dosage and titration schedule (i.e., therapeutic dose achieved within three weeks) [30]. Study ADM will be deemed adequate if both criteria are met [27,30]. We will examine psychotherapy protocol adequacy with regard to use of a treatment manual, provision of therapy by trained therapists, and verification of treatment integrity [29–31]. Treatment quality characteristics will be extracted by two raters independently and disagreements will be resolved by consensus [29–31]. If consensus cannot be reached, a third rater will be consulted. Characteristics not reported in the publications will be requested from the authors [29–31].

Risk of bias in the included studies will be assessed using the Cochrane risk-of-bias tool for randomized trials version 2 [56], both before and after the IPD are obtained. Ratings will be primarily based on information reported in the publications, though risk of bias due to deviations from the intended intervention, due to missing outcome data, and in selection of the reported result will be assessed using the IPD (i.e., domain 3.1) or the IPD meta-analysis’ abilities to correct for the relevant risk of bias (i.e., domains 2.6–2.7, 3.2, and 5.1–5.3). Two raters will independently assess the risk-of-bias tool at study level. Disagreements will be resolved by consensus [30,31]. If consensus cannot be reached a third rater will be consulted. As studies are expected to be included that were published before the universal adoption of reporting guidelines for randomized clinical trials [57], requisite information will be requested from the authors if not reported in the publications [31].

2.7. IPD integrity checks

We will perform three data integrity checks [29–31]. First, we will check whether the dataset includes the full intention-to-treat sample (defined as all participants randomized to treatment in the study) and otherwise matches the data reported in the published article [29–31]. To this end, means and standard deviations or raw counts for all baseline characteristics, and observed pre-treatment and post-treatment scores reported in the article will also be calculated from the dataset and both will be compared [29–31]. Second, we will check whether all outcome and potential predictor variables reported in the article are included in the dataset [29–31]. Third, the outcome and predictor variables will be checked for inconsistent, invalid, or out-of-range values [29–31], including values that conflict with the primary study’s or this work’s eligibility criteria (e.g., age < 18). Discrepancies resulting from these data integrity checks will be discussed with the authors [29–31]. In addition, authors will be contacted if concerns arise regarding duplicate participants across trials (e.g., when multiple trials are conducted by the same author group).

2.8. Building the IPD database

All datasets received that contain IPD for at least one continuous depressive symptom measure will be considered for quantitative synthesis and will be pooled in a single database [29–31]. Predictor variables are expected to be assessed differently across individual studies and will therefore be harmonized [29–31]. Categorical variables (e.g., marital status) will be recoded so that they contain similar categories and corresponding coding schemes across all studies that include that variable (e.g., 0=single, 1=married, 2=separated/divorced, 3=widowed, 4=not married, but living together, 5=other).

To improve rigor and reproducibility of our data processing and cleaning pipelines, we developed a spreadsheet- and code-based system that also increases efficiency and scalability, allowing for changes to be instantaneously implemented across the database, and facilitating addition of new data. We will describe this system in more detail on the project’s Open Science Framework page.

2.9. Missing data imputation

Data will be imputed for variables that are ‘systematically’ missing for certain studies and values that are ‘sporadically’ missing for certain individual participants. We will use methods for multiple imputation that account for the clustering of participants in different studies [58,59], utilizing both baseline variables as well as all intermediate and endpoint outcomes available in the datasets [60]. If there are no systematically missing predictors in the studies, we will perform multiple imputation of missing data for each study separately as our primary analysis, and use the procedure described above as sensitivity analysis. Single imputation will also be considered, for instance for imputation of baseline variables, if it follows recommended advice [61].

2.10. Modelling approach

First, we will estimate the average relative treatment effects (mean differences) using pairwise meta-analyses and a random effects NMA based on aggregate data. NMA enables us to obtain relative treatment effects for all treatment comparisons, even for those without direct evidence [62], as long as they are connected in the network. One prerequisite for NMA is transitivity, i.e., that the included studies do not differ with respect to the distribution of effect modifiers when grouped by comparison. Transitivity also means that a ‘mega-trial’ would be possible, i.e., a hypothetical trial comparing all treatments in the network, and that all patients would in principle be eligible to receive any of these treatments. We will assess transitivity conceptually and by examining candidate effect modifiers across comparisons. Lack of transitivity may affect the consistency of the network, or the extent to which the direct and indirect estimated effect of a particular comparison align. When direct and indirect estimated effects are inconsistent (i.e., do not align), NMA results may be invalid. Therefore, we will evaluate statistical consistency using the design-by-treatment interaction model [63] and the back calculation method [64]. We will assume a common heterogeneity parameter in the network, and we will assess its extent by looking at prediction intervals. For these analyses, we will use the netmeta package in R [65].

To model the relative individualized treatment effects amongst the five treatments, we will conduct a two-stage IPD-NMA in the rjags R package [66]. In the first stage, we will analyze each study separately using a suitable regression model (e.g., linear regression adjusting for baseline) to estimate the treatment effect at average values of the predictors as well as all treatment-predictor interactions. For estimating treatment-predictor interactions, we will consider using penalization (shrinkage). Such methods tend to shrink (decrease the values of) the estimated coefficients as compared to usual maximum likelihood approaches. A simulation has shown that penalization methods may lead to more accurate predictions (i.e., smaller mean squared error) regarding patient-specific treatment effects in IPD meta-analysis compared to unpenalized methods [67]. We will fit the model in each of the multiply imputed datasets for each study. Then, we will combine posterior samples of the model parameters across the imputed datasets, to estimate study-level parameters and their corresponding uncertainty [68].

In the second stage, we will synthesize all study-specific results (treatment effect estimates, and interactions, and their standard errors) using a (multivariate) NMA model, where, as above, we will assume common between-study heterogeneity across all treatment comparisons. The end product will be a set of estimates of the treatment effect at average values of the predictors as well as treatment-predictor interactions (i.e., effect modification). The final model will allow us, given a participant’s baseline values, to predict individualized treatment effects between any two treatments (T_A and T_B) in the network, where treatment effect means the predicted difference in outcome if the individual received T_A versus T_B.

2.11. Cross-validation of the model

We will assess the performance of the model with internal-external cross-validation [69,70]. More specifically, we will repeat the analytical process described above after taking one study out of our sample and use the remaining studies to fit our model. Then, we will apply the model to the left-out study, and for each patient therein we will calculate the predicted treatment effects among all treatments in the study. Finally, using these predictions, we will evaluate model performance within and across studies using meta-analysis methods. Conceptually, this evaluation will follow the Personalized Advantage Index approach described by DeRubeis and colleagues [71] in which outcomes of participants who received the treatment recommended by the model are compared against the outcomes of participants who received a non-recommended treatment [71]. This has been also termed “population benefit” index [72]. This measure combines aspects of discrimination-for-benefit and calibration-for-benefit. A useful model for personalized treatment effects should have an index larger than the average treatment effect. We will also assess calibration-for-benefit purely by producing calibration plots and fitting a regression for benefit [72]. Analyses will be done with the predieval R package (v.0.1.1) [73].

2.12. Assessment of meta-biases

To assess the potential impact of risk-of-bias, we will conduct a random effect NMA excluding studies at high risk of bias. We will visually compare the 95% confidence intervals of the mean relative treatment effects for all treatment comparisons between this and the primary analysis.

We will assess potential data-availability bias by comparing effect sizes (i.e., mean relative treatment effects) between studies for which IPD were and were not available with a subgroup NMA. To facilitate comparability, effect sizes will be calculated based on the effect size data extracted from the publications upon inclusion in the METAPSY database for all studies (rather than being based on the IPD when available) [31].

Small study effects (potential publication bias) will be assessed by examining asymmetry in contour-enhanced funnel plots with Egger’s test of the intercept for pairwise comparisons including ten or more trials [30,31,74]. The analyses will include both studies for which IPD were versus were not available and will, therefore, also be based on the effect size data extracted from the publications [31]. We will assess the strength of the body of evidence based on the number of included studies and participants as well as the quality of the included studies [30,31].

2.13. Ethical standards

Institutional Review Board (IRB) approval is not required for this project, because we will work with anonymized data from treatment studies that have already been completed [29–31]. IRB approval might be required for the investigators to share their IPD depending on their institution’s policies [29–31]. It will be the responsibility of the investigators to obtain IRB approval if their institution’s policies require them to do so [29–31]. By signing the data sharing agreement, the authors who share their IPD will declare that those data were collected and will be transferred according to all applicable local and international laws and regulations [29–31]. Furthermore, they declare that all IPD will be anonymized, so that no personal data are transferred [31].