1. Introduction

Approximately 20–34% of patients with schizophrenia continue to experience persistent symptoms despite adequate treatment with at least two antipsychotic drugs, and are termed treatment resistant [1, 2]. Treatment-resistance in schizophrenia (TRS) is associated with up to 11 times greater direct healthcare costs [3] and a delay in detection of TRS is associated with worse outcomes such as longer hospitalisation and delay in symptomatic improvement [4–6]. Therefore, early identification of TRS patients is a key priority [7], and modelling has shown that even modest predictive power may have clinical utility [8].

At present there is some evidence which indicates that TRS patients show abnormalities in glutamate transmission, comparatively normal dopamine transmission, and significant decreases in grey matter compared to treatment-responsive patients [9]. However, these abnormalities can only be detected using expensive and invasive brain scans, and have only modest predictive validity [10]. Genetic tests would be cheaper, but also lack predictive power [9]. In contrast, clinical and demographic information from electronic health records (EHRs) is readily available, non-invasive and inexpensive. A small number of previous studies have used clinical information to predict TRS. A recent systematic literature review of observational research [11] identified young age at illness onset as a robust predictor of TRS. In addition, other factors that were associated with TRS were psychiatric diagnosis, level of functioning, male gender, and season of birth. However, existing observational research has suffered from three main limitations. Firstly, studies using population registries have lacked detailed clinical information [12, 13]. Secondly, studies that have investigated incident cases of psychosis longitudinally have been subject to significant attrition [1], therefore potentially not capturing service users who are most unwell. Thirdly, most studies have adopted traditional frequentist statistical approaches that are not optimised for prediction [11].

This study aimed to develop a prognostic tool of TRS in a large and diverse clinical cohort, with comprehensive coverage of patients using mental health services in a defined geographical area. To achieve this, we approximated as closely as possible the operationalized TRRIP [14] definition of TRS (which is a consensus guidelines that operationalise criteria for determining TRS, providing a benchmark for research, in view of previous inconsistencies in defining and measuring treatment resistance and response) and examined a range of possible predictors using clinically rich data from EHRs.

2. Material and methods

2.4. Treatment resistant schizophrenia definition

A structured field for TRS was not available, and it was not possible to develop an NLP application that could parse the complex sequence of decisions required to arrive at a TRS diagnosis. Therefore, to determine the point at which a patient became TRS, we developed a bespoke algorithm to target potentially relevant EHRs, which were then manually examined. TRS was defined as evidence of failure to respond to two different antipsychotic prescriptions for six or more weeks, and a prescription of a third new antipsychotic. A six-week antipsychotic trial duration was chosen as a pre-requisite following NICE guidelines [17] and TRRIP TRS definition [14]. A patient was also classified as TRS, if one of their three antipsychotics was clozapine. In the UK almost all prescriptions of clozapine are for TRS- The British National Formulary, which provides the official national prescribing guidelines, lists only two indications for clozapine (1) Schizophrenia in patients unresponsive to, or intolerant of, conventional antipsychotic drugs; (2) Psychosis in Parkinson’s disease. As our cohort did not include patients with the latter diagnosis we assumed that cases of clozapine was likely to be an indicator of TRS.

Fig 1 illustrates the process of manually coding the antipsychotic treatment trials. All antipsychotic trials prescribed since 1/01/2007 were examined to determine if they were: 1) an adequate trial, lasting for at least six weeks (we assessed this by examining the first six weeks of EHRs following an antipsychotic prescription) and 2) a qualifying trial, meaning that antipsychotics were switched due to failure to respond as opposed to treatment non-adherence or treatment side-effects. To determine this we examined the EHRs for the six weeks before and after a medication switch. Where the reasons were unclear, if no evidence of switching medication due to non-adherence or side-effects was detected, then treatment failure was assumed to be the reason. For each patient, this process was repeated for all identified antipsychotic trials until TRS was identified or there were no further antipsychotics prescribed. S1 Fig summarises the decision-making process to identifying TRS.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Antipsychotic coding process timeline for coding outcome data.

https://doi.org/10.1371/journal.pone.0274864.g001

Since manually coding case-notes was very time-consuming, taking approximately one hour per patient, it was not feasible to manually code all patients who met the inclusion criteria to determine their TRS status. Consequently, a random sample comprising 10% of all patients who met the above inclusion criteria was selected for manual coding. S2 Fig describes this process. Where the EHRs indicated that fewer than three antipsychotics were prescribed, the patient was coded as not treatment resistant (unless there was a prescription of clozapine). EHRs of patients where there was evidence of three or more antipsychotics being prescribed were examined in detail following the procedures outlined above.

3. Results

In total 15,129 patients met the inclusion criteria and the information for 1,515 patients was manually coded (a random 10% of the total cohort). Out of the patients who were manually coded, 627 were identified as having three or more antipsychotic medication prescriptions. Out of those cases, 206 (33%) were identified as TRS, 413 (66%) non-TRS, and 7 (1%) were not categorised due to lack of information. Data checking identified inconsistencies (observed in 5.2%), leading to changes in the TRS date of 1.4% of all manually coded cases, namely changing their TRS date. Out of the 888 patients with fewer than three antipsychotics recorded, 47 (5%) were prescribed clozapine and therefore categorised as TRS. The remaining patients were categorised as non-TRS (N = 841, 95%). Overall, 253 (17%) patients were identified as having TRS, 1,255 (83%) as non-TRS, and 7 (0.5%) did not have sufficient information to be categorised and were not included in the analysis.

The analysis included data from 1435 individuals between 01/01/2007 and 31/12/2017 (see S2 Fig). The number of potential predictors in the model was 31. For a Cox model with 181 events (we excluded 72 patients due to being prescribed clozapine at the start of the observation window, and one who died few days afterwards, see S2 Fig) at, 8946.4 person-years, mean follow-up time of 6.23 years, 47 parameters in the model and a Cox-Snell adjusted R-squared of 0.30 or larger, a sample size of 1430 is sufficient to minimise overfitting and to ensure precise estimates of key parameters [34]. Median time to event was 2232 days (interquartile range = 1075–3694, range 2–4016). The period prevalence of TRS was 12.6% (95% CI: 10.9–14.3): 181 TRS and 1254 non-TRS.

Table 1 summarises the distribution of the exposure variables for the overall cohort and for those patients who were identified as TRS. The data contained 1044 (72.75%) complete cases, and the mean percentage of missing data per variable was 9.52% (SD 9.84, range 0–21.88%). Fig 2 is a Kaplan-Meier curve of the survival probability of TRS, which illustrates that the hazard of developing TRS was approximately constant across time in the study time window (1^st January 2007- 31^st December 2017).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Kaplan-Meier curve of the survival probability for treatment resistant schizophrenia.

https://doi.org/10.1371/journal.pone.0274864.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics of treatment resistant schizophrenia predictors.

https://doi.org/10.1371/journal.pone.0274864.t001

S1 Table summarises the model performance. The internally validated estimate of C-index for Lasso-Cox regression was 0.60, indicating a weak prediction model. The validated calibration slope of 1.27 indicated that the Cox model was slightly underfitting and therefore underestimating risk for patients at high risk while overestimating for patients at low risk. The validated ratio δ between PPV and prevalence was always greater than one, reaching 1.69 at 2 years. S2 Table summarises the validated estimates of the model’s performance- for example at 2 years the model achieved a validated PPV 0.04 (prevalence at 2 years was 0.02); NPV 0.98; sensitivity 0.60; specificity 0.63.

Table 2 displays estimated Lasso coefficients for all 31 potential predictors (47 main effect parameters). Seven variables were selected as predictors of TRS in the Lasso-Cox model (corresponding to the 1 SE penalty): having more inpatient days in the three months before and after the first antipsychotic; having more community face-to-face clinical contact in the three months before the first antipsychotic; having a minor cognitive problem requiring no action (vs not having a cognitive problem or having a severe one) according to HoNOS score, and younger age at the time of the first antipsychotic. Having a diagnosis of other prolonged psychosis emerged as a protective factor to developing TRS. The Kaplan-Meier survival curves per levels of the identified predictors are presented in S3–S10 Figs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Survival analysis Lasso Cox regression selected predictors (one standard error penalty).

https://doi.org/10.1371/journal.pone.0274864.t002

Fig 3 is a nomogram for Cox Lasso regression to calculate individual normalized prognostic indexes, given by the linear predictor line, for TRS. The coefficients are based on the Lasso Cox model. The nomogram allows us to compute the normalized prognostic index (PI) for a new patient. The PI is a single-number summary of the combined effects of a patient’s risk factors and is a common method of describing the risk for an individual. In other words, the PI is a linear combination of the risk factors, with the estimated regression coefficients as weights. The exponentiated PI gives the relative risk of each patient in comparison with a baseline patient. The PI is normalized by subtracting the mean PI. S3 Table further describes the normalized prognostic indexes (PI) for TRS translated into probabilities of developing TRS at 1, 2, 5 and 10 years. Appendix 1 describes how to compute the PI in further detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Nomogram for Cox Lasso regression to calculate individual normalized prognostic indexes (PI, given by the linear predictor line) for treatment resistant schizophrenia.

Coefficients are based on the Lasso Cox model.—To predict the rate of survival using the nomogram, one can take the example of a 30-year-old patient, with Schizophrenia, having 34 inpatient days recorded at 3 months before the 1st antipsychotic (AP) date, 50 inpatient days recorded 3 months after 1st AP date, 15 community face-to-face days recorded 3 months before 1st AP date, having a minor problem requiring no action and having mood disorders as comorbidity, has a total point score of 78 + 33 + 28 + 6 + 18 + 10 + 0 = 173. This corresponds to a normalized prognostic index of 0.57 (linear predictor line) for TRS, meaning that the patient has a probability to become TRS at 1 year falling in the range 4.14%-7.66%, at 2 years falling in the range 7.62%-13.89%, at 5 years in the range 16.45%-28.74% and at 10 years in the range 26.57%-44.13% (see S3 Table).

https://doi.org/10.1371/journal.pone.0274864.g003

In total 361 patients were already taking antipsychotics at the time the EHRs database was established. These were included in the main analysis, but their date of first antipsychotic may have been before 1 Jan 2007. Therefore, we conducted a sensitivity analysis where we excluded patients with the first antipsychotic date within the first 3 months of 2007 (n = 361 excluded, see S4 Table). We also performed a sensitivity analysis excluding patients who died in the observation window (n = 168 excluded) (see S5 Table). Both sensitivity models performed less well than the full model in relation to validated discrimination and calibration, but the corrected C-indexes were similar across all three analyses, which supported the robustness of the full model and further suggests that the events of death and TRS are not competing risks (see S6 Table).

4. Discussion

Our results indicated that the best predictors of TRS were having a diagnosis of schizophrenia or schizoaffective disorder (as opposed to a diagnosis of other psychosis); more inpatient days (in the three months before and after the first antipsychotic); having more community face-to-face clinical contact (in the three months before the first antipsychotic); having a minor cognitive problem requiring no action (as opposed to no problem or having a severe one); and younger age at the time of the first antipsychotic.

The prevalence of TRS was 17%, which is somewhat lower than previous studies in this field [1, 12, 42, 43]. One possible explanation is the use of more stringent criteria for TRS than some previous studies. In line with existing NICE guidelines [17], our study pre-specified that an adequate trial had a minimum of six weeks, as opposed to other previous research, which has allowed trials of shorter duration [1]. Having a shorter antipsychotic trial would translate to patients being classified as TRS earlier in their treatment trajectory. Furthermore, while many studies have used proxies such as hospital admissions [12] to determine response to each antipsychotic, we manually rated response to ensure that each switch in antipsychotic medication was due to non-response as opposed to side-effects or non-adherence. This represents a more robust way of determining TRS, which is in line with TRRIP guidance [14].

Our results further indicated that the hazard of developing TRS was constant over a period of 10 years. This suggests that the probability of developing TRS does not stabilise over the course of patients’ treatment trajectory. This is consistent with the findings of Wimberley et al [12], which also reported that the cumulative incidence of TRS (assessed by different indicators including 90-day polypharmacy), increased steadily over time, but does not seem to fit with data showing that the majority of TRS patients are treatment resistant from the outset [1, 7]. However, both the current study and the Wimberley et al [12] study defined TRS from switches in antipsychotic initiated by clinicians, which in turn rely on clinicians recognising, and acting on, treatment failure. There is good evidence that the detection of TRS in clinical services is often delayed by months or years [44]. Thus, the data are compatible with a situation in which TRS develops early in many patients, but is only recognised by clinicians, and thus identified by our algorithm, months or years later.

With a C-statistic of 0.60, the prediction accuracy of the final model was not sufficient to be of clinical use alone. However, in line with existing literature [12, 13, 45, 46], we found that socio-demographic, clinical and service use predictors of TRS were younger age at the illness onset, a diagnosis of other prolonged psychosis, and higher levels of service use, as measured by inpatient days and clinical face-to-face contacts. The identified predictors were robust and remained in sensitivity analyses, where we excluded patients whose prescription of the first antipsychotic was in the first three months of the observation window (which are more likely to be patients with prolonged psychosis); and patients who died between 01/01/2007 and 31/12/2017.

Younger age at onset is the most consistent risk factor identified to date for TRS [11], and this study provides further evidence in support of this. The presence of minor cognitive problems needing no action (an item on the HoNOS scale) was also associated with TRS. Other research has identified verbal intelligence and fluency as associated with TRS [47]. The fact that mild, but not more severe cognitive impairment was identified is consistent with clinical impressions that the level of cognitive impairment associated with schizophrenia is mild and generally does not require specific intervention. However, it is important to highlight here that what we report in this study is the patient’s and clinician’s perception of cognitive problems, as captured by HoNOS, as opposed to cognitive performance captured by standardised cognitive testing.

This study had several strengths. SLAM is one of Europe’s largest mental health providers and is near-universal provider of mental health care in its geographical area. Therefore, we were able to capture a large and representative sample of patients who receive national health care provision in this area. In addition, CRIS is a longitudinal data source, which encompasses clinical information from both structured and free-text fields from both inpatient and outpatient clinical settings. This enabled us to investigate TRS over a ten-year period and examine rich and diverse clinical information and therefore a wide range of possible predictors. Furthermore, our analyses were conducted on a manually coded dataset, which not only identified antipsychotic episodes in accordance with existing clinical guidelines [14], but was also able to determine reasons for switching, something that has been problematic in previous research.

There are several potential limitations that need to be considered when interpreting our findings. It is possible that our definition of TRS, which required both failed trials of antipsychotics to exceed six weeks duration, could have led to underestimating TRS cases. Less than six weeks may be adequate in some cases to establish non-response, and such trials were not counted in our algorithm. On the other hand, employing a stricter criterion to establishing TRS arguably ensures that fewer false positive cases were detected. In addition, we did not consider antipsychotic dose in assessing the adequacy of the antipsychotic trials and further research could explore the role of this on the time it takes for a patient to become TRS. The present study only considered medication trials from 01/01/2007 onwards. Consequently, some individuals may have become TRS earlier than the date identified. In relation to psychiatric diagnosis of the service users in this cohort, we included individuals who had schizophrenia, schizoaffective and other prolonged psychosis diagnosis due to the low number of patients diagnosed with schizophrenia at their illness onset, which is likely to reflect a reluctance to diagnose people with schizophrenia early in their illness. Furthermore, the prescription of the first antipsychotic from 01/01/2007 was not necessarily the patients’ first treatment, we minimised this by discounting all patients who appeared to have their first antipsychotic prescription in the first three months of the database (01/01/07–30/03/07) in the sensitivity analyses. We were only able to include factors, which were recorded in the clinical notes and had sufficient information present to allow us to include in the models. Therefore, we were unable to include potential predictors, such as smoking, due to large number of missing data. Lastly, it is important to note that inpatient days/clinical events/clinical contacts may not be constant throughout treatment (there may be periods of higher contact, namely at illness onset and around TRS date). Therefore, if a patient became TRS close to the start of the observation period (01/01/2007), this patient might be receiving frequent contact over the limited follow-up period and would appear to be a more frequent user of services than someone who became TRS later.

Our findings have potential implications for further clinical research. With a C-statistic of 0.60, our model has insufficient predictive power to be clinically useful in isolation, but it is possible that the risk factors identified here could be incorporated into a multivariate model including information from biomarkers, such as neuroimaging or genetic markers [10]. Moreover, the internally validated PPV for the survival model predicting TRS at each time point was larger than the prevalence at the same time point. Therefore, if one were to use this model as a stratification tool to identify patients with a higher chance of developing TRS, for example when recruiting patients for an RCT, fewer patients would be needed to yield a similar amount of TRS patients. For example, a trial recruiting patients identified as having a high probability of TRS by the model will yield 1.69 times as many TRS patients in 2 years than a trial recruiting unselected patients. It will therefore need to recruit 41% fewer patients to obtain a given number of TRS patients at 2 years. In relation to further research, the external validity of the model examined here can be further tested by replicating the study on a similar cohort from other settings. In conclusion, routinely collected information, readily available at the start of treatment, gives some indication of TRS but is unlikely to be adequate alone. Future studies that develop clinical risk prediction tools can combine these predictors with other identified predictors, which will increase the likelihood of success. In addition, further work investigating the biological underpinnings would be extremely useful.

Supporting information

Acknowledgments

The authors wish to acknowledge the contribution of Dr. Leonardo Koeser to the ascertainment of the manually coded TRS dataset.