Study design

Greater Manchester in the Northwest of England has an ethnically diverse population with high levels of deprivation and consists of ten localities with a total population of 2.7 million. CO@h was implemented widely for monitoring clinical deterioration in patients diagnosed with COVID-19 in Greater Manchester. Referrals were taken from all healthcare settings including primary care, urgent care, online systems ambulance services, nursing homes and hospital discharges. However, use of CO@h was not mandatory, and it was expected that its adoption would show substantial variability, allowing identification of potentially eligible patients who did and did not receive the intervention. The model used was the OxyWatch portable non-invasive oximeter, by ChoiceMMed (https://www.win-health.com/oxywatch-fingertip-pulse-oximeter). A bespoke dashboard was built in the Greater Manchester Care Record (GMCR—a joint electronic patient record across the city region). However, this was not used in all ten localities as some areas opted to use existing systems to record activity.

We used data from the Greater Manchester Secure Data Environment (GM SDE) which includes linked primary care and specialist health records [5]. Data were obtained on 6/4/2023 and were accessed from that date until 26/1/2024. We undertook a ‘target trial’ emulation approach, which applies design principles of trials to the analysis of non-randomised comparative data, explicitly matching the analysis to the trial it is emulating [6–8]. Design features of the target trial and the emulated target trial are specified in Fig 1 and we report our study in line with published guidelines [9].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Specification and emulation of a target trial.

https://doi.org/10.1371/journal.pone.0310822.g001

When first set up in September 2020, the original criteria for eligibility for CO@h were:

1. patients with COVID-19 who were symptomatic and aged 65+(later extended to 50+) or
2. ‘clinically extremely vulnerable’ to COVID-19 or
3. identified by a clinician as at high risk of deterioration from acute COVID-19.

To emulate participant eligibility, we identified individuals in the GM SDE between 50–85 years of age with a positive COVID-19 test (February 2020 to June 2022) who received CO@h and were not admitted to the hospital or who died within 24 hours of diagnosis.

Treatment assignment and follow-up

Instead of random assignment, we initially retrieved data from anonymised patients identified in the GM SDE as enrolled in CO@h. These were to be matched to similar patients in the GM SDE who were not identified as enrolled in CO@h.

The identification of suitable comparator patients was complicated by the fact that only 2 of the 10 areas in Greater Manchester routinely identified those enrolled in CO@h in the GM SDE. Therefore, we could be sure that comparator patients in those 2 areas would not have received CO@h. However, in the remaining 8 areas, people in those areas not coded as enrolled in CO@h would have included some people who had been enrolled on CO@h and others who were not. Therefore, the comparator group would have a degree of ‘contamination’.

We therefore undertook 3 analyses:

1. Comparison A compared those identified as enrolled in CO@h in 2 areas, with similar patients across all 10 areas
2. Comparison B compared those identified as enrolled in CO@h in 2 areas, with similar patients across the remaining 8 areas
3. Comparison C compared those identified as enrolled in CO@h in 2 areas, with similar patients from those 2 areas only

Comparison A and B are both influenced by contamination of the comparator group, although rates of contamination would be higher in B than in A. Comparison C is not affected by contamination, but may be more vulnerable to other selection biases in those areas which determined who was and who was not enrolled in CO@h.

Control patients were exactly matched by gender, age (within 1 year) and date of positive COVID-19 test (within 30 days) to participants who were not enrolled (ratio of 15 comparators per intervention patient). To maximise comparability on additional characteristics, we then used weighted coarse exact matching [10] on the categories of the following variables: Charlson score quartiles, ethnicity category, deprivation quintiles (as measured by the Index of Multiple Deprivation), and admissions within the 12 months prior to COVID-19 diagnosis. After defining the treatment variable, coarse exact matching involves three steps: (1) grouping each variable in defined categories (2) sorting all units into strata, each of which has the same values of the coarsened variable (3) removing units in any strata that do not include at least one treated and one comparator. The result is typically less model-dependent, has lower bias and increased efficiency compared with other matching methods, and can be combined with model based strategies (such as inverse probability of treatment weighting) to estimate a causal effect [10, 11]. Conditional on this matching, it is expected that the distribution of observed baseline characteristics would be similar between groups. Balance was formally evaluated by estimating standardised mean differences and variance ratios.

In a target trial, the natural baseline would be the time when the intervention is randomly allocated. The optimal way to emulate the target trial was to define time zero for the intervention as the date when a patient received the intervention as identified in the GM SDE, and the same start date for matched controls. Each patient was then ‘followed up’ to 1 year after the index date or until June 2022 if this period was shorter. The follow-up ended at that time or until death if the patient died within the 12 months after COVID-19 diagnosis.

Outcomes

Primary outcomes were (a) unplanned hospitalisation for any reason and (b) all-cause mortality. Both were reported at 28 days, 90 days and 1 year after COVID-19 diagnosis. A secondary outcome was length of hospital stay for those admitted.

Statistical analysis

We describe participant characteristics in the intervention and comparator groups using measures of central tendency and dispersion for continuous variables and proportions for categorical variables. We compared the distribution of all covariates between the groups using t-tests or non-parametric tests for continuous variables and chi² test for proportions.

We assessed our primary outcomes using multiple logistic regression and survival models (competing risk models for unplanned hospitalisations and Cox regression for all-cause mortality). We estimated odds ratios and hazard/subhazard ratios of the intervention on the primary outcomes, adjusting for age, gender, ethnicity, Charlson quartiles, IMD quintiles and admissions within the 12 months prior to COVID-19 diagnosis, as there are likely to be varying risks for unplanned admissions and mortality for these groups that may confound the relationship between CO@h and outcomes of interest. Finally, we used augmented inverse probability of treatment weighting [12, 13] to estimate the average treatment effects of CO@h on mortality. This estimator requires (1) specification of a binary regression model for the propensity score, and (2) specification of a regression model for the outcome variable. This so-called ‘double robustness’ allows the estimator to remain consistent for the average treatment effect even if either the propensity score model or the outcome regression is mis-specified but the other is properly specified. It has been shown that the augmented inverse probability weighted estimator is superior to other methods of model-based matching [12]. Length of stay was analysed using Poisson or negative binomial regression, if overdispersion was evident visually and statistically (the variance was greater than the mean). Data was complete for all variables except for ethnicity, for which a “missing” category was used. All analyses were done with Stata v18.

Statement of principal findings

Using target trial emulation with routine NHS data, our analyses suggested that CO@h was associated with a higher frequency of unplanned admissions and a reduction in mortality up to 1 year later, after matching and adjustment for clinical and sociodemographic characteristics. In absolute terms, this difference translates to 2–10 deaths averted per 1000 patients.

Strengths and limitations of the study

Using local routine data, we were able to identify a proportion of the patients in the region receiving the intervention, as well as data on diagnosis, admissions and mortality dates, demographic and clinical characteristics, local setting information and use of hospital services. We adopted a target trial approach to enhance rigour through pre-specification of our eligibility criteria and analysis. Target trial emulation not only includes methods to balance comparisons to limit bias and confounding but also considers important additional elements of the trial design, such as defining clear eligibility criteria of potential participants, treatment assignment and deployment, definition of ‘time zero’ (starting point of the intervention and control group) and follow up, and definition and measurement of prespecified outcomes [6, 7]. Coarse exact matching achieved a good balance between groups, and we controlled for numerous potential confounders. However, several limitations also exist. Although the methods we used are amongst the best available, we can never be certain on causality, and the quality of evidence can never match that of a randomised controlled trial. No matter how many variables are included in the modelling to account for measured confounding, there are always risks: not achieving good enough balance for the important drivers of the outcome of interest (since matching is a dimension reduction method); the possibility of residual confounding by disease severity or other factors (unmeasured confounding), for example, weight or BMI; finally, these are observational data from health care records, and their quality is, more often than not, not very high. It also is possible that some patients who were admitted to hospital and required oxygen were prescribed Dexamethasone and/or were part of the RECOVERY trial in 2021 [14]. However, by definition, the patients admitted to the CO@H service were not on oxygen and were considered low risk, thus it can be assumed that the vast majority of patients did not have access to COVID19 specific treatments. However, as we don’t have prescribing data, we cannot confirm this.

Patients and professionals were aware of the receipt of CO@h and this awareness may have affected their behaviour (e.g. patients and their families using CO@h may have been more conscious of certain signs and symptoms or followed recommendations more strictly). This is similar to any pragmatic trial, which would likely be the optimal approach for the evaluation of an intervention like CO@h. Lack of independent outcome validation using routine data may risk misleading estimates due to differential outcome ascertainment, However, we do not expect our measures to be influenced because they are routinely recorded for all patients.

Table 4 presents the characteristics of our study alongside published evaluations. Our analysis had the largest sample size and was able to assess outcomes up to 1 year. The studies used different criteria for patient eligibility, especially on the age groups included and the settings. Our study identified patients recorded as having been referred to the CO@h, matched on several baseline demographic and clinical characteristics with those who were not coded as receiving the intervention, and included patients 50 years or over. Other studies included much younger patients, starting at 18 years, and included patients attending A&E or admitted to the hospital after having received (or not) CO@h prior to admission. These differences in patient characteristics and settings may account for differences in study results, suggesting that patients had dissimilar baseline risks if they attended the emergency department or were admitted to the hospital compared to patients enrolled in an ongoing successful monitoring programme or identified from a dataset of care records.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparison of patient level studies evaluating CO@h.

https://doi.org/10.1371/journal.pone.0310822.t004

As well as the issues identified in coding of CO@h, patients tagged as having received the intervention were considered the intervention group, which means that those who were identified as eligible but who refused might have been included in the comparator group, therefore limiting the emulation of intention to treat analysis by not including them in the treated group as originally intended. This pragmatic assessment explored the impact of the policy of providing CO@h, and we had no data on patient adherence to the protocol. The coding issues identified highlight that importance of accurate and consistent coding of innovative interventions if the potential of routine NHS data for analysis of interventions (as opposed to epidemiological uses) is to be realised.

Meaning of the study: Possible mechanisms and implications for clinicians or policymakers

Beaney et al. state that monitoring programs like CO@h should be viewed ‘as a pathway to support appropriate escalation and decision-making for assessment or in-patient admission’ [2]. Most studies have treated post-COVID-19 admissions as a ‘process’ outcome and mortality as an end point separately, although in the discussion it is usually mentioned that better use of services implies an expected greater number of admissions among patients on CO@h, and that may be part of the benefit on mortality due to timely detection of low oxygen saturation and escalation [15]. There is some evidence from this study that supports this view. In our study, the risk of post-COVID-19 admissions was higher for CO@h patients, considering death as a competing event. Although timely in-patient hospital admissions might be a partial mediator of the protective effect of CO@h on mortality, a post hoc sub-group exploration found a greater reduction in adjusted mortality risk among those patients who did not have an in-patient admission (adjusted OR 0.27 (95% CI 0.13–0.58) and more uncertainty in mortality risk among those who had at least one unplanned in-patient admission post-COVID-19 (adjusted OR 0.68 (95% CI 0.39–1.18). These findings may suggest that the effect of the CO@h programme is more evident among patients presenting with a less severe clinical profile and not always mediated by timely admissions.

Unanswered questions and future research

Our study adds to the developing evidence base on the benefits of CO@h, including a longer term assessment of outcomes. Key questions about this technology remain, including: a comprehensive assessment of cost-effectiveness (which would in turn require assessment of quality of life in patients which is not routinely collected); wider patient and carer impacts (including quality of life, treatment burden and satisfaction); impacts on the workforce; and wider impacts on hospital capacity and efficiency. Further analysis on subgroups may be indicated, and would be important in the context of wider evidence showing differential accuracy in the technology associated with levels of skin pigmentation [16]. Work could also usefully explore the involvement of other services in supporting people on CO@h that might be supporting reduced risks to health without admission, as well as more general barriers and facilitators to the implementation of CO@h which might be relevant to other innovations based on remote monitoring such as virtual wards [17–19].