Figures
Abstract
The objective was to compare specialty-specific 7- and 30-day outcomes between virtual care visits and in-person visits which occurred during the SARS-CoV-2 pandemic. Using administrative data from provincial databases in Ontario, ambulatory care visits occurring virtually and in-person during specific timeframes within the pandemic were analyzed. Virtual care visits were matched with corresponding in-person visits based on multiple baseline patient characteristics. We assessed short-term patient outcomes at 7 and 30 days, including subsequent visits, hospital and ICU admissions, surgeries, and mortality and compared them using multivariate logistic regression. Odds ratios were calculated as measures of association between populations. For statistical significance, we used 99% confidence intervals to account for the increased likelihood of chance findings due to the multiple comparisons conducted. Overall, 9,340,519 visits were compared between populations using a 1:1 match on a 20% random sample of the available eligible visits. Over 70% of patients included were seen by a General Practitioner. With few exceptions and across almost all specialties, revisits, ED visits, admissions, ICU and OR use, and mortality were found to be more frequent for patients seen in person. When using the administrative data available to policy makers, there is no evidence to suggest that, in the short-term, virtual care is less safe than in person care. The causes for worse in-person outcomes are not yet clear although are likely related to the streaming of more acutely unwell patients towards in-person care.
Author summary
Following the declaration of the SARS-CoV-2 pandemic, virtual care was rapidly adopted to address patient and provider needs while reducing the risk of contagion. However, the impact of this swift transition remains understudied, and it is unclear which patients, conditions, and medical specialties are best suited for virtual care. In this study, we analyzed 18 million patient visits in Ontario, Canada, comparing outcomes between in-person and virtual care. Focusing on the medical specialty of the treating physician, we assessed how often patients required urgent or emergent care, surgery, hospitalization, or experienced death within 7 and 30 days of their visit. Our findings showed that, across nearly all specialties, patients seen in person had worse outcomes and were more likely to need additional care than those seen virtually. This likely reflects a tendency for sicker patients to be directed to in-person care, even without formal system guidance. Overall, virtual care patients had positive outcomes, with no specialty showing increased risk of harm. Future research should aim to identify and standardize acuity criteria to guide and expand virtual care services effectively."
Citation: Mondoux S, Battaglia F, Gayowsky A, Clayton N, Langmann C, Miller P, et al. (2025) Specialty-specific Evaluation of Virtual care Outcomes: A retrospective QUality and safety analysis (S-EVOQUe). PLOS Digit Health 4(1): e0000708. https://doi.org/10.1371/journal.pdig.0000708
Editor: Fiona Kolbinger, Purdue University, UNITED STATES OF AMERICA
Received: August 2, 2023; Accepted: November 22, 2024; Published: January 29, 2025
Copyright: © 2025 Mondoux et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The dataset from this study is held securely in coded form at ICES. While data sharing agreements prohibit ICES from making the dataset publicly available, access may be granted to those who meet pre-specified criteria for confidential access, available at www.ices.on.ca/DAS. The full dataset creation plan and underlying analytic code are available from sICESMcMasterAdmin@ices.on.ca, understanding that the computer programs may rely upon coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification. Additional information and contact coordinates may be found at https://www.ices.on.ca/data-repository-requirements/.
Funding: This work was supported by a Juravinski Research Institute Innovation Grant (SM) and a Physician Services Incorporated (PSI) Canada grant under the Graham Farquharson Knowledge translation fellowship (SM). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The Juravinski Research Institute grant covered the cost of data extraction and the PSI grant provided protected time to SM as lead author.
Competing interests: NC was a paid research coordinator as part of this project. SM, FB, CL, PM, AP, AD and KG are all clinicians providing clinical care using both virtual and in-person modalities throughout the pandemic. AG is a paid employee of the Institute of Clinical Evaluative Sciences (ICES).
Introduction
Virtual care has rapidly emerged as an integral component of contemporary medical practice. Initially conceived as a solution to overcome geographical barriers [1] its significance escalated exponentially during the COVID-19 pandemic, enabling the continuous delivery of healthcare services amidst social distancing mandates [2]. Consequently, virtual care has swiftly evolved into a normalized care alternative for both patients and healthcare providers. Nonetheless, its widespread adoption necessitates a comprehensive evaluation of its short-term and long-term safety profiles to ensure the absence of any potential risks to patients. The critical examination of this diagnostic and care modality is imperative to prevent the perpetuation of embedded biases, both in terms of care outcomes and public policy, as we forge ahead.
Our previous research [3] has shed light on differences in outcomes between in-person and virtual care during the SARS-CoV-2 pandemic. Additional research into physician specialty-specific outcomes was required to ensure that results were not driven by a subset of specialty groupings. Although our previous study suggests that higher acuity patients may be streamed towards in-person care, it is possible that a specialty-specific analysis might provide more insight into these findings. Although ambulatory visits only rarely are provided with an acuity rating (in the ED environment), this analysis may allow for the analysis of the systematic application of a pragmatic gestalt in patient triage.
Furthermore, examining disparities in specialty-specific outcomes may provide insights into whether specific patient subgroups have factors that favor either virtual or in-person care. Such details can potentially inform and shape public policy surrounding the implementation and utilization of virtual care across various jurisdictions. Therefore, building upon our previously published work, we aimed to compare physician specialty-specific outcomes between virtual care ambulatory visits and in-person care. Specifically, the main objective of our study was to compare subsequent ambulatory visits, hospital and ICU admissions, surgeries, and mortality at 7- and 30-day between patients with initial virtual care visits and in-person visits across different specialities.
Methods
In this retrospective observational study, we explored ambulatory health care visits captured in administrative databases held at ICES, formerly known as the Institute for Clinical Evaluative Sciences, within the province of Ontario. We focused on virtual care visits and in-person ambulatory care visits occurring during the SARS-CoV-2 pandemic. The data extraction period spanned from March 14, 2020, when the pandemic was officially declared in Ontario, to March 13, 2022. To ensure the accuracy of our results and prevent the inclusion of subsequent outcomes as new index visits, a washout period of 30 days was implemented for each visit. We excluded index visits associated with outpatient surgery, chemotherapy, radiation, or COVID vaccinations based on billing data. In cases where multiple outpatient visits were billed on the same day, we carefully examined the distribution of specialties for those visits. If primary care or emergency medicine (EM) visits were recorded on such days, only the primary care or EM visits were retained for our analysis. Records that lacked primary care or EM visits or contained both types were excluded from our study. Virtual care visits were identified using billing codes specifically developed for virtual care billing purposes, encompassing all modifications made to the codes as the pandemic progressed. To ensure the selection of appropriate in-person intra-pandemic visits, we conducted a thorough review of the schedule of benefits, utilizing only office-based outpatient billing codes. Patients without a valid Ontario health card were excluded from analysis.
Data sources
To ascertain physician billings for medically necessary care, we used the provincial Ontario Health Insurance Plan (OHIP) database. The Canadian Institute of Health Information National Ambulatory Care Reporting System (CIHI-NACRS) database was used to identify emergency department (ED) visits, which contains anonymized and abstracted data for all ED visits in Ontario. Information regarding acute care hospitalizations and in-patient surgical care was obtained from CIHI’s Discharge Abstract Database (DAD). The Same Day Surgery databases were utilized to capture outpatient surgeries. Mortality data, including out-of-hospital deaths, for all Ontario residents were obtained from The Registered Persons Database (RPDB). The Resident Assessment Instrument for Home Care (RAI-HC) was used to identify long-term care residents and recipients of homecare. The Immigration, Refugees and Citizenship Canada (IRCC) database was used to identify immigration status. The Ontario Drug Benefit (ODB) database contains medical prescriptions for individuals aged 65 and above, as well as patients with provincial drug coverage. These databases were also used to identify patient comorbidities. Data were linked between databases using unique encoded identifiers and analyzed at ICES. Ontario has universal healthcare coverage for medically necessary care therefore, these databases contain the majority of healthcare utilization data within the province. All data for analysis was extracted in March 2023.
Population matching
Virtual care visits were matched 1:1 to in-person ambulatory care visits on a visit-to-visit basis, hard-matching on the variables outlined in Table 1. Patients may have been included in the study on multiple occasions if they had multiple visits during the study period. The variables utilized for matching encompassed demographics (age, sex, LHIN, rurality), the Ontario Marginalization Index and its components, long-term care resident status, Johns Hopkins ADG score, and previous healthcare utilization.
Because of the large number of virtual and in-person visits, matching was done on a random 20% sample of visits from each cohort. Additionally, several baseline characteristics of each population were compared after matching, including income quintile, immigration status, frailty, receipt of home care, receipt of ODB, and comorbidities. Balance in baseline covariates between the matched groups was assessed using standardized differences. Standardized differences below 0.10 were considered indicative of well-matched groups.
Outcome measures and analyses
The study primarily aimed to assess the incidence of acute revisits for care, hospital admissions, and mortality within 7 and 30 days across OHIP-defined specialty categories. These primary outcomes were derived from a previous study that reported aggregate outcomes [3]. Notably, when evaluating primary care specialties, the outcome measures "revisit to same specialty" and "in-person/virtual visit to primary care" were considered equivalent. Our analysis encompassed a total of eight outcomes:
- Revisit to same physician/specialty within 7 and 30 days
- In-person visit to primary care within 7 and 30 days
- Virtual visit to primary care within 7 and 30 days
- Hospital admission within 7 and 30 days
- Surgery within 7 and 30 days
- ICU admission within 7 and 30 days
- ED visit within 7 and 30 days
- Mortality at 7 and 30 days
Subsequent visits were identified by analyzing physician billings in the OHIP database within the specified timeframe. Mortality data was obtained by referencing the RPDB within the designated timeframes. Specialty-specific grouping was conducted based on the groupings defined in Table 1. These groupings were designed to ensure sufficient patient volumes within each group, enabling matching between populations. Specialty types were determined using specialty identifying codes associated with each provider in the OHIP Database. However, it is important to note that in some cases, practitioners may have a practice that differs from their specialty code (e.g., a General Practitioner working in an emergency department). Therefore, only in-office Primary Care billing codes were used to represent in-person care. For virtual care, specialty-specific codes were unavailable during the pandemic, so virtual billing codes were attributed to the specialist code under which they were billed. In cases where dual practice was possible, visits were coded according to physician specialty codes, as disambiguation was not possible.
To compare outcomes between populations, multivariate logistic regression was employed. Odds ratios (ORs) with 99% confidence intervals (CIs) were selected to account for the numerous comparisons performed in the analysis. These results were presented using Forrest plots.
Ethics and approval
ICES’s collection and use of data is authorized under Section 45 of Ontario’s Personal Health Information Protection Act (PHIPA) as a prescribed entity, which is exempt from review by a Research Ethics Board[4,5]. The use of the data in this study is authorized under section 45 and approved by ICES’s Privacy and Legal Office.
Results
Population matching
After exclusions, there were 47,092,297 visits in the virtual care visits and 46,270,897 visits in the in-person care visits who were eligible for matching. Table 2 outlines the exclusion criteria applied as well as the quantitative results of these exclusions. A total of 9,340,519 patients were matched within each population. Patient visits required a valid health insurance number and billing data on the visit which included provider specialty. Multiple visits on a single day were also excluded based on the previously mentioned strategy.
Excellent matching results were achieved between populations. A full description of this match is available within our previous published manuscript [3]. Of the matched sample, a total of 6,592,845 visits in each cohort were conducted by physicians with family practice billing codes, representing 70.6% of the total sample. Aside from the general practitioner specialty, medical subspecialties (8.5%), psychiatry (3.5%), pediatrics (3.4%) and surgery (3.2%) represent the 5 specialty groupings with the highest overall virtual care usage.
Patient outcomes
An analysis of the same outcome measures between the defined populations is summarized in Tables 3 and 4. Forrest plots are demonstrated in Figs 1 and 2. Overall differences in patterns of outcomes based on specialty are visible within the data although overall trends are relatively similar.
Interpretation
Outcome comparisons.
Within the revisit category, most specialties had an increase in return visits for in-person care visits. As mentioned in previous publications that may be attributable to the fact that patients presenting initially for in-person care may have represented more unwell or acute patients, or otherwise required in-person follow up. The exceptions to this were anesthesia, dermatology, pediatrics and the “other” category where return visits were increased for virtual care. All these results were seen for both the 7 and 30-day outcome timeframes.
Regarding in-person follow up to general practitioners, many specialties demonstrated no significant differences between in-person visits or virtual visits. Visits were more likely to occur in person by their GP after being seen virtually by Emergency Medicine, Medicine subspecialties, Oncology and Surgery in the 7-day outcomes analysis. This was also seen at 30 days with Emergency Medicine and Oncology. Patients were more likely to be seen in person by their GP if seen in person previously by their GP or by pediatrics at 7 days. At 30 days, this in-person cohort grew to include Dermatology, Geriatrics, Internal Medicine, OBGYN and Surgery in addition to Pediatrics and GP. The difference in 7 and 30-day timeframes may be attributable to a time delay which is integral to the booking period for in-person follow ups particularly for GPs.
Patients were more likely to be seen virtually by their GP if they were seen in person by Medicine subspecialties and GI at 7 days which expanded to include Urology at 30 days. Otherwise, OBGYN, Oncology and Geriatrics had no differences in virtual follow-up by GPs. All other specialties had higher likelihoods of being seen in virtual follow up if first seen virtually. This may be indicative of a “streaming” phenomenon whereby patients who made use of virtual care continued to be seen through this modality in most clinical settings.
Hospital admissions, OR use and ICU admissions, were all increased in patients who were seen in person within the 7-day timeframe across all specialties, except for Anesthesia. At the 30-day timeframe, hospital admission and ICU use was increased for virtual patients who were seen by Anesthesia and Surgery. OR use was increased for these same virtual specialties in addition to Geriatrics. The inclusion of both Anesthesia and Surgery in this category may reflect preoperative assessments done virtually before surgeries which required post-operative care in an ICU environment. The cause of significant differences between Geriatrics visits within this category is somewhat less clear.
ED presentations were increased in patients seen virtually by OBGYN, and Internal Medicine within 7 days. At 30 days, this was only true for patients seen by Surgery virtually. Patients with visits done in person had higher subsequent ED presentations within Emergency Medicine, GP, Gastroenterology, Psychiatry and Urology at 7 days with the addition of Gastroenterology, Medicine Subspecialties, Oncology and Pediatrics at 30 days.
Finally, mortality was greater in patients seen virtually by Oncology at 7 days although this dissipated at 30days where no specialties demonstrated increase mortality when seen virtually. Otherwise, mortality was increased when patients were seen in person within GP, GI, Geriatrics, Internal Medicine, Medicine subspecialties, and Pediatrics at 7days. Within the 30-day window, Pediatrics mortality was no longer significant although those seen in person in Emergency Medicine had higher rates of mortality within this timeframe.
Limitations
This study has multiple limitations. Frist, there were visits that were generated on the same calendar day by the same patient but with different clinicians. Our data does not include time-of-day information; therefore, we were unable to assess the chronological order of these visits. To address this limitation, we tried to retain Emergency Department (ED) or primary care visits that occurred on these days. However, there is a possibility of excluding important data because of these exclusions. Nonetheless, the total number of visits excluded because of this was approximately 1% or less of the final population sizes and therefore unlikely to have significantly changed our results.
As noted in our previous studies, we were unable to match for patient acuity between groups. Although this represents an important match criterion, there are no systematic triaging method for outpatient care in Ontario or Canada except for emergency medicine. Although this remains a methodological limitation, it allows for a pragmatic analysis of non-prescriptive system responses to acuity. Although we have limited evidence, the difference in acuity and the likelihood that more acute patients were streamed to in-person care is the most likely candidate explaining worse outcome for patients seen in-person.
It is also important to mention that short-term outcomes are not to be conflated with those experience by patients over several months or years after they experienced care. Many articles describe delay in diagnosis [6–8] and it is possible that these have led to worse outcomes among specific patient types. Our analysis limits itself to 30 days due to contiguity of data and a relatively short timeframe since the declared end of the pandemic.
Conclusion
When examining individual specialties and specialty groups, it appears overall trends in short-term outcomes between in-person and virtual visits demonstrates that those seen in person had more severe short-term outcomes. These outcomes are not driven principally by any specific specialty grouping. It is likely that this effect is driven principally by the streaming of more unwell patients towards in-person visits.
Acknowledgments
This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). DParts of this material are based on data and/or information compiled and provided by CIHI, the Ontario Ministry of Health, and Ontario Health. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. We thank the Toronto Community Health Profiles Partnership for providing access to the Ontario Marginalization Index.
References
- 1. Glazier RH, Green ME, Wu FC, Frymire E, Kopp A, Kiran T. Shifts in office and virtual primary care during the early COVID-19 pandemic in Ontario, Canada. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne. 2021;193(6):E200–E210. http://www.ncbi.nlm.nih.gov/pubmed/33558406. pmid:33558406
- 2. Bhatia RS, Chu C, Pang A, Tadrous M, Stamenova V, Cram P. Virtual care use before and during the COVID-19 pandemic: a repeated cross-sectional study. CMAJ open. 2021;9(1):E107–E114. http://www.ncbi.nlm.nih.gov/pubmed/33597307. pmid:33597307
- 3. Mondoux S, Battaglia F, Gayowsky A, Clayton N, Langmann C, Miller P, Pardhan A, Mathews J, Drossos A, Grewal K. Was Virtual Care as Safe as In-Person Care? Analyzing Patient Outcomes at Seven and Thirty Days in Ontario during the COVID-19 Pandemic. Healthcare quarterly (Toronto, Ont.). 2023;26(3):31–36. http://www.ncbi.nlm.nih.gov/pubmed/38018786. pmid:38018786
- 4. Government of Ontario. Ontario Regulation 329/04: General, Personal Health Information Protection Act. 2021 [accessed 2021 Dec 10]. https://www.ontario.ca/laws/regulation/040329#BK19
- 5. Government of Ontario. Personal Health Information Protection Act, 2004, S.O. 2004, c. 3, Sched. A. 2014 [accessed 2020 Nov 17]. https://www.ontario.ca/laws/statute/04p03#BK63
- 6. Bickel A, Ganam S, Abu Shakra I, Farkash I, Francis R, Karra N, Merei F, Cohen I, Kakiashvili E. Delayed diagnosis and subsequently increased severity of acute appendicitis (compatible with clinical-pathologic grounds) during the COVID-19 pandemic: an observational case-control study. BMC gastroenterology. 2022;22(1):19. http://www.ncbi.nlm.nih.gov/pubmed/35016616. pmid:35016616
- 7. Lo Giudice G, Colella G, Boschetti CE, Colella C, Tartaro G, Cirillo N. Increased Delay in Diagnosis, but Not Treatment, Among Patients With Oral Cancer During the COVID-19 Pandemic. JAMA otolaryngology—head & neck surgery. 2023;149(1):91–92. http://www.ncbi.nlm.nih.gov/pubmed/36394852. pmid:36394852
- 8. Hsiehchen D, Muquith M, Haque W, Espinoza M, Yopp A, Beg MS. Clinical Efficiency and Safety Outcomes of Virtual Care for Oncology Patients During the COVID-19 Pandemic. JCO oncology practice. 2021;17(9):e1327–e1332. http://www.ncbi.nlm.nih.gov/pubmed/34152833. pmid:34152833