https://orcid.org/0000-0002-7481-9671
Figures
Abstract
Rationale
Robust COVID-19 outcomes classification is important for ongoing epidemiology research on acute and post-acute COVID-19 conditions. Protocolized medical record review is an established method to validate endpoints for clinical trials and cardiovascular epidemiology cohorts; however, a protocol to adjudicate hospitalizations for COVID-19 among epidemiology cohorts was lacking.
Objectives
We developed a protocol to ascertain and adjudicate hospitalized COVID-19 across a meta-cohort of 14 US prospective cohort studies. This report describes the first three years of protocol implementation (October 1, 2020—October 1, 2023) and evaluates its repeatability and performance compared to classification by administrative codes.
Methods
The protocol was adapted from cohort approaches to clinical cardiovascular events ascertainment and adjudication. Potential COVID-19 hospitalizations and deaths were identified by self-/proxy-report and, in some cases, active surveillance. Medical records were requested from hospitals and adjudicated for COVID-19 outcomes by clinically trained personnel according to a standardized rubric. Inter-rater agreement was assessed. The sensitivity and specificity of discharge diagnosis codes was compared to adjudicated diagnoses.
Measurements and main results
The study obtained medical records for 1,167 potential COVID-19 hospitalizations, which underwent protocolized adjudication. Adjudication confirmed COVID-19 infection was present for 1,030 (88%) events, of which COVID-19 was not the cause of hospitalization for 78 (8%). Of 952 hospitalizations determined by adjudicators to be caused by COVID-19, 319 (34%) participants were critically ill and 210 (22%) died. Pneumonia was confirmed in 822 (86%) and acute kidney injury in 350 (37%); other cardiovascular and thrombotic complications were rare (2–5%). Interrater reliability among adjudicators was high (kappa = 0.85–1.00) except for myocardial infarction (kappa = 0.60). Compared to adjudication, sensitivity of discharge diagnosis codes was higher for pneumonia (84%) and pulmonary embolism (81%) than for other complications (48–70%).
Conclusions
Protocolized adjudication confirmed four out of five COVID-19 hospitalizations in a US meta-cohort and confirmed cases of pneumonia, pulmonary embolism, and other conditions that were not indicated by discharge diagnosis codes. These results highlight the importance of validating health outcomes for use in research on COVID-19 and post-COVID-19 conditions, and some limitations of claims-based data.
Citation: Oelsner EC, Krishnaswamy A, Rustamov R, Balte PP, Ali T, Allen NB, et al. (2025) Classifying COVID-19 hospitalizations in epidemiology cohort studies: The C4R study. PLoS ONE 20(2): e0316198. https://doi.org/10.1371/journal.pone.0316198
Editor: Kamal Sharma, UN Mehta Institute of Cardiology and Research Center, INDIA
Received: September 16, 2024; Accepted: December 8, 2024; Published: February 10, 2025
Copyright: © 2025 Oelsner et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: To analyze de-identified individual participant data, researchers must receive authorization for access. This authorization is required to maintain the integrity of the data and protect participant privacy. Those wishing to access C4R data will need to submit a Data Access Request (DAR) through the database of Phenotypes and Genotypes (dbGaP) (https://www.ncbi.nlm.nih.gov/gap/). Authorized researchers can then access C4R data from BioData Catalyst® (BDC) (https://biodatacatalyst.nhlbi.nih.gov/). BioData Catalyst (BDC) is a cloud-based research ecosystem that permits access to and a platform for the analysis of scientific data supported by NHLBI. BDC currently hosts C4R datasets from 11 of the 14 C4R cohorts with additional C4R Data becoming accessible in the coming months. Cohort data from many of the C4R parent studies is also accessible through a dbGaP data access request. BDC provides instructions for finding and viewing C4R and cohort data as well as the steps to request authorization to use de-identified individual participant data for scientific analysis within BDC. Accession numbers are XXXX.
Funding: The Collaborative Cohort of Cohorts for COVID-19 Research (C4R) Study is supported by National Heart, Lung, and Blood Institute (NHLBI)–Collaborating Network of Networks for Evaluating COVID-19 and Therapeutic Strategies (CONNECTS)/Researching COVID to Enhance Recovery (RECOVER) grant OT2HL156812, with co-funding from the National Institute of Neurological Disorders and Stroke (NINDS) and the National Institute on Aging (NIA). The Principal Investigator is ECO. Additional funding for participating cohorts is as follows: ARIC The Atherosclerosis Risk in Communities Study has been funded in whole or in part by the NHLBI, National Institutes of Health (NIH), US Department of Health and Human Services, under contracts 75N92022D00001, 75N92022D00002, 75N92022D00003, 75N92022D00004, and 75N92022D00005. Neurocognitive data are collected under grants U01 2U01HL096812, 2U01HL096814, 2U01HL096899, 2U01HL096902, and 2U01HL096917 from the NHLBI, the NINDS, the NIA, and the National Institute on Deafness and Other Communication Disorders. The authors thank the staff and participants of the ARIC Study for their important contributions CARDIA The Coronary Artery Risk Development in Young Adults Study (CARDIA) is conducted and supported by the National Heart, Lung, and Blood Institute (NHLBI) in collaboration with the University of Alabama at Birmingham (75N92023D00002 & 75N92023D00005), Northwestern University (75N92023D00004), University of Minnesota (75N92023D00006), and Kaiser Foundation Research Institute (75N92023D00003). This manuscript has been reviewed by CARDIA for scientific content. COPDGene The Genetic Epidemiology of COPD (COPDGene) Study was supported by awards U01 HL089897 and U01 HL089856 from the NHLBI. COPDGene is also supported by the COPD Foundation through contributions made to an industry advisory board comprised of AstraZeneca AB (Cambridge, United Kingdom), Boehringer-Ingelheim (Ingelheim am Rhein, Germany), Genentech, Inc. (South San Francisco, California), GlaxoSmithKline plc (London, United Kingdom), Novartis International AG (Basel, Switzerland), Pfizer, Inc. (New York, New York), Siemens AG (Berlin, Germany), and Sunovion Pharmaceuticals Inc. (Marlborough, Massachusetts). FHS The Framingham Heart Study has received support from the NHLBI (grant N01-HC-25195, contract HHSN268201500001I, and grant contract 75N92019D00031). HCHS/SOL The Hispanic Community Health Study/Study of Latinos (HCHS/SOL) is a collaborative study supported by contracts between the NHLBI and the University of North Carolina (contract HHSN268201300001I/N01-HC-65233), the University of Miami (contract HHSN268201300004I/N01-HC-65234), Albert Einstein College of Medicine (contract HHSN268201300002I/N01-HC-65235), the University of Illinois at Chicago (contract HHSN268201300003I/N01-HC-65236 (Northwestern University)), and San Diego State University (contract HHSN268201300005I/N01-HC-65237). The following institutes/centers/offices have contributed to the HCHS/SOL through a transfer of funds to the NHLBI: the National Institute on Minority Health and Health Disparities, the National Institute on Deafness and Other Communication Disorders, the National Institute of Dental and Craniofacial Research, the National Institute of Diabetes and Digestive and Kidney Diseases, the NINDS, and the NIH Office of Dietary Supplements. JHS The Jackson Heart Study is supported by and conducted in collaboration with Jackson State University (contract HHSN268201800013I), Tougaloo College (contract HHSN268201800014I), the Mississippi State Department of Health (contract HHSN268201800015I), the University of Mississippi Medical Center (contracts HHSN268201800010I, HHSN268201800011I, and HHSN268201800012I), the NHLBI, and the National Institute on Minority Health and Health Disparities. MASALA The Mediators of Atherosclerosis in South Asians Living in America (MASALA) Study was supported by grant R01HL093009 from the NHLBI, the National Center for Research Resources, and the National Center for Advancing Translational Sciences, NIH, through University of California, San Francisco–Clinical and Translational Science Institute grant UL1RR024131. MESA The Multi-Ethnic Study of Atherosclerosis (MESA) and the MESA SNP Health Association Resource (SHARe) are conducted and supported by the NHLBI in collaboration with the MESA investigators. Support for MESA is provided by grants and contracts 75N92020D00001, HHSN268201500003I, N01-HC-95159, 75N92020D00005, N01-HC-95160, 75N92020D00002, N01-HC-95161, 75N92020D00003, N01-HC-95162, 75N92020D00006, N01-HC-95163, 75N92020D00004, N01-HC-95164, 75N92020D00007, N01-HC-95165, N01-HC-95166, N01-HC-95167, N01-HC-95168, N01-HC-95169, R01-HL077612, R01-HL093081, R01-HL130506, R01-HL127028, R01-HL127659, R01-HL098433, R01-HL101250, and R01-HL135009 from the NHLBI; grant R01-AG058969 from the NIA; and grants UL1-TR-000040, UL1-TR-001079, and UL1-TR-001420 from the National Center for Advancing Translational Sciences. NHLBI Pooled Cohorts Study The NHLBI Pooled Cohorts Study was supported by grants R21HL153700, K23HL130627, R21HL129924, and R21HL121457 from the NIH/NHLBI. NOMAS The Northern Manhattan Study was supported by grants R01 NS29993 and R01 NS48134 from the NINDS and grant R01 AG066162 from the NIA. PrePF The Prevent Pulmonary Fibrosis cohort study was established in 2000 and has been supported by NIH awards Z01-ES101947, R01-HL095393, RC2-HL1011715, R21/33-HL120770, R01-HL097163, Z01-HL134585, UH2/3-HL123442, P01-HL092870, UG3/UH3-HL151865, and DoD W81XWH-17-1-0597. REGARDS The REeasons for Geographic and Racial Differences in Stroke (REGARDS) Study is supported by cooperative agreement U01 NS041588, co-funded by the NINDS and the NIA. Additional funding was provided by R01 HL80477 and R01 HL165452 from the NHLBI. SARP Research by the principal and co–principal investigators of the Severe Asthma Research Program was funded by the NIH/NHLBI (grants U10 HL109164, U10 HL109257, U10 HL109146, U10 HL109172, U10 HL109250, U10 HL109168, U10 HL109152, and U10 HL109086). Additional support was provided through industry partnerships with the following companies: AstraZeneca, Boehringer-Ingelheim, Genentech, GlaxoSmithKline, MedImmune, Inc. (Gaithersburg, Maryland), Novartis, Regeneron Pharmaceuticals, Inc. (Tarrytown, New York), Sanofi S.A. (Paris, France), and Teva Pharmaceuticals USA (North Wales, Pennsylvania). Spirometers used in Severe Asthma Research Program III were provided by nSpire Health, Inc. (Longmont, Colorado). SPIROMICS The Subpopulations and Intermediate Outcome Measures in COPD Study (SPIROMICS) has been funded by contracts with the NIH/NHLBI (contracts HHSN268200900013C, HHSN268200900014C, HHSN268200900015C, HHSN268200900016C, HHSN268200900017C, HHSN268200900018C, HHSN268200900019C, and HHSN268200900020C) and grants from the NIH/NHLBI (grants U01 HL137880 and U24 HL141762) and supplemented through contributions made to the Foundation for the NIH and the COPD Foundation by AstraZeneca, MedImmune, Bayer Corporation (Whippany, New Jersey), Bellerophon Therapeutics (Warren, New Jersey), Boehringer-Ingelheim, Chiesi Farmaceutici S.p.A. (Parma, Italia), the Forest Research Institute, Inc. (Jersey City, New Jersey), GlaxoSmithKline, Grifols Therapeutics, Inc. (Research Triangle Park, North Carolina), Ikaria, Inc. (Hampton, New Jersey), Novartis, Nycomed Pharma GmbH (Zurich, Switzerland), ProterixBio, Inc. (Billerica, Massachusetts), Regeneron, Sanofi, Sunovion, Takeda Pharmaceutical Company (Tokyo, Japan), Theravance Biopharma, Inc. (South San Francisco, California), and Mylan N.V. (White Sulphur Springs, West Virginia). SHS The Strong Heart Study has been funded in whole or in part with federal funds from the National Heart, Lung, and Blood Institute, National Institute of Health, Department of Health and Human Services, under contract numbers 75N92019D00027, 75N92019D00028, 75N92019D00029, & 75N92019D00030. The study was previously supported by research grants: R01HL109315, R01HL109301, R01HL109284, R01HL109282, and R01HL109319 and by cooperative agreements: U01HL41642, U01HL41652, U01HL41654, U01HL65520, and U01HL65521. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Indian Health Service (IHS). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: A Krishnaswamy left Columbia University and joined Merck (Epidemiology) as a project manager following completion of her work on this paper. Mitchell Elkind receives royalties from UpToDate for a chapter on neurological complications of COVID-19; receives study drug in kind from the BMS-Pfizer Alliance for Eliquis and ancillary funding from Roche, both for an NIH-funded trial of stroke prevention. E B Levitan receives research funding from Amgen (to UAB) unrelated to the current work and personal fees for serving on a DSMB from University of Pittsburgh for an NIH funded trial. David Schwartz is founder and chief scientific officer of Eleven P15, a company dedicated to the diagnosis, prevention, and treatment of early presentations of pulmonary fibrosis. Joyce Lee serves as a consultant for Eleven P15. Sally Wenzel receives funding for consulting and clinical trials from AstraZeneca, GSK, Sanofi-Genzyme, Novartis, Knopp; she also receives research support from Pieris and Regeneron. PG Woodruff has served as a consultant for Roche, Sanofi, Amegne and Astra Zeneca outside the context of this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Coronavirus disease 2019 (COVID-19) has been a leading cause of hospitalizations and deaths since 2020 [1, 2]. Large population-based studies with accurate characterization of hospitalized COVID-19 and its acute complications–as well as clinical, biomarker, and lifestyle data from before and after infection–are urgently needed to understand mechanisms of acute disease and post-acute sequelae of COVID-19 (PASC), which impacts up to 57% of hospitalized COVID-19 patients [3].
Many extant US cohort studies have collected extensive data on clinical and subclinical diseases and their risk factors, including behavior, cognition, biomarkers, and social determinants of health. Since enrollment for these studies was completed prior to the COVID-19 pandemic, they offer a unique opportunity to study risk factors for incident COVID-19 while minimizing the referral, survival, and recall biases that are common to COVID-19 case series and disease-based studies. However, in the absence of a US national healthcare system, most longstanding US epidemiology studies are not able to establish comprehensive linkages to electronic health records (EHRs) and have depended on medical records ascertainment and adjudication for cardiovascular and respiratory health outcomes. A protocol to define hospitalizations for COVID-19 among epidemiology cohorts was lacking.
The Collaborative Cohort of Cohorts for COVID-19 Research (C4R) study is a unique meta-cohort of NIH-funded prospective, observational epidemiology cohort studies that was funded to perform standardized, prospective ascertainment of COVID-19, including physician adjudication of medical records for COVID-19 hospitalizations and deaths (hereafter, “events”). The primary purpose of this report is to describe the C4R events adjudication protocol and preliminary experience applying the protocol from February 2021 to June 2023. This report includes a comparison of ICD-based versus physician adjudicated protocol-based classification for COVID-related fatal and non-fatal hospitalizations.
Materials and methods
Study design
C4R enrolled adult participants from 14 long-standing cardiovascular, neurological, and respiratory cohorts [4]: Atherosclerosis Risk in Communities (ARIC) [5, 6], Coronary Artery Risk Development in Young Adults (CARDIA) [7], Genetic Epidemiology of COPD (COPDGene) [8], Framingham Heart Study (FHS) [9], Hispanic Community Health Study/Study of Latinos (HCHS/SOL) [10–13], Jackson Heart Study (JHS) [14–16], Mediators of Atherosclerosis in South Asians Living in America (MASALA) [17, 18], Multi-Ethnic Study of Atherosclerosis (MESA) [19], Northern Manhattan Study (NOMAS) [20], Prevent Pulmonary Fibrosis (PrePF) [21], REeasons for Geographic and Racial Differences in Stroke (REGARDS) [22], Severe Asthma Research Program (SARP) [23], Subpopulations and Intermediate Outcome Measures in COPD Study (SPIROMICS) [24], and the Strong Heart Study (SHS) [25, 26]. Details on each of the component cohorts are provided in S1 Appendix and S1 Table. Cohort participants previously consented to in-person, telephone, and/or e-mail contact and for abstraction of medical records.
C4R received funding in 2020 to perform standardized prospective data collection on COVID-19 and to harmonize pre-pandemic deep phenotyping available in the cohorts. Columbia University developed the standard questionnaires and protocols and served as the Data Coordination and Harmonization Center (DCHC) for C4R (Columbia University Institutional Review Board, AAAT3035). Of note, in some cohorts, COVID-19 questionnaires were initiated before the development of a standard C4R questionnaire; these questionnaires were later harmonized with the C4R questionnaire and classified as C4R questionnaires.
As previously reported [4], following a cohort ancillary studies model, researchers in each cohort study were directly responsible for accomplishing data collection in accordance with the standard protocols and under the supervision of their own observational studies monitoring board, steering committee, institutional review board (IRB), and any other applicable regulatory authorities. Columbia University served as the Data Coordination and Harmonization Center (DCHC) for C4R (Columbia University Institutional Review Board, AAAT3035). A full list of cohort IRBs supervising implementation of the C4R protocols is provided in S2 Table.
Following cohort-specific IRB approval and consent processes (including verbal, remote, and traditional written informed consent), adult participants in the pre-existing cohort studies were enrolled into the C4R ancillary study on a rolling basis from April 9, 2020, to February 28, 2023. COVID-19 data collection was accomplished by two waves of questionnaires (Wave 1: April 2020–May 2022; Wave 2: February 2021–February 2023), SARS-CoV-2 serosurvey (February 2021–February 2023), and ascertainment and adjudication of COVID-19 events, which are the subject of this report.
For the purposes of the work presented in this manuscript, data were accessed at Columbia University from July 1, 2021, to October 1, 2023.
Protocol development and coordination
The C4R events protocol is included in S2 Appendix. The protocol was developed by investigators and cohort personnel with experience in clinical events ascertainment [5, 7, 9, 11, 12, 17, 19, 20, 26–33] and approved by the C4R Cohort Coordinating Committee (CCC) in February 2021. Relevant administrative and review forms were coded into the C4R Events REDCap [34, 35] toolkit, which was made available to the cohorts via a central (Columbia) instance or for local adaptation by cohort data coordinating centers (DCCs). For the central instance, cohort-specific Data Access Groups (DAGs) ensured that each cohort was only able to enter or access its own participant data. Cohorts provided bimonthly tracking reports regarding cohort-specific elements of protocol completion to the DCHC, which integrated these reports and generated status reports to the Events Subcommittee, the CCC, and funding agencies. De-identified COVID-19 events data were made available for analysis on the C4R Analysis Commons and shared with cohort-specific DCCs for cohort use and transfer to other data sharing platforms.
COVID-19 events ascertainment
Potential COVID-19 hospitalizations and deaths were ascertained by cohorts via several mechanisms. C4R questionnaires, which were administered in two waves across all 14 cohorts, included questions regarding hospitalization for COVID-19 (S3 Table). If necessary (e.g., in cases of participant dementia or death), questionnaires were administered to a participant’s proxy. Additional COVID-19-related hospitalizations and deaths were identified by regular non-C4R follow-up calls conducted in 9 cohorts (ARIC, CARDIA, FHS, HCHS/SOL, JHS, MESA, NOMAS, REGARDS, SHS) to collect information on all-cause hospitalization and vital status. These data were supplemented with information collected at in-person exams, which were conducted in all cohorts during the pandemic period, except for NOMAS and REGARDS. Various non-questionnaire ascertainment methods, such as active surveillance of local EHR systems and other sources (e.g., obituaries), were performed by ARIC, FHS, JHS, SARP, SHS, and selected clinical sites in CARDIA, COPDGene, and MESA. Most cohorts supplemented vital status data with National Death Index (NDI) searches, although these are subject to reporting lags and were not available for the pandemic period at the time of this report.
After confirming participant/proxy consent, cohort staff requested copies of medical records for potential COVID-19-related hospitalizations and deaths, including physician notes (admission, consultation, discharge), radiology and laboratory reports, electrocardiogram reports, and discharge diagnoses (including discharge diagnosis ICD codes). Where applicable, death certificates were obtained. Medical records submitted for central review at the DCHC were de-identified prior to secure file transfer.
Adjudication
Eligibility for adjudication.
Hospitalizations (fatal and non-fatal) and out-of-hospital deaths were eligible for protocolized medical record review if they were assigned, in any position, any of the following COVID-19 discharge diagnoses based on the International Classification of Diseases, Tenth Revision (ICD-10), Clinical Modification codes [36]: confirmed COVID-19 (U07.1), post-infectious state after COVID-19 (U09.9), Multisystem inflammatory syndrome associated with COVID-19 (M35.81), Personal History of COVID-19 (Z86.16), Pneumonia due to coronavirus disease 2019 (J12.82), Other viral pneumonia (J12.89), or, for events occurring prior to May 1 2020, other coronavirus (B97.29). In the absence of these discharge diagnoses, records were considered eligible for review if there was evidence of a positive COVID-19 test, physician suspicion of COVID-19, or next-of-kin interview indicating suspected or known COVID-19 infection. Episodes of treatment in the Emergency Department (ED) for >24 hours were classified as hospitalizations since many hospitals used the ED for inpatient care during the heights of COVID-19 surges.
Adjudication process.
Eligible COVID-19-related hospitalizations (fatal and non-fatal) and out-of-hospital deaths were subjected to adjudication by physicians and/or clinical nurse practitioners with experience in evaluating hospitalized COVID-19. Reviewers were trained via webinar and individualized instruction. Three cohorts (FHS, REGARDS, SARP) elected to perform review by local adjudicators; records from the remaining cohorts were reviewed centrally at the DCHC. Adjudication entailed data abstraction, including information on oxygenation levels and medication administration, followed by classification of COVID-19 outcomes. All COVID-19 outcomes were defined as definite, probable, or absent, based on specific criteria, including symptoms and test results, and adjudicator judgment (S4 Table).
Adjudication of COVID-19 infection.
Adjudication of Definite COVID-19 Infection required evidence of a positive SARS-CoV-2 test. Per protocol, an administrative criterion (i.e., assignment of ICD U07.1) was sufficient to classify Definite COVID-19 infection, since this diagnosis code specifically requires positive testing for SARS-CoV-2; of note, for all other outcomes, administrative criteria were insufficient for definite classification but could be used to justify a probable classification. Definite COVID-19 infection was necessary to define any other COVID-19 outcome as definite. Adjudication of Probable COVID-19 Infection required anticipated signs and symptoms of COVID-19 without confirmatory testing. In the absence of adjudicated Definite or Probable COVID-19 infection, no additional outcomes were adjudicated.
Adjudication of COVID-19 hospitalization.
In the context of Definite or Probable SARS-CoV-2 infection, hospitalization “due to” COVID-19 was defined as being admitted to the hospital for COVID-19-related signs or symptoms; developing COVID-19-related signs or symptoms during hospitalization; or, death in the ED. Whereas, hospitalization “with” COVID-19 was defined as being diagnosed with SARS-CoV-2 infection, yet being admitted to the hospital for a reason other than COVID-19 related signs or symptoms, without developing COVID-19 signs or symptoms during hospitalization.
Adjudication of COVID-19 severity.
Definitions of severe and critical disease were based on NIH COVID-19 treatment guidelines [37].
Adjudication of COVID-19 complications.
While treating physician notes and administrative criteria could be used for Probable classification of COVID-19 complications, Definite classification of COVID-19 pneumonia, pulmonary embolus (PE), deep vein thrombosis (DVT), or stroke required radiologic evidence in the medical record. Definite classification of acute kidney injury (AKI) was based on reported blood creatinine levels or initiation of renal replacement therapy. Definite COVID-19 myocardial infarction (MI) was defined based on biomarker and electrocardiographic or pathologic criteria to identify cases of MI caused by acute atherothrombotic coronary artery disease, or “Type 1” MI, in the context of COVID-19 infection [38]. Probable COVID-19 MI was defined more broadly and likely captures Type 1 and Type 2 MI. Myocardial injury was defined as a maximum recorded troponin level that was greater than two times the upper limit of normal (ULN), with or without associated ischemic symptoms.
Re-adjudication.
Following completion of adjudication, reviewers could request a second independent adjudication for challenging cases. A random 10% subset of records was also submitted for a second independent adjudication.
Analysis.
Characteristics of C4R participants, with and without an ascertained or adjudicated COVID-19 event, were tabulated. The incidence of events ascertained and adjudicated per month (based on date of event) was plotted. Since we found that records relating to out-of-hospital deaths (e.g., death certificates) did not contain sufficient data for adjudication, the following analyses were limited to adjudicated fatal and non-fatal COVID-19 hospitalizations among participants who consented to data sharing on the C4R Analysis Commons. The incidence of COVID-19-related outcomes was tabulated, by level of certainty. Interrater agreement was assessed via positive agreement, negative agreement, and the Cohen’s κ-statistic [39]. The performance of discharge diagnosis ICD code-based classification was compared against definite C4R classifications, which were treated as the reference standards; probable classifications were excluded from these comparisons because they included discharge diagnosis ICD codes in the diagnostic criteria. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were calculated. Also, myocardial injury was compared to adjudicated classifications of definite COVID-19 MI. Analyses were performed in SAS Studio software (SAS Institute, Cary, NC) on the C4R Analysis Commons.
Results
COVID-19 events ascertainment
Among 49,790 C4R participants, 1,974 potential COVID-19-related hospitalization and/or deaths among 1,772 (3.6%) participants were ascertained between January 2020 and June 2023 (Fig 1), for an estimated incidence density rate of 11.9 per 1,000 person-years of follow-up. Overall, the cohorts reported that half (53%) of these events were identified by a C4R questionnaire. As of June 2023, 1,768 (90%) medical records were requested, of which 1,523 (86%) were obtained. The most common reasons for failure to obtain medical records were inability to obtain participant consent and lack of response from the hospital.
Incidence is calculated per month and based on date of event.
Following 3 years of protocol funding (October 1, 2020—October 1, 2023), 1,237 of 1,974 (63%) events were adjudicated over 28 months. After exclusion of 57 out-of-hospital deaths due to incomplete information and 13 non-fatal hospitalizations with consent restrictions, there were 1,167 (59% of the 1,974) events available for analysis, of which 1,030 had evidence of COVID-19 (88% of 1,167 events available for analysis) (Fig 2).
Table 1 compares socio-demographic and clinical characteristics of 1,772 participants among whom a potential COVID-19 hospitalization was ascertained, including those for whom events were (N = 1098) and were not yet (N = 674) adjudicated as of October 2023, versus those with no COVID-19 hospitalization or death ascertained through June 2023 (N = 48,018). Compared to participants without a potential COVID-19 hospitalization or death, those with an ascertained event were more likely to be male, to report American Indian ancestry, and to have a history of pre-pandemic smoking, diabetes, or hypertension; they were less likely to have education beyond college and less likely to be vaccinated compared to those without ascertained events. Notably, among participants with ascertained potential COVID-19 events, characteristics of those with adjudicated records were generally similar to those pending adjudication. More than one COVID-19 hospitalization was ascertained in 50 (3%) participants.
Adjudication
Adjudication process.
There were 7 reviewers at the C4R DCHC, 5 at FHS, 3 at REGARDS, and 2 at SARP. The median length of medical records was 59 pages (IQR: 37, 105). The average time to review was approximately 30 minutes per record.
Adjudicated outcomes.
Definite SARS-CoV-2 infection was adjudicated in 87% of 1,167 events, whereas probable infection was adjudicated in only 1%. Among 1,030 adjudicated hospitalized events in which SARS-CoV-2 infection was definite or probable, COVID-19 illness was diagnosed as the cause of 952 (93%) of the hospitalizations. Table 2 describes the adjudicated probable and definite diagnoses of SARS-CoV-2 infection, acute COVID-19 illness severity, and acute COVID-19 complications following the criteria described in S3 Table: 77% were adjudicated as severe COVID-19, 31% as critical COVID-19, 80% had adjudicated COVID-19-associated pneumonia, and 34% had adjudicated COVID-19-associated AKI. Adjudicated fatal hospitalization due to COVID-19 and other cardiopulmonary complications such as PE, DVT, and MI were less common.
Adjudicator agreement.
Of 139/1237 (11%) hospitalizations that underwent a second adjudication, there was almost perfect agreement for adjudication of COVID-19 infection, critical illness, stroke, PE, DVT, and fatal hospitalization (Table 3). There was also strong agreement for hospitalization, severe illness, pneumonia, and renal failure, and moderate agreement for MI.
Comparison of adjudication vs. ICD codes.
Compared to adjudicated diagnoses with a definite certainty level, discharge ICD code-based classification was sensitive and specific for pneumonia (sensitivity = 84%, specificity = 90%), but less sensitive (57–81%) for cardiovascular and renal complications (Table 4). The PPV of ICD-based classification for COVID-19-related pneumonia (J12.82, pneumonia due to coronavirus disease 2019, J12.89, other viral pneumonia), AKI (N17, Acute kidney injury), PE (ICD Code I26, PE), DVT (I82, DVT) and stroke (I63, Cerebral infarction) was excellent, ranging from 81–97%; however, the PPV for COVID-19-related MI (I21, Acute myocardial infarction) was 33%.
Comparison of adjudicated MI vs. myocardial injury.
A substantial number of participants with adjudicated events had myocardial injury that was not adjudicated as COVID-19-related MI. Among the 1,030 hospitalized events in which SARS-CoV-2 infection was adjudicated as definite or probable, 170 (25%) demonstrated myocardial injury. Of these 170 events with myocardial injury, 25 (15%) were assigned an ICD code for MI, 5 (3%) were adjudicated as definite COVID-19-related MI, and 38 (22%) were adjudicated as probable COVID-19-related MI; the majority (75%) were not adjudicated as definite or probable COVID-19-related MI. Of note, these groups are not directly comparable with the groups described in Table 4, which only included events with ICD codes available and excluded events with probable MI diagnoses.
Discussion
Protocolized adjudication confirmed four out of five hospitalizations for COVID-19 in a US meta-cohort of prospective epidemiology cohorts and adjudicated cases of pneumonia, PE, and other conditions that were not indicated by discharge diagnosis codes. These results illustrate the importance of systematic medical record review for robust classification of COVID-19 outcomes in epidemiology cohort studies, which provide unique opportunities to study antecedent risk factors for acute COVID-19 and post-COVID conditions.
Over three years, C4R ascertained at least one potential COVID-19-related hospitalization or out-of-hospital death in 3.6% of participants, equivalent to an incidence per 1,000 person-years of follow-up of 11.9. For comparison, in 8 of the C4R cohorts, the incidence per 1,000 person-years for atherosclerotic cardiovascular disease events has been estimated at 9.7 [40]. Hence, the results of C4R events ascertainment to date highlight the major impact of COVID-19 on cohort participants and provide a substantial, longitudinal dataset to support well-powered analyses of risk factors and sequelae.
Nonetheless, our experience highlights certain limitations of standard cohort events surveillance for ascertainment of COVID-19-related events. Of hospitalizations that were ascertained as potentially COVID-19-related, SARS-CoV-2 infection could not be confirmed in one in eight events. These cases could have been due to incorrect self-reporting of a COVID-19 hospitalization on a C4R or cohort questionnaire, or active surveillance methods that were more sensitive than specific. Of note, some cases where infection could not be confirmed may have been true infections, but the medical records lacked sufficient detail for adjudication. Hence, rather than censoring all non-confirmed events, C4R is assigning a certainty level for all its COVID-19 outcomes, so that investigators can select the outcome most suitable for their specific research needs.
Our findings also demonstrate the limitations of discharge diagnosis codes to define hospitalization “for” versus “with” SARS-CoV-2 infection. Of hospitalizations with confirmed SARS-CoV-2 infection, symptoms of COVID-19 illness did not contribute to the hospitalization in 7% of cases. These findings are similar to prior reports examining the role of SARS-CoV-2 in hospitalizations using EHR systems. Adjudicated C4R outcomes will allow investigators to exclude these cases of hospitalization with incidental COVID-19 from studies designed to examine risks and sequelae of severe COVID-19 illness.
Furthermore, we found that discharge diagnosis codes were insensitive for cardiopulmonary and renal complications. We adjudicated a substantial number of cases of COVID-19 pneumonia, AKI, and MI among records without a corresponding discharge diagnosis code. This may be related, in part, to the major challenges of charting during COVID-19 surges, when documentation standards were modified to prioritize direct patient care activities. Nonetheless, before the pandemic period, previous investigations on the use of administrative data to identify cases of pneumonia have found that ICD codes are imprecise and can result in a substantial number of pneumonia cases going undetected [41, 42]. These results are supported by earlier findings that ICD-10 code N17 often misses AKI during hospitalizations for kidney transplant patients [43]. Potential misestimation of the incidence of MI using claims or administrative data has also been well-described [44–47]. This has been one justification for longstanding—albeit labor- and time-intensive—ASCVD events adjudication programs in many of the C4R cohorts.
As expected, we found that a substantial proportion of hospitalized events included evidence of myocardial injury, defined by troponin values two times greater than the upper limit of normal; however, only a small subset of these cases was adjudicated as COVID-19 MI. We elected to restrict definite MI to confirmed cases of type 1 (ST-elevation MI) or type 3 MI (MI resulting in death with pathological evidence of MI). It may be difficult to differentiate type 2 MI/supply-demand mismatch, which requires documentation of a rise and fall in troponin levels and symptoms of ischemia, from myocardial injury (at least one elevated troponin level), based solely on review of obtained medical records. Prior studies demonstrated that elevated troponin values are common in patients hospitalized with COVID-19 [48–50], and may occur with conditions other than MI, such as cardiomyopathy, acute cor pulmonale, arrhythmias, or cardiogenic shock [51]. Our results suggest that the true incidence of type 1 MI, due to plaque disruption and thrombosis leading to coronary occlusion, was rare in the context of acute COVID-19, supporting the utility of protocolized adjudication in validating these outcomes for COVID-19 cardiovascular research.
Altogether, our findings have several implications for EHR-based studies that do not adjudicate endpoints. Our results suggest that ICD-based events definitions could overestimate the number of cases of hospitalization due to COVID-19 illness and underestimate COVID-related complications. In addition to incorrectly estimating the prevalence and incidence of COVID-related outcomes, the observed measurement errors could reduce the robustness and reproducibility of epidemiologic analyses. If ICD misclassification was nondifferential according to risk factors of interest, it would reduce precision and increase the risk of type 2 error; if misclassification was associated with a risk factor of interest, this could potentially bias results away from the null hypothesis. Of note, EHR-based studies often use complex algorithms to define clinical outcomes and do not rely on ICD codes alone. Nonetheless, our findings underscore the importance of validating outcome definitions via comparison with robust approaches such as protocolized events adjudication.
Strengths and limitations
Strengths of this study include prospective ascertainment of potential COVID-19-related hospitalizations and deaths in a well-characterized, multi-ethnic, US community-based sample of adults that is relatively free of referral, survival, and recall biases compared with studies that only include hospitalized patients. The adjudication protocol, which was modeled on gold-standard epidemiology cohort events adjudication in many of the C4R cohorts, was fully standardized across the cohorts and implemented at scale to generate robust events data for analysis within three years of program initiation.
Nonetheless, certain limitations must be considered, in addition to those noted above. Medical records have not yet been obtained for 37% of cases due to lack of participant consent, lack of response from hospital systems, or other operational delays that are common to cohort events ascertainment operations. This highlights the need for novel approaches for cohorts to access medical records, such as the consenting of participants to share their own electronic medical records—now available to patients via the 21st Century Cures Act [52]—with cohorts [52]. The characteristics of participants with missing versus available medical records in C4R were comparable, so there was no clear evidence of selection bias. Some medical records had incomplete data available for adjudication, which could be related to relaxation of documentation requirements during the pandemic. Few out-of-hospital deaths were ascertained, and medical records for out-of-hospital deaths were often incomplete, due to various reasons, including lack of court documents from the family of the deceased to obtain records, or hospital decision to not grant access to the decedent’s medical records for research purposes. To address this limitation, additional information on out-of-hospital deaths associated with COVID-19 will be obtained from the NDI, which provides complete ascertainment of deaths in the US, including ICD-codes; unfortunately, there are substantial time lags inherent in NDI reporting.
Conclusions
Ascertainment and adjudication of COVID-19 hospitalizations and deaths in longstanding NIH-funded cohort studies were feasible, albeit time- and resource-intensive, and our results illustrate the importance of systematic medical records adjudication for robust classification of COVID-19 events. Adjudication confirmed SARS-CoV-2 infection in 88% of ascertained events and found that infection may have been incidental to the hospitalization in 7% of the cases. Compared to adjudication, discharge diagnosis codes were insensitive for acute cardiovascular and renal complications of COVID-19. Novel approaches to expedite medical records access and linkage would augment unique opportunities for COVID-19 and other health outcomes research, particularly for emerging diseases, in NIH-funded epidemiology cohorts.
Supporting information
S1 Table. Characteristics of participants in C4R cohorts, United States, March 1, 2020.
https://doi.org/10.1371/journal.pone.0316198.s003
(DOCX)
S2 Table. Cohort Institutional Review Boards (IRBs) supervising implementation of the C4R protocols.
https://doi.org/10.1371/journal.pone.0316198.s004
(DOCX)
S3 Table. Selected COVID-19 questionnaire elements and their inclusion by cohorts in questionnaires for C4R, United States, April 2020–February 2023.
https://doi.org/10.1371/journal.pone.0316198.s005
(DOCX)
S4 Table. Criteria for classification of COVID-19 diagnoses as definite or probable based on medical record review.
https://doi.org/10.1371/journal.pone.0316198.s006
(DOCX)
Acknowledgments
We thank the participants of each cohort for their dedication to the studies. The authors thank the staff and participants of the ARIC study for their important contributions.
The authors thank the staff and participants of the CARDIA study for their important contributions. Framingham Heart Study wished to acknowledge the study participants and staff members who supported the FHS-C4R effort: Cathy D’Augustine, Lindsay Clayson, Karen Mutalik, and Ken Nieto. We appreciate the support and guidance of investigators from the Study Design and Molecular Epidemiology Core of the Vermont Center for Cardiovascular and Brain Health. The authors thank the other investigators, the staff, and the participants of the REGARDS study for their valuable contributions. In particular, the REGARDS would like to thank Karen Marshall, Melissa Garner, J. David Rhodes, April Sisson, Todd M. Brown, and Parag Goyal. A full list of participating REGARDS investigators and institutions can be found at: https://www.uab.edu/soph/regardsstudy/. The authors thank the UNC CSCC, including Lisa Reeves and Kwanhye Jung for their many contributions. The authors express gratitude to the reviewers and abstractors involved in the morbidity and mortality surveillance, as well as to the other study investigators, the dedicated study staff, and the participants of the Strong Heart Study for their significant contributions. The authors thank Milagros Ventura and Greg Neils of the Columbia Data Coordinating Center for management of secure data transmission and development of the REDCap system to capture adjudication data. The authors thank Consuelo Mora McLaughlin and Janet T De Rosa from Columbia University for their work on NOMAS events ascertainment.
References
- 1.
Leading Causes of Death. National Center for Health Statistics, Centers for Disease Control and Prevention
- 2. Shiels MS, Haque AT, Berrington de González A, Freedman ND. Leading Causes of Death in the US During the COVID-19 Pandemic, March 2020 to October 2021. JAMA Internal Medicine. 2022;182(8):883–6. pmid:35788262
- 3. Chen C, Haupert SR, Zimmermann L, Shi X, Fritsche LG, Mukherjee B. Global Prevalence of Post-Coronavirus Disease 2019 (COVID-19) Condition or Long COVID: A Meta-Analysis and Systematic Review The Journal of Infectious Diseases. 2022;226(9):1593–607. pmid:35429399
- 4. Oelsner EC, Krishnaswamy A, Balte PP, Allen NB, Ali T, Anugu P, et al. Collaborative Cohort of Cohorts for COVID-19 Research (C4R) Study: Study Design. American Journal of Epidemiology. 2022;191(7):1153–73. pmid:35279711
- 5. The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. The ARIC investigators. Am J Epidemiol. 1989;129(4):687–702. Epub 1989/04/01. pmid:2646917.
- 6. Wright JD, Folsom AR, Coresh J, Sharrett AR, Couper D, Wagenknecht LE, et al. The ARIC (Atherosclerosis Risk In Communities) Study: JACC Focus Seminar 3/8. J Am Coll Cardiol. 2021;77(23):2939–59. pmid:34112321.
- 7. Friedman GD, Cutter GR, Donahue RP, Hughes GH, Hulley SB, Jacobs DR Jr., et al. CARDIA: study design, recruitment, and some characteristics of the examined subjects. J Clin Epidemiol. 1988;41(11):1105–16. Epub 1988/01/01. pmid:3204420.
- 8. Regan EA, Hokanson JE, Murphy JR, Make B, Lynch DA, Beaty TH, et al. Genetic epidemiology of COPD (COPDGene) study design. COPD. 2010;7(1):32–43. Epub 2010/03/11. pmid:20214461.
- 9. Tsao CW, Vasan RS. Cohort Profile: The Framingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology. Int J Epidemiol. 2015;44(6):1800–13. Epub 2015/12/26. pmid:26705418.
- 10. Daviglus ML, Talavera GA, Aviles-Santa ML, Allison M, Cai J, Criqui MH, et al. Prevalence of major cardiovascular risk factors and cardiovascular diseases among Hispanic/Latino individuals of diverse backgrounds in the United States. JAMA. 2012;308(17):1775–84. Epub 2012/11/03. pmid:23117778.
- 11. Lavange LM, Kalsbeek WD, Sorlie PD, Aviles-Santa LM, Kaplan RC, Barnhart J, et al. Sample design and cohort selection in the Hispanic Community Health Study/Study of Latinos. Ann Epidemiol. 2010;20(8):642–9. Epub 2010/07/09. pmid:20609344.
- 12. Sorlie PD, Aviles-Santa LM, Wassertheil-Smoller S, Kaplan RC, Daviglus ML, Giachello AL, et al. Design and implementation of the Hispanic Community Health Study/Study of Latinos. Ann Epidemiol. 2010;20(8):629–41. Epub 2010/07/09. pmid:20609343.
- 13. Pirzada A, Cai J, Heiss G, Sotres-Alvarez D, Gallo LC, Youngblood ME, et al. Evolving Science on Cardiovascular Disease Among Hispanic/Latino Adults: JACC International. J Am Coll Cardiol. 2023;81(15):1505–20. pmid:37045521.
- 14. Carpenter MA, Crow R, Steffes M, Rock W, Heilbraun J, Evans G, et al. Laboratory, reading center, and coordinating center data management methods in the Jackson Heart Study. Am J Med Sci. 2004;328(3):131–44. Epub 2004/09/16. pmid:15367870.
- 15. Keku E, Rosamond W, Taylor HA Jr., Garrison R, Wyatt SB, Richard M, et al. Cardiovascular disease event classification in the Jackson Heart Study: methods and procedures. Ethn Dis. 2005;15(4 Suppl 6):S6-62-70. Epub 2005/12/02. pmid:16317987.
- 16. Taylor HA Jr., Wilson JG, Jones DW, Sarpong DF, Srinivasan A, Garrison RJ, et al. Toward resolution of cardiovascular health disparities in African Americans: design and methods of the Jackson Heart Study. Ethn Dis. 2005;15(4 Suppl 6):S6-4-17. Epub 2005/12/02. pmid:16320381.
- 17. Kanaya AM, Kandula N, Herrington D, Budoff MJ, Hulley S, Vittinghoff E, et al. Mediators of Atherosclerosis in South Asians Living in America (MASALA) study: objectives, methods, and cohort description. Clin Cardiol. 2013;36(12):713–20. Epub 2013/11/07. pmid:24194499.
- 18. Kanaya AM, Chang A, Schembri M, Puri-Taneja A, Srivastava S, Dave SS, et al. Recruitment and retention of US South Asians for an epidemiologic cohort: Experience from the MASALA study. J Clin Transl Sci. 2019;3(2–3):97–104. Epub 2019/10/30. pmid:31660232.
- 19. Bild DE, Bluemke DA, Burke GL, Detrano R, Diez Roux AV, Folsom AR, et al. Multi-Ethnic Study of Atherosclerosis: objectives and design. Am J Epidemiol. 2002;156(9):871–81. Epub 2002/10/25. pmid:12397006.
- 20. Sacco RL, Boden-Albala B, Gan R, Chen X, Kargman DE, Shea S, et al. Stroke incidence among white, black, and Hispanic residents of an urban community: the Northern Manhattan Stroke Study. Am J Epidemiol. 1998;147(3):259–68. Epub 1998/03/03. pmid:9482500.
- 21. Mathai SK, Humphries S, Kropski JA, Blackwell TS, Powers J, Walts AD, et al. MUC5B variant is associated with visually and quantitatively detected preclinical pulmonary fibrosis. Thorax. 2019;74(12):1131–9. Epub 2019/09/29. pmid:31558622.
- 22. Howard VJ, Cushman M, Pulley L, Gomez CR, Go RC, Prineas RJ, et al. The reasons for geographic and racial differences in stroke study: objectives and design. Neuroepidemiology. 2005;25(3):135–43. Epub 2005/07/02. pmid:15990444.
- 23. Teague WG, Phillips BR, Fahy JV, Wenzel SE, Fitzpatrick AM, Moore WC, et al. Baseline Features of the Severe Asthma Research Program (SARP III) Cohort: Differences with Age. J Allergy Clin Immunol Pract. 2018;6(2):545–54.e4. Epub 2017/09/04. pmid:28866107.
- 24. Couper D, LaVange LM, Han M, Barr RG, Bleecker E, Hoffman EA, et al. Design of the Subpopulations and Intermediate Outcomes in COPD Study (SPIROMICS). Thorax. 2014;69(5):491–4. Epub 2013/09/14. pmid:24029743.
- 25. Lee ET, Welty TK, Fabsitz R, Cowan LD, Le NA, Oopik AJ, et al. The Strong Heart Study. A study of cardiovascular disease in American Indians: design and methods. Am J Epidemiol. 1990;132(6):1141–55. Epub 1990/12/01. pmid:2260546.
- 26. North KE, Howard BV, Welty TK, Best LG, Lee ET, Yeh JL, et al. Genetic and environmental contributions to cardiovascular disease risk in American Indians: the strong heart family study. Am J Epidemiol. 2003;157(4):303–14. Epub 2003/02/13. pmid:12578801.
- 27. Wyatt SB, Diekelmann N, Henderson F, Andrew ME, Billingsley G, Felder SH, et al. A community-driven model of research participation: the Jackson Heart Study Participant Recruitment and Retention Study. Ethn Dis. 2003;13(4):438–55. Epub 2003/11/25. pmid:14632263.
- 28. Oelsner EC, Loehr LR, Henderson AG, Donohue KM, Enright PL, Kalhan R, et al. Classifying Chronic Lower Respiratory Disease Events in Epidemiologic Cohort Studies. Ann Am Thorac Soc. 2016;13(7):1057–66. Epub 2016/04/19. pmid:27088163.
- 29. Zakai NA, McClure LA, Judd SE, Safford MM, Folsom AR, Lutsey PL, et al. Racial and regional differences in venous thromboembolism in the United States in 3 cohorts. Circulation. 2014;129(14):1502–9. Epub 2014/02/11. pmid:24508826.
- 30. Howard VJ, Kleindorfer DO, Judd SE, McClure LA, Safford MM, Rhodes JD, et al. Disparities in stroke incidence contributing to disparities in stroke mortality. Ann Neurol. 2011;69(4):619–27. Epub 20110317. pmid:21416498.
- 31. Safford MM, Brown TM, Muntner PM, Durant RW, Glasser S, Halanych JH, et al. Association of race and sex with risk of incident acute coronary heart disease events. JAMA. 2012;308(17):1768–74. pmid:23117777.
- 32. Cushman M, Tsai AW, White RH, Heckbert SR, Rosamond WD, Enright P, et al. Deep vein thrombosis and pulmonary embolism in two cohorts: the longitudinal investigation of thromboembolism etiology. Am J Med. 2004;117(1):19–25. pmid:15210384.
- 33. Carr JJ, Jacobs DR Jr., Terry JG, Shay CM, Sidney S, Liu K, et al. Association of Coronary Artery Calcium in Adults Aged 32 to 46 Years With Incident Coronary Heart Disease and Death. JAMA Cardiol. 2017;2(4):391–9. pmid:28196265.
- 34. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. Epub 2019/05/13. pmid:31078660.
- 35. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. Epub 2008/10/22. pmid:18929686.
- 36.
Prevention CfDCa. ICD-10-CM Official Coding and Reporting Guidelines April 1, 2020 through September 30, 2020 2020 (Hyattsville, MD: National Center for Health Statistics). Epub https://www.cdc.gov/nchs/data/icd/COVID-19-guidelines-final.pdf.
- 37.
COVID-19 Treatment Guidelines Panel. Coronavirus Disease 2019 (COVID-19) Treatment Guidelines. National Institutes of Health. https://www.covid19treatmentguidelines.nih.gov/. Accessed October 27, 2023.
- 38. Thygesen K, Alpert JS, Jaffe AS, Chaitman BR, Bax JJ, Morrow DA, et al. Fourth Universal Definition of Myocardial Infarction (2018). J Am Coll Cardiol. 2018;72(18):2231–64. Epub 2018/08/30. pmid:30153967.
- 39. Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol. 2009;9:5. Epub 2009/01/23. pmid:19159474.
- 40. Khera R, Pandey A, Ayers CR, Carnethon MR, Greenland P, Ndumele CE, et al. Performance of the Pooled Cohort Equations to Estimate Atherosclerotic Cardiovascular Disease Risk by Body Mass Index. JAMA Network Open. 2020;3(10):e2023242–e. pmid:33119108
- 41. Aronsky D, Haug PJ, Lagor C, Dean NC. Accuracy of administrative data for identifying patients with pneumonia. Am J Med Qual. 2005;20(6):319–28. Epub 2005/11/11. pmid:16280395.
- 42. van de Garde EM, Oosterheert JJ, Bonten M, Kaplan RC, Leufkens HG. International classification of diseases codes showed modest sensitivity for detecting community-acquired pneumonia. J Clin Epidemiol. 2007;60(8):834–8. Epub 2007/07/04. pmid:17606180.
- 43. Molnar AO, van Walraven C, McArthur E, Fergusson D, Garg AX, Knoll G. Validation of administrative database codes for acute kidney injury in kidney transplant recipients. Can J Kidney Health Dis. 2016;3:18. Epub 2016/04/09. pmid:27057318.
- 44. Colantonio LD, Levitan EB, Yun H, Kilgore ML, Rhodes JD, Howard G, et al. Use of Medicare Claims Data for the Identification of Myocardial Infarction: The Reasons for Geographic And Racial Differences in Stroke Study. Med Care. 2018;56(12):1051–9. Epub 2018/10/27. pmid:30363020.
- 45. Faridi KF, Tamez H, Butala NM, Song Y, Shen C, Secemsky EA, et al. Comparability of Event Adjudication Versus Administrative Billing Claims for Outcome Ascertainment in the DAPT Study. Circulation: Cardiovascular Quality and Outcomes. 2021;14(1):e006589. pmid:33435731
- 46. Crane HM, Heckbert SR, Drozd DR, Budoff MJ, Delaney JAC, Rodriguez C, et al. Lessons Learned From the Design and Implementation of Myocardial Infarction Adjudication Tailored for HIV Clinical Cohorts. American Journal of Epidemiology. 2014;179(8):996–1005. pmid:24618065
- 47. Mahaffey KW, Harrington RA, Akkerhuis M, Kleiman NS, Berdan LG, Crenshaw BS, et al. Systematic adjudication of myocardial infarction end-points in an international clinical trial. Curr Control Trials Cardiovasc Med. 2001;2(4):180–6. Epub 2002/01/25. pmid:11806793.
- 48. Efros O, Barda N, Meisel E, Leibowitz A, Fardman A, Rahav G, et al. Myocardial injury in hospitalized patients with COVID-19 infection-Risk factors and outcomes. PLoS One. 2021;16(2):e0247800. Epub 2021/02/27. pmid:33635914.
- 49. Del Prete A, Conway F, Della Rocca DG, Biondi-Zoccai G, De Felice F, Musto C, et al. COVID-19, Acute Myocardial Injury, and Infarction. Card Electrophysiol Clin. 2022;14(1):29–39. Epub 2022/03/01. pmid:35221083.
- 50. Bavishi C, Bonow RO, Trivedi V, Abbott JD, Messerli FH, Bhatt DL. Special Article—Acute myocardial injury in patients hospitalized with COVID-19 infection: A review. Progress in Cardiovascular Diseases. 2020;63(5):682–9. pmid:32512122
- 51. Gupta A, Madhavan MV, Sehgal K, Nair N, Mahajan S, Sehrawat TS, et al. Extrapulmonary manifestations of COVID-19. Nature Medicine. 2020;26(7):1017–32. pmid:32651579
- 52.
21st Century Cures Act, Pub. L. No. 114–255 (2016), https://www.govinfo.gov/app/details/PLAW-114publ255.