Incidence rates.

For each PA the Incidence rates are calculated for all permutations of 10-year age groups, sex, and calendar year strata in each data source. These rates were computed by dividing the number of identified cases of the PA (i.e., the number of subjects meeting each Phenotype Algorithm criteria) by the total person-time at risk. The person-time at risk is the sum of person-years contributed by all eligible individuals within a data source (i.e. the patient’s population in a data source). Rates are expressed per 1000 person-years. The incidence rates are provided for all years of available data. By exploring the estimated incidence rate a reviewer can 1. cross-reference the observed rate with expected or known epidemiological trends reported in the existing literature (e.g. known incidence of a disease in a general population) 2. evaluate the stability of the IR overtime. Typically, incidence rates stratified by calendar year are anticipated to be present in a continuous, unvarying monotonous temporal pattern with no abrupt shifts. Should any interruptions in this pattern be observed, it could suggest alterations in either clinical practices or data capture processes, leading to potential inconsistencies in PA performance. 3. to explore whether the distribution of age and gender aligns with the known epidemiological trends for each phenotype. Additionally, this diagnostic approach can be utilized to ascertain if any data source exhibited an IR inconsistent with others, suggesting potential issues with specific definitions within that data source. On the other hand, consistent incidence estimate across multiple data sources can suggest positive reliability of the PA.

Index event breakdown.

Index event breakdown shows the count of cohort entries where a specific code in the PA’s entry event criteria, coincided with the index date of cohort entry. In other words, these are codes that likely triggered the cohort entry. The frequency of these codes allows us to assess their individual contributions to the cohort.

If most individuals are entering the cohort based on a limited number of the total specified codes in the PA, this could potentially point towards specificity errors. A higher occurrence of codes that might be semantically narrower compared to the clinical definition of the phenotype may suggest sensitivity errors. Additionally, any variations in the rank order of codes among different data sources might indicate measurement heterogeneity.

Visit context.

This diagnostic presents the count of individuals who experienced different types of healthcare visits (outpatient, inpatient, emergency department) in relation to the index date, as follows: 1) ‘Before’ represents visits that concluded within 30 days prior to the index date. 2) ’During’ accounts for visits that began before and extended up to or beyond the index date. 3) ’Simultaneous’ covers visits that initiated on the index date. 4) ’After’ includes visits that commenced within 30 days post the index date.

We anticipate that certain types of visits will be more common for specific patient phenotypes. For instance, severe acute conditions requiring intensive care will probably result in inpatient visits. A preponderance of unanticipated visit types might suggest a specificity error.

Cohort overlap.

The cohort overlap diagnostics conducts pairwise comparison of cohorts from two PAs, reporting the individuals identified by either one or both PAs, as well as only one PA. This diagnostics in CohortDiagnostics is visualized using a Venn diagram or a table. Examining the overlap between two different PAs representing the same disease can provide insights into the potential sensitivity loss associated with one algorithm compared to the other.

Cohort characterization.

Cohort characterization diagnostics provides a overview of the cohort using descriptive statistics on demographic factors, condition, drug exposures, measurements, and occurrences of procedure codes. For each selected data source, CohortDiagnostics displays the prevalence of all observed clinical events (denoted by codes) at different time periods relative to the index date. Default time windows include a) 365 to 31 days prior to the index date, b) 30 to 1 day before the index date, c) on the index date, d) 1 to 30 days post the index date, and e) 31 to 365 days post the index date.

Clinical events are represented through one or more codes. The prevalence is given for each code individually, and some are grouped using a vocabulary hierarchy. This diagnostic feature allows us to simulate, at the cohort level, the process by which clinicians establish and confirm clinical diagnoses. We expect individuals diagnosed with a certain disease to exhibit its signs and symptoms on or before the index date. Similarly, we anticipate diagnostic tests related to the disease to occur on or before its onset, followed by relevant treatment occurrences on or after onset. A lack of such expected characteristics might point to misclassification errors. To enable comparative analyses across multiple PAs, the tool carries out pairwise comparisons of all observed characteristics for each assessed PA. The results, including proportions or means and the standardized (mean) difference for each covariate, are presented in tables and scatter plots.

Application. Our evaluation of PAs focuses on two distinct scenarios: First, a researcher may examine a single PA on its own merit, by looking for possible misclassification errors across one or more data sources. This evaluation could guide the researcher in adjusting the PA in successive iterations to reduce possible misclassification errors. Second, a researcher may compare the diagnostic performance of two or more PAs that represent the same clinical concept in the same data source. This would help the researcher infer which PA might has lower misclassification errors, allowing them to choose from the two the PA that offers the best performance.

To illustrate these two scenarios, we implemented our proposed framework on two clinical concepts of interest—SLE and AD. Before evaluating the PA, we ensure we understand the known clinical profile of persons we are attempting to capture in the data source. This is done by writing a clinical description for medical condition/disease, with elements like overview, presentation, diagnostic evaluation, therapy plan, risk factors, and prognosis. The authored clinical description serves as a tool that enables documentation of the shared understanding among researchers of the target clinical idea. It also provides justification for the phenotype development design choices and expected clinical attributes to look for during phenotype evaluation.

Phenotypes.

System Lupus Erythematous. SLE is an autoimmune disease with a wide range of severity characterized by periods of exacerbation and relative quiescence and occurs predominantly among women of child-bearing age (15 to 44 years). Based on SLE clinical description, we developed a PA that allows patients to enter the cohort on the earliest of either a diagnosis code, treatment for (ie, hydroxychloroquine, steroids, biologics, or immunosuppressants) or signs and symptoms related to SLE (ie, Inflammatory dermatosis, rash, joint or back pain, endocarditis), as long as there was at least one diagnosis code for SLE within 0 to 90 days from the entry date. All patients were required to have at least 365 days of continuous observation prior to the index date. The full PA for SLE including condition and drug codes and temporal logic is in supplementary material S2 File. The validity of this PA to identify patients with SLE against a diagnostic predictive model, which serves as the gold standard, was estimated in a prior study utilizing the same data sources used in this paper [17]. Sensitivity ranged between 0.64 and 0.98 and PPV ranged from 0.244 to 0.74.

Alzheimer’s disease. AD is an age associated progressive neurodegenerative disorder and the most common cause of dementia [18]. For AD, we constructed 2 PAs. The first AD PA (referred to as the simple PA) allows patients to enter the cohort on first occurrence of an AD diagnosis. Prior studies estimated that the sensitivity of Medicare claims to identify Alzheimer’s disease using an AD diagnosis was 64.2%, and the positive predictive value (PPV) was 58.3% [19]. The second AD cohort is a more restrictive and is derived from the work by Imfeld et. al., [20] requiring one of 3 inclusion criteria: 1) the first occurrence of AD diagnosis as the entry event criterion, and any of the following inclusion criteria in relation to entry date: a) a prescription on or after for AD drug, b) a second AD diagnosis any time after, c) a prior dementia test, d) a prior, simultaneous, or subsequent dementia symptom, or e) if the first occurrence was diagnosed in an inpatient setting; or having the 2) first occurrence of dementia followed by at least 2 prescriptions for AD drugs; or 3) prescription for AD drugs followed by a diagnosis of AD. Individuals were excluded if they were under 18 years of age at cohort start date, were subsequently diagnosed with diseases that, when present, make the diagnosis of AD less likely (e.g., Vascular dementia, Lewy Body disease, Pick’s disease), or had an occurrence of a stroke diagnosis within 2 years before index date. In their study, Imfeld et.al. reports that this PA was validated through questionnaire to GPs which confirmed AD diagnosis for 79% of the AD cases.

Datasources. The data sources used in the evaluation are described in supplementary material S1 Table. We included 6 claims based data; JMDC, Merative^TM MarketScan® Commercial Claims and Encounters Database (CCAE), Merative^TM MarketScan® Medicare Supplemental and Coordination of Benefits Database (MDCR), Merative^TM MarketScan® Multi-State Medicaid Database (MDCD), IQVIA® Adjudicated Health Plan Claims Data (Pharmetrics Plus), Optum’s Clinformatics® Data Mart—Socio-Economic Status (Optum SES) and 4 electronic medical record (EHR) data; IQVIA® LPD in Australia (LPDAU), IQVIA® Disease Analyzer France (France DA), IQVIA® Disease Analyzer Germany (German DA), Optum® de-identified Electronic Health Record dataset (Optum EHR). These data sources have been de-identified at source and the involved researchers did not have information to re-identify the data. The data has been standardized to the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) [21, 22]. Extract, transform, and load (ETL) specifications for all data sources except LPDAU, France DA, German DA, and Pharmetrics Plus are available at ETL-LambdaBuilder [23]. The standardized data are assessed using a rigorous data quality process to evaluate conformance, completeness, and plausibility of the data [24]. For this application, we utilized the entirety of the data available without calendar time restrictions or sampling. The calendar time covered in each data source varied; supplementary material S1 Table lists the earliest and latest calendar times for each data source. Subjects were identified as patients with SLE or AD according to the phenotype algorithms detailed in the previous section. The six data sources selected for this application provide diversity in patient demographics, geographic locations, and healthcare settings. This diversity is crucial for evaluating the consistency of our phenotype algorithms across different data sources.

Data analysis (CohortDiagnostics software). CohortDiagnostics is open-source software application written in the R programming language that implements the described theoretical framework [25]. Given a set of instantiated cohorts, a set of cohort definition details, and a connection to a remote database with person level data converted to the OMOP CDM [21, 22] (version 5.3+), CohortDiagnostics produces a set of aggregate summary statistics called Diagnostics. The output contains no patient-level data and has additional privacy protection using minimum cell count thresholds [26]. All output conforms to the prespecified CohortDiagnostics results data model and is formatted as unencrypted comma separated value (.csv) files (an intentional design decision to allow an investigator to audit compliance with privacy governance). The output.csv files, from one or more data sources, may then be combined and the results reviewed using an interactive R Shiny web application called DiagnosticsExplorer. The software and user documentation are available on OHDSI Github repository called CohortDiagnostics [27].

Alzheimer disease.

Insights from cohort overlap. In all data sources, the Imfeld et.al. PA returned fewer patients (19% to 81%) compared with the simpler AD PA (Table 1). [20] In the cohort overlap, among individuals who were present in either cohort, the proportion of individuals present in both cohorts ranged between 18% and 45%. Further, 35% to 81% of individuals were identified only by the simple PA; and 0% to 20% were identified only by the Imfeld et al PA. We can infer that the simpler PA is likely to have higher sensitivity compared with the Imfeld et al PA.

Insights from visit context. The distributions of the visit type around the index date among the 2 cohorts were comparable in most data sources with less than 10% of the individuals in either cohorts identified during or at the start of an inpatient visit. This suggests that neither AD PAs were likely to capture more AD during an acute care event.

Insights from cohort characterization. Table 3 reports a selected set of characteristics from the 2 AD cohorts at baseline i.e., from 365 days before the index date up to and including the index date in the Optum® EHR data source (Data from all other data sources are available in the CohortDiagnostics shiny app). The covariates are defined using the Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT) vocabulary hierarchy grouping. We observe that even though the 2 cohorts were defined using different PAs and have considerably different number of patients with less than 50% overlap, the distributions of the main baseline characteristics were comparable.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Selected baseline characteristics among patients with Alzheimer’s disease by phenotype algorithm in Optum® EHR.

https://doi.org/10.1371/journal.pone.0310634.t003

Fig 4 is a screenshot from CohortDiagnostics that illustrates the covariate balance between the 2 AD PAs in Optum® EHR on 3 different time periods around the index date. Overall, we observed that most features near the diagonal, indicating comparable cohort characteristics distribution between the 2 cohorts. However, some covariates are off the diagonal with a larger Standardized Mean Difference. For example, during 30 days to 1 days before index, we observed higher prevalence of vascular dementia, and other late effects of cerebrovascular accidents in the simple PA compared with the Imfeld et al PA. This suggests that the Imfeld et al PA is less likely to misclassify cerebrovascular accident events as AD. We also observed that the simpler PA had higher utilization of drugs commonly used in AD such as donepezil and memantine in the same immediate period prior to index date. Conversely, the Imfeld et al PA demonstrated higher utilization of these drugs on index. Both PAs had similar utilization after index. This suggests that the simple PA is subject to higher index date misclassification compared with the complex PA.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Covariate balance between the 2 Alzheimer’s disease phenotype algorithms.

https://doi.org/10.1371/journal.pone.0310634.g004

When we compared covariates constructed from codes that were not part of either AD PA entry event, we observed considerable cohort similarity. This suggests that the 2 PAs identified patients with similar clinical profiles despite incomplete cohort overlap. Overall, the descriptive data of these PAs for AD revealed that, while the Imfeld et. al. identified fewer patients (raising concerns about its sensitivity), we did not observe a higher prevalence of clinical characteristics that strongly suggest a higher specificity when compared with the other simple PA.