MIHARI project, a preceding study of MID-NET, adverse event detection database of Ministry Health of Japan—Validation study of the signal detection of adverse events of drugs using export data from EMR and medical claim data

We studied the effectiveness of the direct data collection from electronic medical records (EMR) when it is used for monitoring adverse drug events and also detection of already known adverse events. In this study, medical claim data and SS-MIX2 standardized storage data were used to identify four diseases (diabetes, dyslipidemia, hyperthyroidism, and acute renal failure) and the validity of the outcome definitions was evaluated by calculating positive predictive values (PPV). The maximum positive predictive value (PPV) for diabetes based on medical claim data was 40.7% and that based on prescription data from SS-MIX2 Standardized Storage was 44.7%. The PPV for dyslipidemia was 50% or higher under either of the conditions. The PPV for hyperthyroidism based on disease name data alone was 20–30%, but exceeded 60% when prescription data was included in the evaluation. Acute renal failure was evaluated using information from medical records in addition to the data. The PPV for acute renal failure based on the data of disease names and laboratory examination results was slightly higher at 53.7% and increased to 80–90% when patients who previously had a high serum creatinine (Cre) level were excluded. When defining a disease, it is important to include the condition specific to the disease; furthermore, it is very useful if laboratory examination results are also included. Therefore, the inclusion of laboratory examination results in the definitions, as in the present study, was considered very useful for the analysis of multi-center SS-MIX2 standardized storage data.


Introduction
A number of studies have attempted to utilize secondary data directly from EMR for clinical research, etc. A potential solution to this problem was first demonstrated by the Adverse fluoroquinolone-induced tendon disorders by the secondary use of SS-MIX2 Standardized Storage at an institution [9]. A study on the early detection of drug-related adverse events using standardized patient data (i.e., medical claim data and the data from SS-MIX2 Standardized Storage) from multiple institutions was conducted by Japan's Pharmaceuticals and Medical Devices Agency (PMDA). Here, we describe this Medical Information for Risk Assessment Initiative (MIHARI) project in detail. In this study, we investigated whether known adverse events could be detected from the data of several institutions that use SS-MIX2 standardized storage and compared the results with those from paper-based medical records and electronic data capture (EDC) systems. The database search engine D � D [10] was used in this study. It utilizes CACHE, which is a tree structure [11].
Based on its usefulness, the PMDA initiated a project named Medical Information Database Network (MID-NET) in 2018 for the detection of adverse events, in an attempt to substitute post-marketing surveillance [12][13][14]. In this paper, we describe the MIHARI project, in which we investigated whether known adverse events could be detected from data directly collected

PLOS ONE
Validation study of the signal detection of adverse events of drugs using export data from EMR from the EMR of multiple institutions; this formed the basis of the ongoing MID-MET project.

Materials and methods
To investigate whether known adverse events can be detected by database surveillance using a hospital information system, serving as a substitute for traditional large-scale surveys in which data from each hospital is individually collected by companies, etc., utilizing a large amount of resources.
In this study, medical claim data and SS-MIX2 standardized storage data were used to identify four diseases (diabetes, dyslipidemia, hyperthyroidism, and acute renal failure) and the validity of the outcome definitions was evaluated by calculating positive predictive values (PPV) as indices of validity (Fig 2: Methods for Surveillance). Fig 2 shows the flowchart of surveillance that combination of medical claim data and SS-MIX2 standardized storage data are used for calculation of potential cases after anonymization.
In this study, we used two methods. Method 1 was used to identify diabetes, dyslipidemia, and hyperthyroidism. Method 2 was used to identify acute renal failure.
• Method 1: Evaluation of validity based only on laboratory examination results.

PLOS ONE
Validation study of the signal detection of adverse events of drugs using export data from EMR Subject outcome: New onset of acute renal failure. We adopted two different evaluation methods because the validation study of Method 2, including data from medical records, would require a large amount of resources in terms of time and cost for reviewing; therefore, we also used a simple method (Method 1) based only on laboratory examination results for the evaluation of validity of the outcome definitions.
{Data sources} SS-MIX2 standardized storage data were obtained from the hospitals listed below. Medical claim data and medical records were obtained from two hospitals among them. {Anonymization of data} A second PMDA identification (ID) was assigned to each patient whose data were extracted. In Method 1, irreversible anonymization was performed, before providing the data to PMDA, by discarding the correspondence table between original hospital ID and second ID. In Method 2, despite the fact that irreversible anonymization was desirable because their medical records were utilized, the correspondence table between original hospital ID and second ID was kept under lock and key in each hospital so that the patient data was not identified by outsiders.
{Case identification by outcome definitions} Codes were provided for each disease name, drug, and medical practice according to disease. Some of the codes are provided in Table 1 (Examples of Code Lists). International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10) codes were used for disease names, YJ codes were used for drugs, and the codes of the computerized receipt processing system were used for the related medical practice.
Using these codes, automatically extracted groups (potential cases) were identified based on eight extraction conditions (outcome definitions) for Method 1 ( Table 2: Outcome Definition in Method 1) and one extraction condition (outcome definition) for Method 2 ( Table 3: Outcome Definition in Method 2).
In Method 1, the outcome definitions differed according to data source (medical claim data alone or the use of SS-MIX2 standardized storage data) and the combination of disease name, drug, and medical practice.
In Method 2, the outcome definition was based on the combination of disease name and laboratory examination data obtained only from SS-MIX2 standardized storage data.
{True case identification} "True cases" were identified from the extracted potential cases. Method 1 (diabetes, dyslipidemia, and hyperthyroidism).
The case identification method based on laboratory examination results was adopted, and the method was described with reference to the diagnostic guidelines and the opinions of clinicians (Table 4: The case identification method).

SS-MIX2 Medical Claim
The month of initial definitive diagnosis is the same as or that prior to the month of initial drug prescription. No interval limitation between the month of initial definitive diagnosis and that of initial drug prescription.
Months of initial drug prescription Disease Name and Drug (No interval limitation) The month of initial definitive diagnosis is the same as or that prior to the month of initial drug prescription. The interval between the time of initial definitive diagnosis and the time of initial drug prescription was 1 month or shorter.
Months of initial drug prescription Disease Name and Drug (Interval limitation) Definition 4 Y The month of initial definitive diagnosis is the same as the month of medical practice. Month of initial definitive diagnosis Disease Name and Medical Practice Definition 5 Y The month of initial definitive diagnosis was the same as the month of medical practice. Months of initial drug prescription Drug and Medical Practice The month of initial definitive diagnosis is the same as or before the months of medical practice, and the month of medical practice is the same as or before the month of initial drug prescription. No interval limitation between the month of initial definitive diagnosis and the month of initial drug prescription.
Months of initial drug prescription Disease Name, Medical Practice, and Drug (No interval limitation) The month of initial definitive diagnosis is the same as or before the months of medical practice, and the month of medical practice is the same as or before the month of initial drug prescription. The interval between the time of initial definitive diagnosis and the time of initial drug prescription was 1 month or shorter.
Months of initial drug prescription Disease Name, Medical Practice, and Drug (Interval limitation) 1) Disease names, treatment drugs, and medical practice used for the search were shown in Appendix 1, 2, and 3.
2) Month of initial definitive diagnosis: The month in which the disease used for the definition was initially diagnosed during the data period.
3) Month of initial drug prescription: The month in which the drug used for the definition was initially prescribed during the data period. https://doi.org/10.1371/journal.pone.0255863.t002

PLOS ONE
Validation study of the signal detection of adverse events of drugs using export data from EMR Method 2 (Acute renal failure) True cases were identified based on medical records, with reference to laboratory examination results after consultation with clinicians.

SS-MIX2 Medical Claim
Definition 1 Y Acute elevation of serum Cre levels 3) within 1 month prior to and after the month of initial definitive diagnosis. Disease Name and Rapid Increase in Serum Creatinine Values 1) Disease names used for the search are shown in Appendix 1.
2) Month of initial definitive diagnosis: The month in which the disease used for the definition was initially diagnosed during the data period. Month of initial drug prescription: The month in which the drug used for the definition was initially prescribed during the data period.
3) The definition of acute elevation of serum Cre levels: The serum Cre level within 1 month before and after the index month increased by 0.3 mg/dL or up to 150% compared to the previous serum Cre level of the past 3 months.
The case identification method for diabetes 1.Fasting blood glucose level is 126 mg/dL or higher or casual blood glucose level is 200 mg/dL or higher.
Identified as a true case if the above criteria were met twice or more during the observation period.
The case identification method for dyslipidemia 1.Blood LDL cholesterol level is 140 mg/dL or higher.
2.Blood HDL cholesterol level is less than 40 mg/dL.
3.Blood triglyceride level is 150 mg/dL or higher.
Identified as a true case if the above criteria were met at least once during the observation period.
The case identification method for hyperthyroidism 1.The free T4 level is 1.8 ng/dL or higher or the free T3 level is 4.0 ng/dL or higher.
Identified as a true case if both criteria were met. The case identification method for acute renal failure Cases of acute renal failure were identified by three nephrologists based on acute changes in serum creatinine (Cre) levels.
We decided that patients on long-term dialysis and those who had undergone renal transplantation should be excluded from the analysis. Therefore, patients were divided into subgroups according to previous serum Cre levels, according to laboratory tests in the past 3 months, in order to identify those who previously had high serum Cre levels.
The previous serum Cr levels were classified into three categories: ≦1.2 mg/dL, >1.2 mg/dL and ≦2.0 mg/dL, and >2.0 mg/dL. Cases were classified as "true cases" or "other cases" according to the identification method for each disease. Subsequently, PPV and 95% confidence intervals (95% CI) were calculated as indices for the evaluation of the validity of the outcome definitions. This time, the Wald method was adopted to estimate the confidence interval [15]. The formula used for calculating PPV is as follows: In Method 2 (acute renal failure), true cases were identified from random samples, accounting for 30% of total potential cases. The following formula was used in this case for PPV calculation:

Results
With respect to the outcome definition for newly developed diabetes, the number of cases and PPV based on medical claim data and SS-MIX2 standardized storage data are shown in Table 5 (Number of cases and PPV according to the outcome definition for diabetes). With respect to the outcome definition for diabetes, none of the PPVs reached 50% under any condition. The maximum PPV for "Definition 3-2 Disease Name and Drug (No interval limitation)" using the SS-MIX2 standardized storage data was 44.7%, whereas the maximum PPV for "Definition 6-2 Disease Name, Medical Treatment, and Drug (No interval limitation)" using medical claim data was 40.7%. The PPVs for diabetes were higher among the results obtained from SS-MIX2 standardized storage data than those obtained from medical claim data in any of the definitions.
Considering the outcome definition for newly developed dyslipidemia, the number of cases and PPV calculated using medical claim data and SS-MIX2 standardized storage data are shown in Table 6 (Number of cases and PPV according to the outcome definition for dyslipidemia). The PPV for dyslipidemia was 50% or higher under all of the conditions.
With regard to the outcome definition for newly developed hyperthyroidism, the number of cases and PPV calculated using SS-MIX2 standardized storage data and medical claim data are shown in Table 7 hyperthyroidism). The PPV for hyperthyroidism, based on disease name definition alone, was 20-30%; however, the value exceeded 60% when the drug prescription was included in the conditions. Method 2 was applied to two medical institutions and the subject outcome was defined as "newly developed acute renal failure." A review of medical records was performed with the cooperation of specialists, including multiple nephrologists. For evaluation of the validity using gold standards regarding the information on medical records, the subjects accounting for 30% of the 1,447 potential cases were selected by random sampling. Based on the medical  records, each case was classified as a "true case" or "other case." Evaluation of definition 1 (disease name & acute elevation in serum Cre levels) based on medical record information showed that PPV was as low as 53.7% (Table 8: Number of cases and PPV according to the outcome definition for acute renal failure); however, this value increased to approximately 80-90% when patients who previously had high serum Cre levels were excluded.

Discussion
This study investigated whether known adverse events could be detected by database surveillance using a hospital information system, in an attempt to substitute traditional large-scale surveys in which data from each hospital is individually collected by companies, etc. Study outcomes considered were newly developed diabetes, dyslipidemia, hyperthyroidism, and acute renal failure. The methods used in this study were relatively simple, because the review of laboratory examination results was performed by a single researcher in PMDA, although the review of medical records was done with the cooperation of multiple physicians. Detection of true cases by Method 2 took time and effort, as careful review of medical records by specialists was needed for validation. Contrastingly, Method 1 was more feasible and effective, detecting true cases and providing sufficient PPV only by reviewing the laboratory examination results; thus, this suggested the validity of the method. We believe that the simple validation performed using laboratory examination results alone is useful, if guideline definitions are specified. In addition, a mini-sentinel surveillance showed a method for identifying true cases based on the information of test results alone and reported that the outcomes considered feasible for validation included hyperglycemia, dyslipidemia, hyperthyroidism, etc. [16].
The PPVs of hyperglycemia, dyslipidemia and hyperthyroidism were calculated only by laboratory examination results without reviewing medical records which required cooperation of specialists. This methodology was so effective to detect ADEs that large scale evaluation of ADEs was possible for many drugs which was used subsequent MID-NET project initiated in 2018 in Japan.
SS-MIX Standard Storage Data was used for PPV definition. The reliability of SS-MIX Standardized Storage Data is generally accepted in Japan and also used in clinical studies such as diabetes mellitus [17]. We tried to graph each PPV about the definition of Method 1 (Fig 3. Positive Predictive Values (PPV)___Method 1).There were fewer evaluation items and insufficient statistical analysis, but the following were shown.
The PPV for diabetes was relatively low. This may be attributed to the fact that evaluation was made based mainly on laboratory examination results, despite the inclusion of a wide variety of diabetes patients (those with suspected diagnosis, untreated patients, those under treatment, and those with different treatment effects) in the analysis. The PPV for diabetes was higher in the results obtained from SS-MIX2 standardized storage data than in those obtained from medical claim data in any of the definitions. The PPV for dyslipidemia was 50% or higher in all conditions. This may be due to the fact that many cases could be identified by the disease name definition alone, because of its high specificity. The PPV for hyperthyroidism outcome was 20-30% based on disease name definition alone; however, the value exceeded 60% when drug prescription was included in the definition, suggesting that drug prescription has high specificity for this disease. The total PPV for acute renal failure outcome was 53.7%, but increased to approximately 80-90% when patients who previously had high Cre levels were excluded. The PPV for this disease was higher than that of other diseases. Given that Cre levels are higher in patients on dialysis, etc., exclusion of such patients in this study might have led to higher PPV, but the most probable reason may be that "the definition was based on test results." In addition, PPV based on disease name definition alone was low in all outcomes. The reason for this may be as follows: when medical institutions submit medical care fee claims, they are required to input disease names that are consistent with the medical practice provided; therefore, there may have been cases where the disease name recorded by the medical institution did not accurately reflect the actual clinical conditions. Thus, outcome definitions based on disease name alone may be inappropriate in studies using medical information databases.
This study shows that SS-MIX data extracted from multiple collaborating medical institutions can be used for quantitative study on safety assessment, such as the impact of safety measures, examination of adequacy of assessment and outcome definition. In the future, PMDA will promote the medical information database infrastructure development project in cooperation with the Ministry of Health, Labor and Welfare in order to build a medical information database for the purpose of improving safety measures for drugs, etc. [18].
In the system development of the medical information database, the same extraction program was sent to each medical institution based on the experience in the trial survey using the SS-MIX data so far, and the target data was executed by each medical institution. It has become possible to implement a system from the search and extraction of to the simple aggregation. Moreover, it is considered that the knowledge about the characteristics of the hospital information system data of medical institutions and the utilization method obtained from the trial survey using SS-MIX data such as this time can be utilized when using the medical information database in the future.

Conclusion
When defining a disease, it is important to include the condition specific to the disease; furthermore, it is very useful if laboratory examination results are also included. Therefore, the inclusion of laboratory examination results in the definitions, as in the present study, was considered very useful for the analysis of multi-center SS-MIX2 standardized storage data. In Japan, it is expected that the number of pharmacoepidemiological studies based on the secondary use of large scale databases, such as medical information database, will increase in the future. Accordingly, the importance of validation studies is expected to increase as well. However, it is difficult to evaluate the validity of all outcome definitions using medical records. Therefore, the evaluation method of validity should be selected according to the types of outcomes. For example, the evaluation method using laboratory examination results can be used for diseases that can be diagnosed based on changes in laboratory values. Effective evaluation of the validity of the outcome definitions and accumulation of findings regarding various outcome definitions are considered important. In Japan, the 2018 MID-NET project, involving 24 hospitals (10 medical groups), was initiated based on the promising results presented here (i.e., direct detection of adverse events using EMR data) in an attempt to substitute post-marketing surveillance.