Background and rationale

Individual participant data meta-analyses (IPD-MA) involve the central collection of data for each individual participant in each study included in a systematic review. They offer many advantages over aggregate data meta-analyses, and can provide key evidence to inform guidelines, policy and practice [1]. However, the validity of any meta-analysis depends on the quality of the included studies and their data [2, 3]. One advantage of IPD-MA is that access to raw data allows more detailed data quality checks than are possible when data are synthesised at study level, in aggregate. Addressing any issues identified by these checks enables higher quality data to be included in the meta-analysis which leads to more robust and reliable results. Hence, data quality checks are considered an integral component in the conduct of IPD-MA [4].

Despite this, a recent systematic review evaluating the methodological conduct of all IPD-MA of randomised trials published in English up to September 2019 (n = 323) reported that only 56% conducted checks for invalid, inconsistent, out of range or missing data [5]. This may be due to inadequate resourcing and under-appreciation of the time-intensive nature of such checks [6], which can require at least three to four days per trial and up to several weeks for large and complex datasets [7]. Furthermore, while there is an abundance of literature providing advice on the different types of data quality checks that could/should be conducted, it is not easily actionable. Guidance on which checks should be systematically performed and how these checks should be conducted is lacking. For instance, in the IPD-MA Handbook for Healthcare Research [4], the authors advise that validity, range and consistency of variables should be checked, and recommend using a standardised approach across data checkers, such as following a common checklist and/or statistical code developed for this purpose. However, the specific items on this checklist and how to action them are not elucidated. The PRIME-IPD tool [6, 8, 9] provides a useful framework for preparing IPD for meta-analysis encompassing five steps (Processing, Replication, Imputation, Merging, and Evaluation), but is not prescriptive about how each step should be performed [6]. Authors of PRIME-IPD have acknowledged that the framework needs to be refined and suggest a consensus approach such as a Delphi survey method would be appropriate for this purpose [6, 9].

Objectives

We aim to address this gap by developing an internationally recognised checklist to guide routine data quality checking and cleaning for IPD-MA of randomised trials: the CHecklist for Individual Participant data Processing of Randomised Trials (CHIPPR). While the focus is on randomised trials, many of the checks will also apply to data from other types of studies being brought together for secondary use, such as observational studies. Fig 1 depicts how CHIPPR fits within the overall conduct of an IPD-MA, and distinguishes between three separate but related types of checks that should be undertaken. One type of checks focuses on integrity and identification of potential research misconduct (fabrication, falsification, or plagiarism) [10], which is not within scope of this study and for which a checklist (informed by a scoping review) [11] is already in development. Another type is risk of bias checks, which can be informed by direct checking of IPD, e.g. to assess whether randomisation is robust. The third type is data quality checks and this forms the focus of this paper.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Stages of an individual participant data meta-analysis, highlighting where CHIPPR fits in and interrelated types of data checks (adapted from Rydzewska & Tierney) [4].

https://doi.org/10.1371/journal.pone.0275893.g001

We define data quality as the extent to which dimensions of data meet requirements, where a dimension is a measurable feature of a data concept such as accuracy, completeness, consistency and validity (Table 1) [12, 13]. The three types of data checks overlap in Fig 1 to demonstrate that some checks are common, e.g. identification of implausible dates is a common component across data quality and integrity checks; and checking for imbalance in baseline characteristics is common for risk of bias assessments, integrity checks and data quality. Each checklist item will include methods of assessment and decision criteria to guide management of any issues identified. To facilitate uptake of CHIPPR, we will develop an R-markdown program [14–16] to automate some of the processes described in the checklist. This may be adapted to other statistical software (e.g. STATA, SAS) in the future. The ultimate objective is to improve the quality and reliability of IPD-MA.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Dimensions of data quality.

https://doi.org/10.1371/journal.pone.0275893.t001

Phase 1: Scoping review to identify potential checklist items

We will undertake a scoping review in accordance with Joanna Briggs Institute (JBI) guidelines [17], to determine which data quality checks researchers perform when processing datasets for inclusion in an IPD-MA of randomised trials. A full protocol for this review, covering each relevant item of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) [18], has been preregistered on the Open Science Framework (OSF, https://osf.io/g2unf/).

We will include systematic reviews with IPD-MA of randomised trials on intervention effects published in English. These have previously been identified up to September 2019 in a systematic review by Wang et al. [5], and we will update their search to include all others published up to July 2022. Records will be independently screened by two reviewers, with any discrepancies resolved by discussion or, if required, adjudication by a third reviewer. For each eligible record, we will obtain relevant supplementary materials that are attached or referred to in the publication, e.g. statistical analysis plan, data management plan, PROSPERO registration record. We will extract information on characteristics of the included IPD-MA (e.g. year of publication, country, number of included studies) and descriptions of any procedures used to check, clean, transform or prepare individual participant data for analyses, including any software used. We will use a standardised data extraction form which will be piloted by five independent reviewers and revised accordingly prior to commencing full extraction. Where available, the form will be pre-populated by data extracted by Wang et al. [5] to avoid duplication. Data will be extracted by one reviewer with a random subset of 10% of these checked by a second reviewer. Disagreements will be resolved through discussion, with a third reviewer involved if consensus is not reached. Extracted data will be presented in summary tables and figures.

Next, we will conduct a brief online survey of authors of these IPD-MA to obtain more detailed information about their data quality checking and cleaning processes. The survey will include the data already extracted for their IPD-MA (as above), and will ask authors to add any other data quality checks that are missing, with an emphasis on how these checks were performed. Reminder emails will be sent after two weeks to non-responders. All identified candidate items will be collated and grouped into common themes and domains (such as randomisation, inconsistencies and missing data) using inductive thematic analysis [19].

Lastly, the study team will draw on their expertise and experience as IPD-MA experts, data managers, statisticians, methodologists and clinicians to add new items, exclude items beyond the scope of the research question, and refine the wording and descriptions of each candidate item for inclusion in the first e-Delphi survey round. A patient representative with previous research experience will also be separately consulted to provide feedback on the list of items. Results of the scoping review will be reported in accordance with the PRISMA-ScR [20] and disseminated via peer-reviewed publication.

Phase 2: e-Delphi survey and consensus meeting

We will conduct a two-round modified electronic Delphi (e-Delphi) survey [21] among researchers with expertise in data processing, to achieve consensus on the most important items for checking and cleaning an IPD dataset. The e-Delphi surveys will be managed using Qualtrics software [22] and pilot tested by steering group members before being launched to the expert panel (see below). Participation involves completion of survey round 1 and/or 2, and consent is implied by completion. Responses will be quasi-anonymous [23], meaning they will be anonymous to all but the lead researcher, who will be able to link responses to individuals via a unique identifier. This enables provision of feedback on previous individual responses in subsequent rounds, which is a core feature of the Delphi method [21]. Each survey round will be open for 4 weeks, and reminder emails will be sent after weeks 1 and 3 to enhance response rates. There will be a break of approximately 2–3 weeks between rounds.

Phase 3: Pilot testing

Before pilot testing of the checklist, we will develop an R-markdown template to aid implementation of the checklist. Next, members of the steering group will pilot test the checklist and R-markdown by applying them to IPD-MA they are conducting. We will also invite a convenience sample of systematic reviewers who have experience in conducting IPD-MA to conduct pilot testing. The aim of this phase is to refine the items and explanatory text and to streamline and improve the R-markdown based on participants’ feedback using an iterative process until agreement on the final R-markdown is reached among the steering group.

Patient and public involvement

As part of the scoping review, a patient representative with previous research experience will be consulted to provide feedback on the list of items.