Introduction

In a time of limited health care resources, it is crucial to make informed decisions around resource allocation; to do so, decision makers require robust information to make sound investments. Microsimulation models are computer-based models that can be used to simulate the behaviour of micro-entities such as individuals in response to interventions and/or health policies [1]. Microsimulation models can also be used to test scenarios, which cannot be tested in the real world, such as trials. Thus, as an ex-ante policy evaluation method, microsimulation models are valuable tools to examine potential behavioural and economic effects of interventions and policies and to guide decision-making [1,2]. However, to encourage the uptake of microsimulation models by policy makers, it is important that these models produce valid and robust outputs of high quality. In other words, it is important that the purpose(s) of the model is (are) well-defined, the data employed are appropriate and representative of the population studied, the model is transparent, uncertainty and model validation are considered, and model outputs are generalizable.

Checklists, commonly used in the health economics field, provide researchers with guidance in the reporting and/or quality assessment of a specific type of analysis. These checklists can help improve reporting and its consistency as well as methodological quality can allow readers to assess all important aspects of the study, while facilitating comparison across studies. Currently, there is no formal quality assessment reporting checklist for microsimulation models. Establishing the quality of microsimulation models is important as it can encourage their use by both academics and policy makers [3], the latter of whom are looking for models that produce robust outputs to guide decision making. There are many potentially relevant dimensions of quality to consider when examining microsimulation models; however, it can be challenging to discuss quality in abstract or general terms [3]. The existing reporting guidelines in the health economics field are mainly focused on economic evaluations. For example, the Consolidated Health Economic Evaluation Reporting Standards (i.e., CHEERS) checklist is typically used to evaluate the reporting standards of economic evaluations [4], some of which may make use of microsimulation techniques; however, issues like study perspective and measurement of effectiveness and costs, which are specific to economic evaluations, may not be relevant when assessing microsimulation models, particularly all-purpose, stand-alone microsimulation models. Other reporting checklists, such as the 2014 International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [5] and the 2016 Assessment of the Validation Status of Health-Economic Decision Models (AdViSHE) checklists [6], were mainly developed to examine the credibility and assess the quality of economic models, respectively, and therefore not all elements of these checklists are relevant to microsimulation models. Finally, there is a reporting checklist for discrete event simulations in health care [7]. However, this checklist is specific to discrete event simulations only and, while relevant, it is not directly applicable to other types of simulation models. Moreover, most of these checklists cover reporting standards; while reporting checklists are useful, it is also important to have checklists with a focus on quality standards.

As part of a larger study (i.e., a scoping review of microsimulation models on mental health), a quality assessment reporting checklist for microsimulation models – the Quality Assessment Reporting for Microsimulation Models (QARMM) checklist – was developed [8]. The six main elements of the QARMM checklist are largely based on work done by Sutherland (2018) [3], which describes the necessary dimensions to account for when considering the quality of microsimulation models, as well as the reporting quality checklist developed for discrete event simulations in health care [7]. The six quality assessment criteria are as follows: 1) purpose of the model, 2) data, 3) transparency, 4) uncertainty, 5) validation, and 6) generalizability, where the first three relate to the model development and structure and last three relate to the validity and scope of the results produced by the model. Each item was determined to be worth one point; half points were given in instances where the criterion was not fully met. However, the QARMM checklist has some limitations. First, it was developed specifically for the purpose of assessing the quality of microsimulation models in a scoping review, but it was not validated. Moreover, there are several aspects of the checklist that could be further improved upon. For example, while many checklists score each item equally (such as the CHEERS checklist [4]), there may be some elements that may be more important than others when developing microsimulation models (e.g., validation) and consequently should be assigned more weight (in other words, different weighting criteria should be applied). However, creating different scores for each item requires additional value judgments and the involvement of experts, which was beyond the main purpose of the aforementioned review.

The objective of this protocol is to describe a scoping review whose goal is to retrieve and synthesise the literature on existing quality assessment checklists for microsimulation models (if/where these exist) and/or any literature that provides best practices, guidelines, and recommendations around which elements should be included/accounted for when building microsimulation models.

Methods

Discussion

There are likely no existing formal checklists to assess the quality of microsimulation models, beyond the QARMM checklist [8]. Moreover, no scoping reviews have been undertaken on this topic. A validated quality assessment reporting checklist will be the first of its kind. Thus, this work would fill an important gap in the literature and help inform the development of a new quality reporting checklist devoted specifically to microsimulation models. The development and validation of a quality assessment reporting checklist (and, in turn, general guidance on how to develop and design robust microsimulation models) is likely to greatly improve the quality of future microsimulation models. Thus, it is expected that the development and use of this checklist will have an impact on health care policy/decision making.

Findings from the scoping review will be presented at relevant conferences and published in an appropriate peer-reviewed academic journal. In addition, this information will be used to apply for a grant to develop a refined quality assessment reporting checklist for microsimulation models, building off the results obtained from the scoping review. Any relevant changes/amendments to the scoping review, where applicable, will be noted and justified and subsequently recorded in the study protocol.

Supporting information