How to determine if a model is right for neglected tropical disease decision making

Bruce Y. Lee; Sarah M. Bartsch

doi:10.1371/journal.pntd.0005457

Abstract

Mathematical and computational modeling can transform decision making for neglected tropical diseases (NTDs) if the right model is used for the right question. Modeling can help better understand and address the complex systems involved in making decisions for NTD prevention and control. However, all models, modelers, and modeling are not the same. Thus, decision makers need to better understand if a particular model actually fits their needs. Here are a series of questions that a decision maker can ask when determining whether a model is right for him or her.

Citation: Lee BY, Bartsch SM (2017) How to determine if a model is right for neglected tropical disease decision making. PLoS Negl Trop Dis 11(4): e0005457. https://doi.org/10.1371/journal.pntd.0005457

Editor: Serap Aksoy, Yale School of Public Health, UNITED STATES

Published: April 20, 2017

Copyright: © 2017 Lee, Bartsch. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: This work was supported by the Agency for Healthcare Research and Quality (AHRQ) via grant R01HS023317; the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), Office of Behavioral and Social Sciences Research (OBSSR), and the Global Obesity Prevention Center (GOPC) via grant U54HD070725; NICHD via grant U01 HD086861; and USAID via grant AID-OAA-A-15-00064. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Mathematical and computational modeling can transform decision making for neglected tropical diseases (NTDs) if it is the right model for the right question. Modeling has revolutionized decision making in many other fields such as meteorology, manufacturing, transportation, and air traffic control. For example, prior to the advent of simulation models that could generate weather maps, weather forecasting and planning had to rely on much more haphazard observations. Modeling can help better understand and address the complex systems (e.g., biological, behavioral, environmental, cultural, and economic) involved in making decisions for NTD prevention and control, which unaided can be challenging. Fig 1 shows the timeline/life cycle for a product or policy and how modeling can aid decision making at each step. However, all models, modelers, and modeling are not the same. Models range from those primarily oriented towards scientific inquiry to those that assist decision making with many variations in between. Therefore, decision makers need to better understand if a model actually fits their needs. What follows are the questions that a decision maker can ask when determining whether a model is right for him or her.

Download:

Fig 1. Examples of how modeling can help decision making at each stage in the product and policy development and implementation timeline.

https://doi.org/10.1371/journal.pntd.0005457.g001

Question 1: Does the modeler understand me?

In any business, understanding the client is paramount to designing and providing a product or service. Similarly, without understanding the decision maker, there is risk of developing a model that does not fit or capture the needs of the decision maker. These needs can be complex, subtle, and not apparent to a modeler who is unable to place himself or herself in the shoes of the decision maker. Understanding the decision maker requires humility, a desire to listen and learn, and preferably direct experience. Nothing can really replace experience. However, without true experience, the modeler has to put in considerable extra work to truly relate. For example, a common complaint among physicians and policy makers is that a model does not accurately represent the realistic options and constraints that they face. This raises the second question.

Question 2: Does the model reflect my real-world constraints?

While models are simplifications of the real world, to be useful to a decision maker, the model needs to include the constraints that the decision maker may face. This is a key distinction between models that are designed more for pure scientific inquiry and those that aim to assist decision making. Common real-world constraints include economics (e.g., some interventions may be cost prohibitive), logistics (e.g., getting medications to a population requires functional and far-reaching supply chains) and behavior (e.g., people may not comply with completing a course of medications).

Question 3: Does the model incorporate all the considerations?

Models for decision making are by definition, multi- and inter-disciplinary. A decision is rarely specific to a discipline. In other words, a decision is not purely a biology decision, an epidemiology decision, or an economic decision. Instead, a decision typically spans multiple disciplines and considerations. There may be biological, epidemiological, or economic considerations that need to be integrated.

Question 4: Was the model originally designed for this purpose?

Choosing the right model is important. All models are not the same. Instead, a tremendous diversity of models exist, and different models serve different purposes. For example, some modelers and models are more oriented towards specific biological or transmission processes and scientific inquiry. Others help understand and guide clinical processes. Still others focus more on policy making. No single model or modeler can do all of these or serve other similar disparate purposes at once. Similarly, a wide variety of modeling methodologies exist, ranging from decision analytic models, to compartment models, to network models, to agent-based models, and many other types and hybrids along the spectrum of possibilities. No single methodology is appropriate for all situations or necessarily superior or more complex than other methodologies. Repurposing a model that was originally designed for another purpose can be analogous to using a car as a boat.

Question 5: Is the model too simple or too complex?

The principle of parsimony specifies that a model should not be more complex than needed. In other words, a model should only include what’s necessary. A model is not a replica (i.e., something that is supposed to mirror everything in real life), but instead a distillation of the important factors and relationships germane to a question or decision. For instance, detailed representations of pathogen transmission that do not significantly affect cost or effectiveness may not be necessary in an economic model. On the other extreme, models that oversimplify important processes or leave out key factors or elements may produce misleading results (e.g., a model that determines the value of a treatment but does not account for the costs and health effects).

Question 6: Does the model take the appropriate perspective?

The modeler should clearly identify the perspective of the model, which in turn should match the perspective (i.e., viewpoint) of the decision maker. Different decision makers have different perspectives. For example, an employer may be interested in how a vaccine affects company costs and productivity, whereas an individual patient may be more interested in how a vaccine affects his or her own health and finances.[1–3] A third party payer (or insurance company) may focus on the direct medical costs incurred and saved by a vaccine, which represent what the third party must pay. The societal perspective accounts for both direct medical costs and productivity losses. Model results can differ considerably by perspective (e.g., the annual direct medical costs of Chagas disease may be US$474 but the societal costs are much higher at US$4,660[4]).

Question 7: Does the model use the appropriate time frame?

The modeler should also clearly identify the time frame of the model. Time frame matters in decision making. The model should represent the duration of time that is relevant to decision making. A model that only represents a day after the administration of a vaccine would not capture its potential costs and benefits. For example, capturing the value of a vaccine with a 20-year duration of protection would require running a model for at least 20 years.[5]

Question 8: Does the model require too much data?

Decision makers frequently have to make decisions under uncertainty and data sparse situations. A modeler asking for extensive amounts of data that require too much time to analyze can defeat a model’s utility. A common myth is that models are only as good as the data, i.e., garbage in, garbage out. Depending on the type and structure of the model, this is not necessarily true. Some models rely more heavily on data, such as those in which the modeler essentially fits mathematical equations onto specific datasets. However, other models attempt to represent different relevant mechanisms by other means besides extensive datasets. They can make certain assumptions as long as they are transparent and justifiable, draw learnings from other fields and settings, or develop representations based on observations and theories. Extensive sensitivity analyses can help test the robustness of assumptions and various representations. This is especially relevant for NTDs, for which data is often sparse, of poor quality, or absent. In fact, trying to fit models on poor-quality data can actually lead to misleading findings. The relative lack of data should not prevent the building of models. In fact, building models can help determine what data is needed and the value of additional data, which can guide and prioritize data collections.

Question 9: Does the model generate relevant measures and outcomes?

Each type of decision maker tends to rely on particular measures to make decisions. For example, those with financial responsibilities will focus on financial measures. Health care workers will be very interested in key health outcomes, such as deaths, hospitalizations, and incidence of various types of diseases. Model outcomes should lead to actionable steps. For example, the reproductive rate (R0) may be of interest to an epidemiologist, but have little meaning to a policy maker. Additionally, modelers need to be able to generate graphs and visualizations that communicate findings in a concise and easily comprehendible manner. Many scientific methods of presenting data just do not resonate with decision makers.

Question 10: Is the modeler transparent about what the model can and cannot do?

Just like every tool, product, or service, every model has its strengths and limitations. Misuse of models is common and is analogous to using a hammer as a saw. A model that is very good for a given purpose can be very bad for another. Ultimately, decision makers should be wary of any model that is a “black box” to them. Modelers need to be willing and able to effectively communicate their model’s structure and components, strengths and weaknesses, and associated assumptions and limitations. In many ways, sensitivity analyses (exploring the effects of varying key model parameters) are the most important model results, as they show how it performs under different conditions and circumstances. The analogy is experiencing how a car behaves during a test drive.

Table 1 shows examples of questions decision makers may ask and the models that have been used to answer such questions. While modeling can help address many current challenges of NTD decision making (e.g., complex systems impacted by environmental, economic, behavioral, and biologic factors[6]; subtle and indirect impacts that accumulate over time[7]), both modelers and decision makers have to better appreciate all of the above considerations so that they can choose the right model to fit the right decision.