The Biosurveillance Analytics Resource Directory (BARD): Facilitating the Use of Epidemiological Models for Infectious Disease Surveillance

Epidemiological modeling for infectious disease is important for disease management and its routine implementation needs to be facilitated through better description of models in an operational context. A standardized model characterization process that allows selection or making manual comparisons of available models and their results is currently lacking. A key need is a universal framework to facilitate model description and understanding of its features. Los Alamos National Laboratory (LANL) has developed a comprehensive framework that can be used to characterize an infectious disease model in an operational context. The framework was developed through a consensus among a panel of subject matter experts. In this paper, we describe the framework, its application to model characterization, and the development of the Biosurveillance Analytics Resource Directory (BARD; http://brd.bsvgateway.org/brd/), to facilitate the rapid selection of operational models for specific infectious/communicable diseases. We offer this framework and associated database to stakeholders of the infectious disease modeling field as a tool for standardizing model description and facilitating the use of epidemiological models.


Introduction
The Biosurveillance Analytics Resource Directory (BARD) is a catalogue of models of disease outbreaks, defined as unusually high disease incidence at a given time and location. Included models are designed for outbreak prediction (estimating the probability that an outbreak will occur) and/or forecasting (estimating the "extent and locations of disease spread" once an outbreak has occurred). 1 It contains information about operational models that has been systematically categorized and curated. The BARD is designed to help users rapidly and reliably select epidemiological models appropriate for their real-world situations. In particular, it offers domain-specific search tools several orders of magnitude more efficient in user time and effort than general web tools like Google.
This document describes the usability requirements for the BARD; that is, who will be using the app and what tasks they will be using it for. It does not include information on technical implementation (e.g., whether specific information is contained in the database or pulled on demand from other sources). It also avoids specific design ideas (such as widget descriptions) unless they are necessary to illustrate a requirement.

Assumptions
We list here the key assumptions upon which the BARD design is based. There is no particular order.
1. People will use the web app with a standard web browser and relatively large screen, at least 1280×768 at ~100 dpi. Specifically, targeting mobile devices is a future project.
2. Users can read English reasonably well (at the 8th grade reading level or better). This is selected as a level that is readily available among the target users, even those who are not native English speakers.
3. Users are proficient in basic computer use; that is, while they may be unfamiliar with the BARD, they are familiar with their operating system, browser, and related tools.
4. Users have medical or public health training and understand relevant jargon.
5. There will be no user manual for the app. Rather, the app will be self-explanatory, with context-specific help available where needed.

Use cases
This section is a comprehensive listing of BARD use cases. Within each category, no ordering is implied.

Primary use cases
Our core use cases (i.e., key deliverables). People would have the tasks described below and use the app to support these tasks.

Model selection
Given a specific situation, select from the complete set of plausibly applicable models a small set (say 1-3 models) to explore more deeply. This includes:  Determine which models are available for specific objectives such as disease, location, and biosurveillance objective (prediction or forecasting). Objectives might include answering questions such as: o Should schools be closed, and if so which ones? o How much vaccine is necessary? Who should be vaccinated first? o What is the economic impact of a given outbreak?  Quickly understand broad properties of these models, such as operational readiness.  Understand any model well enough to decide whether to evaluate it more fully. That is, the BARD organizes third-party claims about models to help analysts decide whether to follow up in more detail.  Compare models to understand their differences.  Be able to compare selected models and understand differences between models and the user's requirements.  Understand why some models were excluded (for example, if a plausibly expected model is not in the BARD, understand which inclusion criteria it did not meet).

Model follow-up
Given a specific model, access additional resources regarding that model, doing so as quickly as the model allows (e.g., some models can be downloaded from the web, while others require negotiating access with their developers). These resources might include:  Contact information for model developers.
 Pointers to the model itself (executables and/or source code).  Pointers to user documentation.  Pointers to scientific publications and/or technical documentation.
In other words, this use case supports interaction between analysts/model users and model developers.

Data curation
Management of content stored in the BARD. This includes:  Adding new models.  Verifying existing model information.
 Updating information about existing models.  Removing models which turn out to not meet the inclusion criteria.  Soliciting verification and updates from others.

Secondary use cases
Other things we want to be relatively useful. We will spend limited effort on their design but not actively hinder or break them.

Gap analysis
Understand the current state of the art in operational models for a given context (disease, location, etc.). Identify and characterize gaps and challenges in this state of the art.

New model
Understand and characterize a new model.

Tertiary use cases
Additional things people do that are not so important. These tend to be things that are weakly related to the BARD's purpose, but they are still important to enumerate. We will spend very little design time here.

About
Learn about the database, its team, funding, etc.

Non use cases
We will not support these; they are currently or permanently out of scope.

Evaluation
The BARD does not include information about third-party evaluation or verification of models. That is, there are no recommendations provided, only information produced by model developers. In particular:  The maintainers of the catalogue do not perform evaluations.
 The catalogue does not include information on third-party evaluations of the models.
These activities are beyond the scope of the current project.

Modeling
The BARD is not a modeling environment and does not support running models within the tool.

Primary personas
These are the main personas we will design for. No ordering is implied. This list should be as short as practicable (no more than 4); it is better to pick a few representative personas rather than a longer list with relatively minor variations.

Arlene the global health analyst
 Organization: National or global health organization; e.g., CDC, WHO, HHS  Goals: o Assess the probable future impact of specific diseases in specific locations. o In response to an outbreak, forecast its future course in order to make recommendations for response. o Communicate this awareness and recommendations to decision makers. o Understand the resources that are available with respect to specific possible scenarios. That is, given a particular crisis (real or imagined), understand which tools are available to respond to that crisis. o Understand the current status and probable future course of research and development for models of specific diseases, including those that currently lack reliable surveillance and forecasting.  Education: M.S., Ph.D., or M.D. in epidemiology, public health, or medicine  Behavior/attitude: May be wary of the use of models for decision-making; more likely relies on multiple criteria and will view models as supplementary.  Tech savviness: Moderate. Uses computers throughout her work, but is not interested in technology for its own sake.  Knowledge of disease models: Rudimentary; limitations and assumptions associated with model output will not be readily apparent.

Don the local public health practitioner
 Organization: City, county, or state level public health agency  Goals: o Build and maintain situational awareness regarding any diseases which are causing current or potential local problems. o Make decisions about local strategy and specific local interventions. o Explain and justify these decisions to local politicians and the public.  Education: B.S. or M.S. in public health.  Behavior/attitude: Focused on local problems and local solutions; less interested in the nationwide or global picture. Overworked and under-budgeted. Committed to his responsibility, but his department has suffered from multiple rounds of budget cuts and is understaffed. Wants to quickly complete his task and move on to the next thing. Lacks budget to modify models or perform complex parameterization; i.e., he needs something already tested that is ready to go.  Tech savviness: Moderate. Uses computers throughout his work, but is not interested in technology for its own sake.  Knowledge of models: Varied. Will use models to help in decision-making., but this is not a primary interest or expertise. Limitations and assumptions in model output may not be readily apparent.

Secondary personas
We will consider these personas in our design, but they are not our focus and will not be considered in usability testing.

Non-personas
Design for these personas is explicitly excluded.

Emily the decision maker
Decision makers are indirect users of the BARD. They will rely on reports, presentations, etc. prepared by the direct users noted above.

Fiona the reporter
News people will rely on the above personas for a more digestible view of what is going on.