Initial study design.

Preclinical animal intervention studies are experimental studies in which the investigator controls the intervention [34]. In our approach (Box 1) evidence from randomized animal experiments start as high quality evidence. Inadequate or lack of randomization is part of the risk of bias assessment. The identification of, and classification of different types of study design in preclinical animal research merits further research. For example, cross-over designs, in which all animals receive all treatments, or non-experimental (i.e. observational) animal studies to assess ecological impacts of an exposure occur in the environmental health setting. It is unclear, however, for which types of questions these study designs are used in preclinical animal studies.

In rating the certainty of the evidence we propose to assess–by outcome–the GRADE downgrading factors a) risk of bias, imprecision, inconsistency and publication bias, followed by b) two layers of indirectness and c) considering upgrading (Box 1). The last step is to rate the certainty in the effect taking all factors in conjunction. How indirectness should be weighted in the total rating remains a challenge.

Step 5a: Assess risk of bias, inconsistency, imprecision and publication bias.

Trials may incur risk of misleading results if they are flawed in their design or conduct. Therefore, the risk of bias of the individual studies needs to be assessed as one of the steps of performing a SR. There are several tools available to assess risk of bias of animal studies [36], for example the SYRCLE risk of bias tool [4]. S4 Table shows the main differences between risk of bias of clinical and animal intervention studies. Second, for each outcome the risk of bias across all included studies (body of evidence) is assessed.

Although rigorous SRs demand assessment of risk of bias of the included studies, currently this is not common practice in SR of animal studies [41]. This implies that assessment of risk of bias may still need to be performed if one uses existing SRs. Poor reporting of essential design characteristics of the animal experiments may often hamper this step.

In the probiotics for pancreatitis example, risk of bias was assessed with SYRCLE’s risk of bias tool [4]. Risk of bias was unclear in the majority of the studies due to poor reporting. To tackle this may require contacting study authors.

Inconsistency is typically assessed by considering the overlap between confidence intervals (CI), the magnitude and direction of effect of the individual studies, the p-value of the test for heterogeneity and I² (describes the percentage of the variability in effect estimates that is due to heterogeneity rather than sampling error (chance)) [21]. The GRADE approach suggests rating down the certainty in the evidence if inconsistency in the results of the individual studies remains unexplained after exploration of hypotheses that might explain heterogeneity. If differences in population, intervention, comparators or outcomes or risk of bias provide an explanation for heterogeneity, meta-analysts should offer estimates for the appropriate subgroups (or, if risk of bias explains the inconsistency, use of only the low risk of bias studies might be appropriate).

Several challenges remain in assessing inconsistency. First, heterogeneity in animal research can be expected, as a result from the often exploratory approach. In other words, part of the heterogeneity is intentionally induced and, in that case, should not be part of the certainty of the evidence grading because it can be explained. The issues with regard to inconsistency are: a) how to separate induced and explained from unexplained heterogeneity; and b) how to interpret I². Second, heterogeneous results overall could be consistent within species, therefore two levels of inconsistency can be present: within species and across species. For example, when all species included in the analysis show the same direction of effect, we are more certain that the intervention effect is robust across different species, including humans. In that case we would not downgrade for inconsistency even if the results overall are heterogeneous.

In the probiotics for pancreatitis example, inconsistency was not a concern for mortality: the CIs of the three studies were overlapping and there was no statistical heterogeneity as indicated by an I² of 0.0% and a p value of 0.73. The three studies were conducted in rats. For the histopathology outcomes (6 studies in rats) the CIs did not overlap and I² was 85% (p-value <0.01), indicating substantial heterogeneity. The results of five out of the six studies favored probiotics. However, the effect size (odds ratios) varied greatly among those five studies. For histopathology, inconsistency is a concern and was a reason for downgrading.

To judge imprecision, GRADE suggests focusing on the 95% CIs around the effect estimate and calculating the optimal information size (OIS, the number of patients required for an adequately powered individual trial). Results are imprecise when based on relatively few patients/animals and few events, which usually leads to wide CIs around the (summary) estimate of the effect. For SRs, results are precise if the OIS is met and the CI excludes no effect, or if the CI overlaps with no effect and the CI is very narrow (precise estimate of no effect). GRADE suggests rating down if the OIS is not met, or if the OIS is met and the CI overlaps no effect and fails to exclude important benefit or important harm. This implies setting thresholds for clinically relevant differences [17].

The most important issues for animal studies are how to calculate the OIS and set meaningful thresholds for clinical relevance. In animal intervention studies the experimental unit can be the cage, as for ethical welfare reasons individual housing of the animals is not always possible [34]. This resembles cluster randomization in human research, for which power calculation methods are available. Taking the experimental unit into account in calculating the OIS in animal studies needs further exploration.

Furthermore, as explained in step 4, in interpreting the results of preclinical animal studies, the direction of the effect is often perceived to be more important than the exact magnitude of effect. In that case judgment of imprecision will be based on whether the confidence interval includes no effect or not.

If effect size matters, one could consider categories of effect size (e.g. SMD effect sizes below 0.2 are small, between 0.2 and 0.5 moderate and above 0.8 large). While absolute rules are not available, raters could consider downgrading for imprecision if the CI overlaps two or more categories; in such instances an appropriate explanation should be provided. Another suggestion could be to set the thresholds based on the effect sizes of effective drugs, if such information is available.

As for GRADE applied to human research setting thresholds for appreciable benefit or harm remains challenging [42]–some might argue how relevant and translatable a clinical threshold for animal evidence is. Another challenge, analogous to that in human studies, is describing imprecision when a meta-analysis is not undertaken.

In the probiotics for pancreatitis example, the 95% CI around the Odds Ratio (OR) for mortality was wide, ranging from 0.24 to 1.22 and, applying GRADE’s guidance it included appreciable health benefit and harm possibly altering a clinical decision. The total sample size was small (102 animals in 3 studies). Thus, one would certainly rate down for imprecision.

Publication bias is a systematic underestimation or overestimation of the underlying beneficial or harmful effect due to the selective publication of studies. GRADE suggests considering downgrading for publication bias if the majority of studies was relatively small with positive results and commercially funded or if the review authors failed in conducting a comprehensive search.

Sena et al used the CAMARADES database to identify systematic reviews of animal studies of acute ischemic stroke and investigated the existence and impact of publication bias in these SRs. Based on Egger regression and trim-and-fill analysis the authors concluded that publication bias was likely and treatment efficacy was over estimated [43]. Korevaar and colleagues investigated how often systematic reviewers of laboratory animal experiments evaluate publication bias. Of the 35 included meta-analyses publication bias was considered in 74%, and in 60% the authors tried to formally assess it by funnel plot only or funnel plot and Egger’s test [44, 45]. Whether publication bias in preclinical animal studies can be assessed reliably similar to the clinical setting is open for discussion. The lack of formal registration of planned animal studies and generally small numbers of included studies pose challenges for the assessment of publication bias [46].

In the probiotics for pancreatitis example, we could not formally assess publication bias because of the small number of studies. It might be worthwhile to also consider the grey literature, contacting experts about unpublished studies and reviewing conference abstracts, which we did not do for this example.

Step 5b: Assess indirectness.

Direct evidence comes from research that directly compares the interventions in which we are interested when applied to the populations in which we are interested and measures outcomes important to patients [22]. Certainty in the evidence may decrease when substantial differences exist between the population, the intervention, or the outcomes measured in relevant research studies and those under consideration in the clinical PICO in a SR.

In the context of GRADE for animal studies two layers of indirectness are proposed: the first layer considers indirectness from preclinical animal studies to the preclinical animal PICO. The second layer considers indirectness from animal models (preclinical animal studies) to humans (clinical PICO). This second layer of indirectness is called translatability (see Box 2).

An alternative approach is to consider all indirectness as one factor and to relate the evidence that was found directly to the clinical PICO, as the purpose of synthesizing the animal research was answering a clinical question.

S2 File provides a preliminary list of items related to indirectness. This list is largely based on the work by Henderson et al who identified threats to external/construct validity in preclinical research guidelines [47] and supplemented by our own findings.

An important challenge is to make a judgement about the choice of animal model. For example, does a ‘lower’ animal model representing the same metastasis-associated protein as humans (e.g. transgenic mouse) better reflect clinical pathophysiology than a ‘higher’ animal model expressing the species-specific metastasis-associated protein (e.g. pig)? Different animal models represent different aspects of the disease, and animal models that reflect all aspects of the clinical disease in one model are rare.

In the probiotics for pancreatitis example, there was serious indirectness. For the first layer of the two-layer approach, the timing of inducement of disease causes indirectness, as in some studies probiotics were administered before onset of the disease. The preclinical PICO aimed for probiotic treatment in animals with acute pancreatitis. In addition, one can debate if some of the animal experiments used a model for multi organ failure instead of acute pancreatitis.

For the second layer, indirectness of preclinical evidence to clinical PICO (translatability), histological damage and bacterial translocation were assessed as a measure for loss of function and infectious complications. However, these are surrogate outcomes for patient important outcomes, e.g., histological damage does not necessarily mean loss of function. In the clinical context, our question addresses probiotics administered to patients with severe acute pancreatitis, whereas in most animal studies the probiotics were administered before induction of pancreatitis. Acute pancreatitis was induced in various ways in the included animal models (e.g. sodium taurocholate intradermal, arginine intraperitoneal, glycodeoxycholate intradermal + cerulein intravenous, etc). Because there is no guidance available yet on how to decide which animal model reflects the clinical situation best, we did not take this into account in this example. This guidance will be developed and published in a future paper.

Step 5c: Assess upgrading.

In the GRADE approach, non-randomized (observational) studies start as low certainty evidence, because of the high risk of confounding bias. In some situations however, upgrading the certainty from low to moderate (and perhaps even to high) certainty is appropriate.[23] The factors for upgrading include large magnitude of effect, presence of dose response relationship and opposing direction of plausible residual confounding.

Although most preclinical animal studies are experiments, upgrading may be relevant. The concept of upgrading might be different and warrants further research. For example, upgrading could be relevant where effects are consistent across different species. The US Food and Drug Administration (FDA) prefers and, in many cases requires testing of candidate drugs on at least two species (one rodent and one non-rodent) before moving on to human clinical trials [48]. In our expert meetings and in the environmental health field, consistency in the results between animal species and models is suggested as an upgrading factor [49]. A point of discussion is whether consistency across species should be seen as an upgrading factor, or as a component of inconsistency, indirectness/translatability.

Step 5d: Rating the certainty in the evidence.

To arrive at a rating of the certainty in the evidence (high, moderate, low or very low), all eight factors need to be considered [50]. Similar to GRADE for human studies, the judgment on integrating the factors should be made explicit and transparent. A question could be, for example, how indirectness should be weighted in the overall rating of the evidence.

In the probiotics for pancreatitis example, we had serious concerns about risk of bias and imprecision for the outcome mortality. As a result, we downgraded the evidence by two levels, from high to low certainty in the evidence. We downgraded the evidence by an additional level, from low to very low, because of indirectness.