Evidence-based medicine as a reference class

The independent Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWiG) is responsible for examining the benefits and harms of medical interventions for patients in Germany and describes EBM as the international standard in medicine for the assessment of a medical intervention’s benefits and harms based on “clinical studies and their systematic identification and assessment” [12]. An epistemological principle of EBM is that ”not all evidence is created equal” and there is generally a hierarchy of the quality of evidence with information gathered by controlled clinical observations being most trustworthy, followed by evidence from uncontrolled ones and then biological experiments or individual clinician’s experience on the lower end of trustworthiness [13], though exceptions exist.

The IQWiG describes the possibility to evaluate the extent to which the available evidence is reliable as a specific feature of EBM with the certainty of results of clinical studies depending primarily on internal validity (determined by study design and conduct), external validity (depending on the intervention-choice and selection of included populations) and statistical precision (influenced by study-size and reliability of the recording of outcomes) [12]. There are many interesting aspects to analyze in the context of the global convironmental crisis from an EBM perspective – for instance ethical questions about informed consent in this context. However, the main point focused herein is the absence of any clinical data from planets with civilization similar to ours having undergone comparable changes.

To put this lack of any observational data over the whole course of the disease into perspective, the assessment of drugs by the IQWiG relies on Randomized Controlled Trials (RCT) and uses studies performing non-randomized interventions or just observations only in “justified exceptional cases” [12]. This underlines how unusual the clinical situation for the convironmental disease that humanity is facing seems from the perspective of EBM, since the quality of the data we are using and relying on does not reach the level of low-quality-data that medicine uses only in ‘justified exceptional cases’ due to lack of better data. The clinical data is not just of exceptionally low quality - it does not exist apart from the running singular index case, our current metamorphosis. Furthermore, even data of low clinical quality cannot be generated in any way due to the lack of other Earths facing similar challenges that could be observed and treated over the relevant time-frames. The IQWiG advises to “describe and classify the situation” in situations where “studies with the required validity and precision are generally lacking” [12]. Since this is true for the convironmental disease, this description and classification is trying to be done hereafter.

Current treatment approaches

Quantitative aspects.

Before continuing with the qualitative analysis I give two quantitative examples hinting at the implications for adjusting the current baseline predictions concerning risks of the convironmental disease and its treatment:

It can be pointed at how serious the importance of proper clinical data is taken in EBM by looking at the approval process of COVID-19 vaccines. Preclinical data and the early clinical data suggested high effectiveness of the vaccines with few side-effects – mainly “pain at the injection site, fatigue, and headache” [14]. Nevertheless the European Medicines Agency (EMA) did not approve it until the RCT, with more than 21.000 participants in both control and intervention group, was finished and analyzed by the Institution itself to ensure that „robust scientific opinions are reached“ [15]. This was despite being faced with arguably strong public and political pressure to speed up the decision.

As a second important quantitative aspect, the risk assessments between clinical and non-clinical research fields seem to be very different in scale. For example, the Intergovernmental Panel on Climate Change (IPCC) uses its likelihood language [16] for statements on risks concerning climate change, where a 0–10% probability is considered “very unlikely” whereas the EMA describes a 1–10% likelihood for a certain side-effect of a drug as “common” [17]. Of note, the later is a quantified statement based on real-world cases whereas the former is a likelihood estimation about a risk statement being true, which is not the same. The likelihood language is here used as a comparison because it seems the closest that the IPCC comes to the risk communication stemming from applied data that can be measured in clinical research trials in medicine to assess and quantify risks. Acknowledging that limitation, the IPCC is using a language that seems to vastly downplay clinical risks from a practitioner’s perspective, which can also be hinted at with the following example. The IPCC subsumes “not well known” probabilities “whose potential impacts on society and ecosystems could be high” into the category of “Low-likelihood, high impact outcomes” (see Glossary: [18]). Thereby the IPCC is implying a low-likelihood for these risks without quantitative evidence instead of communicating clearly that those probabilities are “not known” as the EMA does [17]. Simultaneously, it implies that all the other effects subsumed under this category are low-likelihood based on preclinical evidence, which is a very high stakes gamble from a practitioners perspective (Table 1).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Language concerning the likelihood of scientific statements on risks being true (IPCC) and side-effects happening (EMA) (in percent). [16,17].

https://doi.org/10.1371/journal.pclm.0000896.t001

Clinically speaking the ongoing ‘disease’ can be likened to the daunting task of treating a single specimen of a species novel to us, undergoing a range of unprecedented and life-threatening changes, without exterior advice from experienced practitioners and no clinical data available to us that includes real-world feedbacks and processes. The later two would both seem essential from an EBM perspective. All of this while our own human lives depend on a successful treatment, making us simultaneously provider and recipient of the medical care.

Facing the scale of the problem

What is striking when taking a clinicians view – besides the overconfidence in preclinical data - is that the whole scale of the convironmental crisis is rarely being scientifically tackled, possibly because it may seem too big to handle with conventional scientific approaches or the data is simply lacking. Rather than that, a reductionist approach is being chosen by analyzing subunits of the crisis, for example the climate crisis and the biodiversity crisis, which demonstrate this point. Additionally, these two exemplary research communities sat down officially for the first time together only in 2020 [29]. Intensified and systematical cooperation of both research communities is certainly a step in the right direction, since the situation is comparable to two medical practitioners from different specializations treating the same critically ill patient. However, it also shows that in this case these two specialists sat down together in earnest to plan the treatment only recently after the patient’s condition has been slowly worsening for ‘weeks’ (which are real-world decades in this comparison). This is a worrying sign that we are generally not yet facing the problem on the scale it is occurring, zooming in where it would be vital to zoom out. Furthermore, the biodiversity research community decided to develop their own scenario framework [30] because they found that the IPCC scenarios were lacking crucial information in regard to the relationship between nature and people [31]. This shows that just by looking at these two important components of the overall disease to be treated, we currently do not manage to align our understanding and response to the different parts of the convironmental crisis with each other.

This is despite knowledge that “more is different” [32], which transferred to this context implies that a reductionist approach to treating the different components of the convironmental crisis does not in any way imply a constructionist one, where these separate treatments would lead to a successful overall approach if simply added together. This would even apply if we imagined to know all the components of the crisis and had a usable solution to each of them separately, both of which is certainly untrue. Aside from not having clinical data for investigation anyway, the network effects and feedbacks between the different components of the crisis are being excluded from examination in this way. This leads to much higher uncertainties and risks, while it simultaneously has to be expected that humanity as a collective may not fail on most or any of the components that we have to tackle. To draw on a comparison to the sometimes evoked metaphor of climate change as a form of ‘Russian roulette’: the global convironmental crisis would be a form of ‘Russian roulette’ in which there are multiple guns, with their exact amount and type unknown, we do not know how many and which type of bullets are in them, they are interconnected in many and partially unknown ways so that for instance firing one may trigger another or even set off a chain reaction, they move and evolve themselves dynamically, some may fire due to indirect effects without us pulling a trigger and some are invisible to us.

There are notable exceptions from this tendency to reduce the problem into parts that are better analyzable with current mainstream methods. These approaches, some already mentioned, look for instance at systemic risks [33] or possibilities of systemic derailment [5]. On a political level the United Nations Sustainable Development Goals are such an approach [34] though it has important limitations, like not setting any measurable targets on climate change. Additionally, the midterm report on their progress showed that none of the 17 goals is going to be reached based on current levels of implementations [35]. However, while the scale of the problem is tackled within these approaches, the crucial importance of lacking real-world data that is addressed in this article plays no relevant role within them.

Mathematical reinforcement of the argument

Why do the current scientific uncertainties concerning risks seem to differ from a clinicians perspective as shown in this essay? To contextualize: mathematical uncertainty can be seen as a relationship between an individual (or epistemic community, as in this case) and their world that “depends on the subjective perspective and knowledge of the observer” [36]. It is known that even analyzing the same data and hypothesis scientists can come to greatly varying results revealing “a hidden universe of uncertainty” [37]. Transferred to the content of this article, this can help to get across why large scale errors in current understanding of our convironmental crisis that may be surprising to a theorist researching the convironmental crisis or components thereof - based on preclinical data, assumptions and models -, may not be surprising to a clinical practitioner with practical experience in the limitation of preclinical data, but on the contrary something to be highly expectable.

This type fallacy of over-relying on measurable and decontextualized evidence projected into the future is known from other fields. It has been named “McNamara fallacy” in the military context after a United States Secretary of Defense during the Vietnam war, who collected as much data as possible and all the numbers seemed to imply that the United States were winning the war, while reality clearly did not comply to these data-sets and the associated models [38].

Another mathematical argument can be described with the metaphor of the Black Swan: rare events with extreme impact and only retrospective predictability [39]. These are non-objective phenomena resulting from epistemic limitations, which can be psychological, philosophical, mathematical, individual and collective, resulting in situations where “what you don’t know [is] far more relevant than what you do know” [39]. Therefore, in a world that seems to be drowning in data, the scientifically counterintuitive approach taken herein of looking at missing real-world data that additionally cannot be generated should be expected to be vital for successfully targeting the convironmental crisis from a mathematical perspective, too. As Taleb argues “we need to be hyper-conservationists ecologically, since we do not know what we are harming with now. That’s the sound policy under conditions of ignorance and epistemic uncertainty.” [39].

We should better acknowledge known human limitations in performing large scale and long term forecasting [40]. If long term human survival and possible human prosperity are relevant societal targets we should leave our Earth system more room for resilience instead of continuing to force it to novel extremes in a “Great Acceleration”, as we have for decades [41]. Mathematically speaking, in tackling the convironmental crisis, we are currently ignoring limitations of the available data, knowledge and capabilities, while simultaneously building a breeding station for Black Swans.