Ensuring the most appropriate operational data are utilised for ALS decision-making.

Every jurisdiction reported significant challenges accessing the data needed for their ALS. Challenges shifted over the course of the pandemic, with the early phases affected by limited diagnostic capacity and later pandemic phases characterised by undetected and unreported cases and data systems becoming ‘overwhelmed’ (Interviewee 7). Health system capacity was increasingly viewed as the most critical indicator of epidemic severity, and several jurisdictions attempted to estimate incidence and set ALS levels by relying on proxy data such as COVID-19 death rates, ICU bed capacity, or modelling outputs to assess risk.

“The consequences of that is that we were in some ways… making a lot of assumptions or best guesses… about various issues during the pandemic, and our decisions about moving to a higher or lower alert level, even remaining on an alert level, were dictated to by the quality of evidence.” (Interviewee 1).

Alongside these challenges, several respondents also described the difficulty of balancing epidemiologic, economic, and social trade-offs when setting levels. While South Africa’s ALS took account of economic as well as epidemiological concerns, [8] data availability is also likely to have had an impact on the approach to structuring ALS decision-making, as the lack of social or economic data relative to available epidemiologic data hindered precautionary action through assessment of both sets of risks. [12]

Setting clear objectives for ALS.

Some respondents felt that a lack of clear objectives for their ALS contributed to difficulties determining the right data to use to set levels. The emergent pace of implementation and limited time to plan strategically contributed to this lack of clarity, and therefore appropriate decision-making about relevant data streams.

“Be absolutely clear on what your intended outcome is… what are you trying to do with establishing the alert level [system]… And be clear on how the data actually relate to that outcome.” (Interviewee 12)

Shifting the objective of the ALS without appropriately redesigning the system led to challenges with Singapore’s DORSCON system, which had been conceived as an internal coordination tool. [6,13] DORSCON set levels based on a risk assessment that used pre-determined criteria that prioritised risk aversion, and this led to problems when DORSCON was repurposed for use as a public communication tool.

“So when we did risk assessment[s], while we looked at the nature of the disease, the disease transmissibility, the severity… these are not really salient to the public actually – and then there were also other risk factors we needed to account for like the availability of, for example, medical countermeasures, the ability of whether our healthcare system could cope, the effectiveness of our public health measures and… DORSCON didn't take [that] into consideration which was why it [was] kept at orange for over two years.” (Interviewee 6)

This meant that until substantial population immunity from infections or vaccination had been attained, the underlying risk assessment didn’t change. There was also a reluctance to escalate to the top level (red) for fear of causing public alarm. Taken together, these features led DORSCON to be ‘stuck’ at orange for two years of the pandemic. Beyond challenges related to identifying appropriate data and setting objectives, respondents also reflected on how PHSMs linked to ALS were determined and implemented in practice.

Feasibility and practicality of PHSMs.

Several respondents discussed limitations associated with asking community members to adhere to measures for which there were structural barriers to implementation and noted the consequences for effective epidemic control. In Brazil and South Africa, decision-makers had to contend with local social structures and economic constraints that limited the extent to which PHSMs could feasibly be implemented.

“So we had thought ‘we have a significant proportion of the population living in what we describe as informal settlements.’ So social distancing and being able to have ventilation is not a feasible option for them. So you can provide them with all that advice but it’s really impossible.” (Interviewee 1)

In higher-income settings, social and economic considerations were less limiting to the structure and reach of ALS-associated measures, since in some circumstances the provision of resources to facilitate adherence to measures was more available to policymakers.

In South Africa, the structural limitations on peoples’ ability to adhere to more restrictive measures meant that there was a risk that the ALS lost credibility when it recommended measures that community members fundamentally couldn’t implement.

“And some of the measures that we advocated were not relevant and not applicable… so if you're living in an environment where you have six people in a tin shack, social measures are not relevant.” (Interviewee 2)

Conversely, in the Philippines, the provision of structural support to those in lockdown or quarantine was perceived by one respondent to have strengthened the social contract, built trust in public authorities, and improved willingness to adhere.

“Because of the burden of the increasing number of cases of COVID at the height of pandemic, they were forced to be at home and not go out… And then they saw that the government is also responding and providing what they need, the basic needs… [and] so we get… the trust of the public.” (Interviewee 14)

One UK respondent noted the negative epidemiological outcomes associated with socio-economic disparities in the country and differences in the support systems available to community members.

“One area that came up throughout quite a lot of work was … what limited support [region x] sometimes had, for example. So [region x] might always be running quite hot because it doesn' have so much support around it” (Interviewee 5).

The provision of enabling support to adhere to PHSMs was therefore described as having instrumental value for disease control efforts, as well as an important intrinsic value for trust-building and community engagement. [14,15]

Evidence-based PHSMs for improved response credibility.

Several interviewees reported that some mandated measures were not based in public health science and were therefore perceived by the public to be ‘irrational,’ damaging the credibility of the overall epidemic response. In New Zealand, one example arose around a lack of differentiation between measures for indoor and outdoor events.

“As the pandemic had worn on, people started to question the logic of some of the measures. And, you know, an obvious one was [that] outdoor events were subject to the same restrictions as indoor ones, because of ventilation and other things. And so, you know, [people] started to sort of question the logic of that.” (Interviewee 8)

An interviewee from South Africa noted that some measures had no basis in evidence-based outbreak responses.

“There was a regulation that said we couldn't wear open-toed shoes. Now where in the world is there any science that says, open toe shoe would expose you to… even with Ebola unless you're working with the patient who's bleeding profusely then your open toe shoes would be a risk!” (Interviewee 2)

These ‘irrational’ measures were implemented for several reasons in different geographies, with the early days of the pandemic often seeing them driven by uncertainty about COVID-19 transmission mechanisms and therefore a tendency towards risk aversion among decision-makers. In addition to questions about the feasibility and evidentiary basis of PHSMs, respondents emphasised that the legitimacy and effectiveness of ALS depended heavily on community engagement and trust.

Decision-making transparency and public trust.

Perceptions of ALS decision-making transparency affected trust and credibility; the importance of transparent decision-making for trust-building has been demonstrated previously. [16,17] One South African respondent noted how a lack of transparency negatively affected public perceptions.

“[There was a] lot of unhappiness with what people initially saw as the opacity of the decision-making process between the ministerial committee and cabinet decisions, because the advice from the ministerial committee initially was not made public.” (Interviewee 4)

Moreover, it may have been important to consider the reputation of those engaged in decision-making, and the extent to which those actors might be perceived to be motivated towards transparency. In the UK, the decision to develop an ALS was influenced by the perception that similar tools had successfully managed terror threats, and as a result, intelligence community staff were brought into a novel agency called the Joint Biosecurity Centre (JBC) to develop an ALS for COVID-19. This participation of intelligence staff in ALS development may have affected public trust. [9]

“The people who were initially brought in to set up the JBC - there were quite a few people who came from the intelligence community. Because there were two figureheads who the media were quite interested in, it started to take a little bit of a negative slump” (Interviewee 17).

Many jurisdictions reported expending substantial effort on data transparency to circumnavigate accusations of opaque decision-making and to enhance trust. The US Center for Disease Control and Prevention’s (CDC) COVID Community Levels avoided setting levels based on predictive modelling since the complexity of the model ‘back end’ would obfuscate decision-making to the public at a time when trust-building was paramount.

“[The public] want to know, like, where are we going? And at that point, with the amount of distress in the public, I don't think… that was the more quantitatively complex models, and we didn’t have the public trust really to be able to promulgate those.” (Interviewee 10)

Several respondents cited data and methodological transparency as one of the bigger successes of their ALS.

“And it probably has been a paradigm shift for public health response to major incidents, as in public expectations [for] transparency has never been higher. [It] demonstrated what can be done. So, I think it’s hard to describe that as anything other than a big success. And one that was worth the resource invested.” (Interviewee 16)

Separation of technical teams and political decision-makers.

All jurisdictions reported having clear separation between the technical team analysing data, conducting risk assessments, and making recommendations (typically seated within Ministries of Health or associated agencies), and policy or political decision-makers who had the final say on any level-change. This was often because decision-makers had to consider factors beyond public health, which sometimes meant a deviation from technical team recommendations.

“So the risk assessment was what we did behind the scenes, the public health risk assessment. And then we determined from the public health risk assessment, we put a recommendation up as to which alert level we thought the government should adopt. They didn’t always take our advice. Sometimes they went more and sometimes they went less.” (Interviewee 8)

The separation of technical teams and political decision-makers helped to protect decisions (and scientists themselves) from ‘interference’ (Interviewee 17) and the political pressures that arose during very polarised debates about pandemic decision-making.

“At the time, I had a permanent concern about how we position ourselves. And in the end, I think [we needed to] focus as much as possible on a scientific question, to look at the data. We are following what was determined here. We had a better way to defend ourselves against these polarized parties.” (Interviewee 11)

By leveraging the perceived neutrality of scientists, difficult decisions were more easily justified to the public, which was further strengthened by transparent data-based decisions that could be publicly scrutinised.

“So it was a government decision then around the alert level that… you could clearly see what the rationale and the articulation of the advice was by them.” (Interviewee 9)

However, some key informants described challenges with this approach caused by an insufficient mix of technical disciplines within their technical team, which occasionally led to out-of-touch or disciplinarily siloed recommendations.

“A lot of our scientific staff… could be impractical. And, you know, wanted to do it in the most epidemiologically rigorous way, which… can become almost essentially unusable to most members of the general public.” (Interviewee 15)