Policymaker guidance for hazardous biological AI capabilities to date

In part to address these concerns, the governments of the United States and United Kingdom are working with scientists and model developers to take new steps toward designing AI biosecurity evaluations [19]. Evaluations in this context refer to techniques for assessing an AI models’ capabilities, especially capabilities that can cause harm [20]. AI evaluations to date have included automated tasks (e.g., multiple choice questions), dynamic studies in which humans or other AI models attempt to elicit harmful capabilities (“red-teaming”), and randomized trials in which individuals or groups are set to a task with or without access to an AI model (“human uplift studies”), among other methods [20].

Neither the UK nor US governments have yet released standardized biosecurity evaluation methods, though both have indicated their intention to do so. In September 2023, the UK government was the first to establish an expert-led taskforce to address risks from frontier AI models [21]. Subsequently in November 2023, they launched the UK AI Safety Institute, which is responsible for creating and conducting evaluations on leading AI models. The Institute is especially focused on “containing risks that pose significant large-scale harm if left unchecked: chemical and biological capabilities, and cyber offense capabilities” [22]. It has recently released an open-source framework for LLM evaluations called Inspect [23]. The US and UK (alongside several other nations) also signed the Bletchley declaration in November 2023, acknowledging the potential risks of AI [24].

Concurrently in October 2023, the White House promulgated the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (2023 AI EO) [25]. The 2023 AI EO directed various federal agencies with creating “robust, reliable, repeatable and standardized evaluations of AI systems,” with a planned focus on evaluating hazardous biological AI model capabilities among a small number of other significant risks. A subsequent National Security Memorandum published in October 2024 further directed the Department of Defense to “advance classified evaluations of advanced AI models’ capacity to generate or exacerbate deliberate (…) biological threats” [26]. Of note, the 2023 AI EO focused on foundation models—extremely large AI models trained on broad data, containing at least tens of billions of parameters, and exhibiting high levels of performance on various tasks. Of particular concern was “substantially lowering the barrier of entry for non-experts to design, synthesize, acquire, or use chemical, biological, radiological or nuclear (CBRN) weapons”. The 2023 AI EO also called for investing in the use of AI to mitigate such threats. With regards to foundation models trained on primarily biological sequence data, those models which use greater than 10²³ FLOPS of computational power were required to be reported to the US Department of Commerce. Since then, genomic foundation models exceeding this threshold have been created [27,28].

Under the 2023 AI EO, the US National Institute of Standards and Technology (NIST) launched the US AI Safety Institute, which is charged with proposing “guidance and benchmarks for evaluating and auditing AI capabilities with a focus on capabilities through which AI could cause harm,” including by biological means [29]. In May 2024 the US Office of Science and Technology Policy also recommended oversight of dual-use computational models that could enable the design of novel biological agents or enhanced pandemic pathogens [30], given the high-consequence risks these agents and pathogens could pose to the public. In January 2025, the incoming US administration announced the revocation of the 2023 AI EO and thus the removal of reporting requirements for models surpassing the above computational thresholds [31], however, the US AI Safety Institute remains in place.

Additional governmental bodies are continuing to become involved in AI governance. In 2024, the European Union launched an AI Office to monitor, supervise and enforce the EU AI Act, which governs general-purpose models with systemic risk, though models used exclusively for scientific research are excluded from oversight [32].

Prior experience studying dual-use life sciences research can inform AI capabilities of concern

There is no common AI industry approach to evaluating biological AI models for risks. There is, however, prior guidance from scientists, public health professionals, and policymakers regarding life sciences research that “could be directly misapplied to pose a significant threat with broad potential consequences to public health and safety,” known as dual-use research of concern [33]. There is also guidance regarding life sciences research that could result in “enhanced potential pandemic pathogens” [34]. A recent 2024 update to these US policies incorporates lessons learned from prior attempts to govern this realm of research, makes substantial improvements, and includes recommendations to address risks from computational approaches in this domain [30].

The AI model biosecurity conundrum—how to retain AI’s significant benefits while heading off serious concerns around AI model misuse—finds a ready parallel with dual-use research of concern and research intended to create enhanced potential pandemic pathogens. Scientists and policymakers have spent years analyzing which forms of life-sciences research pose serious risks through accidental release, or inadvertent or deliberate misuse [33,35–38]. Although these recommendations have historically been targeted at wet-lab experimentation, they have also been used to assess what information should be shared publicly [33,39,40], also known as information hazards [41]—the type of concern that is critical to consider for AI models with biological information and capabilities. As the AI community develops evaluations, they should take advantage of the scientific expertise and governmental experience instantiated in prior dual-use study, along with incorporating improvements based on how these prior recommendations were insufficient.

Fundamentally, biological AI models aim to greatly facilitate complex wet-lab work, or do entirely in silico that which can only be done now in vitro or in vivo—and in doing so, make it easier for those with access to a relevant model to reduce or dispense with time-consuming and expensive steps. Because biological systems are complex, traditional dual-use research (e.g., studies analyzing pathogen features such as tropism, transmissibility, and virulence) has historically relied on trial and error, methodical experimentation, or directed evolution [36]. AI may allow users to conduct this research faster and at lower cost, as it has in other related research domains [42–46].

Evaluations should assess capabilities, not specific pathogens or threat scenarios

Many biosafety and biosecurity governance approaches have in the past relied on taxonomic lists of specific pathogens to be regulated [47–49]. However, we recommend that evaluation approaches instead focus on AI-enabled capabilities, rather than AI engagement with risks related to specific pathogens. When applying biosecurity in practice, pathogen lists are “both too specific and too ambiguous for many of the uses to which they are applied” [50]. Experts have recognized the shortcomings of pathogen lists for over a decade, and have instead emphasized oversight based on the intention of the research being conducted (for example, an intention to increase a pathogen’s transmissibility) [51]. Many National Academies assessments [35,36,52] seek to avoid taxonomic pathogen lists, given the modular nature of modern biotechnology, which increasingly makes use of parts of organisms to confer new traits. One influential report championed “adopting a broader perspective on the threat spectrum” and urged policymakers to recognize “the limitations inherent in any agent-specific threat list.” Biosecurity measures, the authors argued, should focus instead on the “intrinsic properties of pathogens and toxins that render them a threat, and how such properties … could be manipulated by evolving technologies” [52]. The U.S. National Science Advisory Board for Biosecurity, as well, in 2024 underlined the importance of reviewing experiments for dual-use potential by analyzing whether the experiment involves risky pathogenic characteristics rather than whether it involves specific pathogens and experimental methods [53]. The recently released United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential also places highest priority focus (Category 2) on experiments capable of enhancing the pandemic potential of a pathogen “such that it may pose a significant threat to public health” regardless of the specific progenitor pathogen [30].

When it comes to AI evaluations, too, this capabilities-based approach should be pursued as compared to a list-based approach. Although pathogen list-based approaches create bright lines that make policies easy to follow, they also lack the flexibility to account for emerging technology developments. Such lists can also provide an unwarranted sense of security, reducing vigilance and surveillance of the technological horizon for newly emerging capabilities and serious risks [52]. Taxonomic lists remain useful tools for controlling access to whole organisms, such as for law enforcement or via export controls [50]. However, in the setting of AI where novel risks must be anticipated, capability assessments and anticipating high-consequence harms to the public, not specified pathogens, are better suited to inform risk assessments. As the National Academies panel co-chaired by Stanley Lemon and David Relman (the Lemon-Relman Report) concluded, it is futile to predict how exactly future terrorists or malicious states will attempt to misuse biology [52].

Prioritize evaluations of biological capabilities that enable high-consequence harms

As governments and AI developers design capability-based AI model biosecurity evaluations, they should focus first on capabilities which enable high-consequence harms to the public: that is, those capabilities which could enable, accelerate, or simplify the creation of transmissible biological constructs that could lead to human pandemics, or similar pandemic-like events in animals, plants or the environment.

It will be a significant challenge to evaluate AI models for their ability to contribute to any possible biology-related accident or misdeed, no matter how limited the consequences. Given the effort required, evaluations should first prioritize assessing the highest consequence harms to the public, such as pandemics. We recognize this prioritization differs from traditional biosafety practices, which give greater attention to less consequential, but more common and expected risks. However, we believe prioritization of high consequence is warranted given the large-scale harms to the public that can result from transmissible, pandemic-capable pathogens, as has been noted by other researchers [54]. This prioritization of pandemic-capable pathogens is also reflected in the United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential, where highest priority for oversight is given to Category 2 research involving potential pandemic pathogens [30]. Non-transmissible pathogens and toxins, while also potentially dangerous to individuals and even large groups of people, are self-limited. In contrast, the unchecked spread of transmissible, pandemic-capable pathogens can strain health systems and require severe containment measures which also contribute to public harm; therefore, prevention should be prioritized.

Such prioritizing within biological AI model evaluations would, if implemented correctly, help to limit widespread access to the most concerning models, or require model safety modifications before deployment for others. As experts have long recognized, attempts to constrain the flow of scientific information, especially in biology, face acute practical and legal challenges [35,52]. It is thus all the more important for biosecurity evaluations and responses in the AI setting to use a scalpel rather than an ax, and to focus effort and resources first on prevention of the in silico and laboratory creation of high-consequence harms rather than control of such harms after-the-fact.

Below, we elaborate on how previously identified dual-use capabilities in the life sciences could inform tangible and testable components of biological AI model evaluations. Table 1 lists previously identified categories of dual-use research in the life sciences. We have generated examples of emerging AI “capabilities of concern” corresponding to each category. Of the examples listed, only one has been fully realized and scientifically validated—the optimization and generation of viral serotypes capable of evading immunity [55]. The design of molecules with increased toxicity has been explored computationally [56,57], and several other emerging AI capabilities of concern are under active development.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Categories of dual-use research in the life sciences warranting risk assessment, as previously identified by scientists and US policymakers [30], and corresponding emerging AI capabilities. Note that the emerging AI-enabled capabilities constitute an illustrative, not exhaustive, list.

https://doi.org/10.1371/journal.pcbi.1012975.t001