OpenABM-Covid19—An agent-based model for non-pharmaceutical interventions against COVID-19 including contact tracing

SARS-CoV-2 has spread across the world, causing high mortality and unprecedented restrictions on social and economic activity. Policymakers are assessing how best to navigate through the ongoing epidemic, with computational models being used to predict the spread of infection and assess the impact of public health measures. Here, we present OpenABM-Covid19: an agent-based simulation of the epidemic including detailed age-stratification and realistic social networks. By default the model is parameterised to UK demographics and calibrated to the UK epidemic, however, it can easily be re-parameterised for other countries. OpenABM-Covid19 can evaluate non-pharmaceutical interventions, including both manual and digital contact tracing, and vaccination programmes. It can simulate a population of 1 million people in seconds per day, allowing parameter sweeps and formal statistical model-based inference. The code is open-source and has been developed by teams both inside and outside academia, with an emphasis on formal testing, documentation, modularity and transparency. A key feature of OpenABM-Covid19 are its Python and R interfaces, which has allowed scientists and policymakers to simulate dynamic packages of interventions and help compare options to suppress the COVID-19 epidemic.


Response
[10] -The basic model is not parallelised, however, we do have a meta-model which runs the underlying model on sub-populations and then allows cases to be seeded between populations at the end of each time-step. This is implemented in Python using the multiprocessing module.
The Performance section has been completely rewritten to reflect this and includes a figure on scaling.
Manuscript Change: Line 695, 722, 738, 892, 1105 -line 600 : 1.7 Gb memory for 1 million agents means~ 400 simple precision (or ~ 200 double-precision) values used per agent without contact tracing. What are the main memory consuption to need so much RAM ?

Response[11]
-since the initial submission, work on the code has reduced the memory requirement to 1kb per person. This is split approximately in thirds between storing interactions, networks, and data about individuals. Due to the variable and dynamic number of interactions each user has, interactions are stored as linked-lists requiring 20 bytes per interaction per person (2 longs for pointer to the other individual and the next interaction and 2 shorts for data about traceability of the interaction). Networks need to be stored in addition to interactions because we down-sample them to generate each day's interactions, with each edge requiring 2 longs (16 bytes).
The Performance section has been completely rewritten to reflect this and includes a figure on scaling.

Manuscript Change: Line 738, 1105
-due to high stochastic characteristics of a pandemy evolution (due to the random contacts between agents, and the random contagion from an agent to another), did you study the reproductibility of the simulations, and the range of results obtained with exactly the same parameters ? Some others agent-based approach, which use fixed contact-network, can show a wide variation of the max number of infected people, only due to the random aspect of contagion at each social contact. A study on these variations needs to be added to the manuscrit (These effects are visible for initial conditions with low number of infected people). The results used for decisional help are they an average of many simulations, or only one-shot simulation, without any error bars due to the stochastics effects ? This clarification is imperative, and may need to balance the conclusions obtained by the simulation described in the manuscrit.

Response [12]
: See Response [3] and the accompanying figure which is a study covering the questions asked here.

Manuscript Changes: Lines 251, 1110
-what are the initial conditions ? Some infected at t = 0 ? How many ?

Response [13]:
A small number of individuals are infected at t=0, parameter is n_seed_infection which is set to 5 (see S6 Table).

Reviewer #2:
The paper describes the implementation of an ABM for simulating the spread of Covid-19 within a population of up to 1 million people, although as it is described this number could be increased. Spreading occurs through both static and dynamic contact networks using statistics on contact patterns, which addresses the inhomogeneous spread in different age-groups. Also, disease progression and asymptomatic cases are modelled. The strength of the presented simulation model is the inclusion of a range of different prevention measures in addition to the classic NPIs for contact reduction. As this is a core advantage of this implementation, systematic assessment and testing of intervention strategies should be performed and discussed more deeply.
• The authors state that the model is parameterized for the UK, however, the paper lacks information on the calibration and validation process and how well the model reproduces the historic epidemic curve and efficiency of actual NPIs.

Response [14]:
We have now included a more detailed description of the calibration process in order to obtain the basic fit for the first wave of the epidemic in England.

Manuscript Changes:
A more thorough description of the calibration has been included in the Methods. Figures S22 and S23 have been included to justify these parameter choices. We have also cited the study by Jarvis et al., (2020) highlighting the efficacy of the simulated lockdown after calibration is in line with other studies.
• The model seems to be able to reproduce the characteristics of a certain (?) time interval of the COVID epidemic in the UK. Line 229 "The model provides a good fit to UK data, correctly matching: the cumulative number of deaths; the magnitude of the peak in daily deaths; the timing of the peak in daily deaths to within a few days; and peak hospitalisation to within 25% of the recorded number" This could be supported with additional quantification of errors/correspondence in the comparison between real data and simulation results. How was this correspondence achieved? How good is the quality of fit compared with other models? Several times in the text the authors mention that the model (and parameters) were calibrated, but a description or discussion of the calibration approaches is not found in the manuscript.

Response [16]:
The doubling time of 3.5 days has been used as a part of the calibration to determine a suitable value for the infectious_rate parameter from the early exponential growth. We have added a more thorough description of the calibration approach, including the doubling time, and two figures to highlight different choices of this infectious_rate parameter for different doubling times.
Manuscript Changes: Details on the calibration and two figures on the calibration in the Supplementary Materials.
• Paragraphs with important "messages" should be revised thoroughly. To give a more or less random example in line 600 ff "Performance. The ABM for 1 million individuals takes approximately 3s per day to run…" It is described in the abstract, but missing at this point for what scenario and which time range. Such shortcomings can be resolved easily and can improve the quality of the paper to a large extent.

Response [17]:
The section on Performance has been completely rewritten and an additional figure has been added. See Response [11] for details.

Manuscript Change: Line 738
• Evaluation of results of prevention measures is provided in supporting information only (see comments below). The analysis and assessment of interventions should be systematic with visual displays supporting a thorough discussion. In the current state, intervention scenarios are somehow treated as technical demonstrations.

Response[18]:
The characterisation that the interventions are presented as technical demonstrations is intentional. The aim of this paper is to present a detailed description of both the model and the software interfaces. A systematic analysis of digital contract-tracing strategies has now been published in an article in Nature Digital Medicine. We have added a new paragraph at the start of the Results section outlining both this study and the use of the model by the NHS in the UK. (also see Response [39]).

Manuscript Changes: Lines 160, 905
• The abstract description of the model is well written and insightful, but technical details and modelling decisions are not or provided. The appendix is very useful and the parameters are well described. Even though established approaches in an agent-based simulation of epidemic spread and the state of the art (and its limitations as a motivation for the paper) should be recognized (if it is the goal of the paper to show the methodological improvement). The motivation for modelling decisions and assumptions should be discussed in the presentation of a simulation model. E.g. motivate the use of certain types of networks/graphs and probability distributions! What are the features of the specific mathematical concepts? Why are they suitable to model certain aspects?
Response [20]: Our model was developed first to support the development of digital contact tracing, to support the development of the NHS COVID-19 app, and then has evolved in response to stakeholder requests, from the NHS, from different government departments seeking advice on policies around testing and tracing, and from international stakeholders. A particular strength of our model is the ability to include different types of contact tracing, and it's coupling to realistic epidemiology. This paper presents the model structure which many users have already found convenient for different types of applications, and we feel that a detailed description of the epidemiological questions is out of scope in what is already a long paper. Examples of OpenABM-Covid19 being used to analyse epidemiological questions are now cited in the Results section. We have addressed many of the more technical points throughout our revision.

Response[21]:
The Performance section has been completely rewritten to reflect this and includes a figure on scaling. • How is the model dealing with "unreported cases"? As it is immanently clear, this aspect should be at least mentioned.

Response [24]:
The time-series output of the model reports both total infections (regardless of test status) and total cases (only those who have tested positive.

Manuscript Changes: Line 392
The simulation model is of high quality and shows also high potential. But at the current state, the paper is a conceptual presentation of a simulation model but does not include or present interesting results on one of the three areas 1) epidemiology, 2) HCI and Usability, or 3) technical novelty. The paper should increase focus towards one of those directions. If the model can be adapted as easily as claimed by the authors the model could provide a good framework for additional research for the assessment of NPIs if the model is correctly parameterized for the addressed research question.

Response [25]:
The aim of the paper is to describe the model and its interfaces, and to demonstrate how it can be used to analyse interventions (i.e. your point 2). We believe the strength of our model is its usability, robust code and ability to be extended. To strengthen the work in this direction we have added an R interface to make it usable to a wider audience. A detailed analysis of the performance of the code has been added. Finally, vaccinations have been added to the model showing how easily it can be extended (this feature is being used by NHSE to analyse the effect of the UK vaccine programme on hospital admissions). Whilst this paper does not contain a detailed assessment of a precise epidemiological research question (i.e. your point 1), we now describe and cite work where OpenABM-Covid19 has been used to do this.

Manuscript Changes: None
Model descriptions and modelling process could be set concerning international standards or guidelines like "Modelling Good Research Practices of ISPOR-SMDM" (https://www.sciencedirect.com/science/article/pii/S109830151201652X) . For example "V-8 If using an agent-based model, thoroughly describe the rules governing the agents, the input parameter values, initial conditions, and all sub-models."

Response [28]:
We agree that agent-based models need to be rules for agents should be thoroughly described and parameter values/initial conditions documented. OpenABM-Covid19 is thoroughly documented both on Github (where it has been developed as an open source project) and now described in this paper. All parameters are documented on Github and have been reproduced as Supplementary Tables in this paper.

Reviewer #3: This paper presents a detailed description of the OpenABM-Covid19 model, which is an agent-based COVID-19 transmission model informed by detailed data on contact networks and validated against data from the UK. As a methods paper, it does not include results per se, but rather illustrates the analyses the model can be used for.
Overall, I found the paper to be exceptionally well written and clearly laid out. The model has been carefully conceived, and the use of modern software practices (testing, documentation, concern for adaptability, an easy-to-use Python user interface, etc.) make this study an exemplar for how such models should be developed and communicated. The following comments are mostly intended as nonbinding suggestions for improving the paper.
p. 4, line 56: This could be interpreted to mean that the population size fixed at 1 million, rather than that 1 million is the default.

Response[29]:
Agreed, reworded to make clear this a default population size.
Manuscript Changes: Line 57 p. 5, line 85: I'm not sure I understand how people who are contact-traced are themselves "protected" --wouldn't they be contact traced following exposure to a known positive?

Response [30]:
Due to the clustered nature of social networks, infections tend to be clustered on the network. Therefore, if you have been contact-traced by one of your contacts who is infected, then it is likely that you have other contacts who are infected. These people may be asymptomatic or not isolating, therefore if you contact-traced and quarantine yourself, you are protected from these other locally infected people on your social network.