Theoretical models of disease dynamics on networks can aid our understanding of how infectious diseases spread through a population. Models that incorporate decision-making mechanisms can furthermore capture how behaviour-driven aspects of transmission such as vaccination choices and the use of non-pharmaceutical interventions (NPIs) interact with disease dynamics. However, these two interventions are usually modelled separately. Here, we construct a simulation model of influenza transmission through a contact network, where individuals can choose whether to become vaccinated and/or practice NPIs. These decisions are based on previous experience with the disease, the current state of infection amongst one's contacts, and the personal and social impacts of the choices they make. We find that the interventions interfere with one another: because of negative feedback between intervention uptake and infection prevalence, it is difficult to simultaneously increase uptake of all interventions by changing utilities or perceived risks. However, on account of vaccine efficacy being higher than NPI efficacy, measures to expand NPI practice have only a small net impact on influenza incidence due to strongly mitigating feedback from vaccinating behaviour, whereas expanding vaccine uptake causes a significant net reduction in influenza incidence, despite the reduction of NPI practice in response. As a result, measures that support expansion of only vaccination (such as reducing vaccine cost), or measures that simultaneously support vaccination and NPIs (such as emphasizing harms of influenza infection, or satisfaction from preventing infection in others through both interventions) can significantly reduce influenza incidence, whereas measures that only support expansion of NPI practice (such as making hand sanitizers more available) have little net impact on influenza incidence. (However, measures that improve NPI efficacy may fare better.) We conclude that the impact of interference on programs relying on multiple interventions should be more carefully studied, for both influenza and other infectious diseases.
The spread of infectious diseases can be inhibited by both vaccines and non-pharmaceutical interventions (NPIs) such as hand-washing, respiratory etiquette, and social distancing. Theoretical models of disease spread have incorporated how individuals make decisions concerning these interventions in the face of disease risks and intervention costs. However, most previous models have considered these two interventions separately from one another. Here we combine decision-making processes for both interventions in a single model that simulates influenza spread through a network. Individuals choose interventions based on their past and present experiences with influenza, and the personal and social impacts of their choices. Our model indicates that, due to feedback loops between the interventions via their mutual impact on disease levels, efforts to reduce influenza spread by expanding NPI practice are almost completely mitigated by the resulting drop in vaccine coverage, whereas efforts to expand vaccine coverage are only weakly affected by the response of NPI practice. Furthermore, strategies such as making vaccines more available, stressing the harms of being infected, or stressing the social benefits of preventing infection through both interventions will prevent more disease than only expanding NPI practice through making hand sanitizers more widely available, for example.
Citation: Andrews MA, Bauch CT (2015) Disease Interventions Can Interfere with One Another through Disease-Behaviour Interactions. PLoS Comput Biol 11(6): e1004291. https://doi.org/10.1371/journal.pcbi.1004291
Editor: Sebastian Funk, London School of Hygiene & Tropical Medicine, UNITED KINGDOM
Received: December 5, 2014; Accepted: April 17, 2015; Published: June 5, 2015
Copyright: © 2015 Andrews, Bauch. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: Data are available in the supporting information files.
Funding: This research was supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant to CTB (Grant #: 298305-2009, http://www.nserc-crsng.gc.ca/). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Infectious diseases continue to threaten human health throughout the world [1, 2]. In order to help alleviate these impacts, researchers have utilized mathematical models to improve our understanding of infectious disease dynamics . In many cases, these models assume that human behaviour does not change over time or respond to disease dynamics in epidemiologically relevant ways. However, individual behaviour often does both influence–and evolve in response to–disease dynamics. For example, when vaccination is not mandatory, the prevalence of an infectious disease can depend on individual decisions of whether or not to vaccinate [4, 5]. Other behavioural practices that impact the spread of a disease include non-pharmaceutical interventions (NPIs) [6–10]. For susceptible individuals, NPIs can include hand washing or general avoidance of infectious individuals. For infectious individuals, these can include reducing contact with susceptible contacts, hand washing, or strict respiratory etiquette.
Mathematical models of the behavioral epidemiology of infectious diseases capture interplay between disease dynamics and individual behaviour  (we will refer to these as “disease-behaviour” models hereafter). These and similar types of models have focussed on vaccinating decisions where individual decision-making occurs according to a strategic environment or is determined by some other utility-based or rule-based mechanism [12–20]. Using such frameworks, Bauch , Fu et al. , Salathé and Bonhoeffer , and Reluga et al.  use models with imitation dynamics to predict potential vaccine uptake in populations. Similarly, Xia and Liu  base vaccination decisions not only on minimization of the associated costs, but also the impact that social influence has on each individual. d’Onofrio et al.  use an information dependent model where vaccination decisions are based on private and public information gathered about a disease. Further approaches by Vardavas et al.  incorporate memory of past disease prevalence, and Wells and Bauch  include memory of previous infections to study the effect of these factors on vaccinating behaviour. Other research has explored the impact of increased individual sexual risk behaviour on disease incidence, in response to the introduction of a hypothetical HIV/AIDS vaccine [25, 26], or how risk perception, HIV prevalence and sexual behaviour interact with one another in a core group population .
Other disease-behaviour models incorporate social distancing and other NPI-related behaviours. For example, Reluga  analyzes a differential game in which individuals choose a daily investment in social distancing in order to reduce the risk of infection. Funk et al.  allow information of a disease to spread over a network, and individuals protect themselves according to the quality of information they possess. Gross et al.  and Shaw and Schwartz [31, 32] study adaptive networks, where susceptible nodes rewire their connections from infectious to non-infectious nodes at a certain rate. Along the same vein, Zanette and Risau-Gusman  allow susceptible nodes to either permanently sever a connection with an infectious node, or rewire to another randomly chosen (and possibly infectious) node. Del Valle et al.  assume some individuals lower their contact rates once an epidemic is detected, whereas Glass et al.  and Kelso et al.  use complex contact networks which include families, schools, and workplaces to test differing social distancing methods such as school closures and the effects of staying at home while infectious.
Hence, disease-behaviour models studying either vaccinating behaviour or NPI behaviour separately from one another are relatively abundant, but models incorporating both types behaviour are rare, to our knowledge. However, for many infectious diseases (such as influenza) both NPIs and vaccines are part of infection control strategies, and both also respond to disease dynamics [4, 35]. Under these circumstances, it becomes important to study disease-behaviour interactions in populations where both vaccinating behaviour and NPI behaviour respond to disease dynamics. The effectiveness of one type of intervention may interfere with the effectiveness of the other intervention, through the mediator of disease dynamics. The objective of our research is to explore the interplay between individual decision-making (which is driven by methods from decision field theory ), regarding vaccines, NPIs, and disease dynamics in the context of influenza transmission and control, and to study the implications for disease mitigation strategies.
We model the vaccination decision process using random walk subjective expected utility (SEU) theory, an intermediate stage in the mathematical derivation of decision field theory . This approach allows us to capture the decision-making process of individuals in an uncertain environment. In the case of vaccination, the uncertainty lies in the chance of becoming infected in a given influenza season, depending on whether or not the individual is vaccinated and how effective the vaccine is.
For the vaccinating decision, each susceptible individual compares the possible outcomes stemming from the “Yes” branch versus the possible outcomes stemming from the “No” branch (Fig 1a). The difference in these expected utilities, or ‘valence’ (VY(t)−VN(t)), updates an individual’s preference, P(t), towards choosing one of these actions. The preference state of each individual is updated daily according to the rule: (1)
(a) Diagram representing the vaccination choice problem. (b) Diagram representing the infectious NPI choice problem. (c) Diagram representing the susceptible NPI choice problem.
If P(t) reaches a specified threshold, θ, then an individual decides to become vaccinated in that influenza season. Conversely, if P(t) reaches −θ, the individual decides not to become vaccinated that season. Intermediate values −θ < P(t) < θ can be interpreted as an individual being undecided regarding whether or not to vaccinate. If a choice is made, an individual’s preference state remains constant at P(t) = θ or P(t) = −θ until the beginning of the next season, when it then resets towards (1−s)Pend where Pend is the preference at the end of the last influenza season, and s is a memory decay factor.
Let us now define the social utility parameters associated with the vaccinating decision. The quantity EI < 0 is the negative utility received (cost incurred) when an individual gets infected; EV < 0 is the negative utility received (cost incurred) when an individual vaccinates; ES > 0 is the positive utility received when an individual takes actions that they perceive will inhibit the spread of infection and therefore saves others from becoming infected; and EH < 0 is the negative utility received (cost incurred) when an individual believes they are responsible for harming a neighbour by infecting them . The baseline values for these and other parameters can be found in Table 1. If an individual chooses to not vaccinate during a season, they may become infected that season. If an individual instead chooses to vaccinate, the vaccine is effective for that season with probability ϵV (the vaccine efficacy) and otherwise the individual remains susceptible for the remainder of the season. The wi parameters (0 ≤ wi ≤ 1) represent the ‘subjective probability weights’ that determine the possible outcomes that are considered on a given day. In the case of vaccination, w1 is the probability an individual perceives of being infected if they do not vaccinate in a particular season. We assume w1 depends on how many of an individual’s neighbours have been infected in the current season, as well as their memory of the cumulative incidence from previous seasons: (2) where σ controls the relative importance of incidence from current versus past seasons, Xn is the cumulative number of neighbours who have become infected in the current influenza season n, MH(x) = 1−e−κH x where κH is a proportionality constant controlling the perceived chance of becoming infected in a season, and ξn−1 is an individual’s memory of incidence from past seasons: (3) where Yn is the cumulative number of an individual’s neighbours, including themselves, that have been infected by the end of season n. In this way, the memory of past influenza incidence declines with time according to σ.
Individuals may vaccinate during certain days of the year 280 < t or t < 40 (Fig 2), where t = 0 is January 1st, and the influenza season from the previous year is considered to end on day t = 285. We use this constraint because it reflects the typical timing of public influenza vaccination programs in fall and winter in many northern hemisphere countries, such as the United States and Canada. At the end of an influenza season, we set Yn = Xn and then incorporate Yn into the individual’s memory.
Example time series of vaccination (blue) and incidence (green) over a season. When a vaccine becomes available in early October, uptake increases in anticipation of the upcoming influenza season. Confidence intervals represent two standard deviations of outputs for the 100 parameter sets (see Model Calibration section).
The second subjective probability weight is (4) where MC(x) = 1−e−κC x, Nvac and Nsusc are the number of currently vaccinated and susceptible neighbours, respectively, and κC controls the perceived probability of infection. We interpret w5 as an individual’s perceived probability of infecting one or more neighbours, and the term w5 EH captures the future outcome of an individual potentially infecting his/her neighbours that season after becoming infectious themselves, ultimately leading to a negative utility. To complete the outcomes of this branch, we note that the perceived probability of not becoming infected when choosing to not vaccinate is simply w2 = 1−w1. This outcome leads to a utility of 0.
On the ‘Yes’ branch, we define w3 = ϵV as the perceived probability that an individual is efficaciously vaccinated, thus w4 = 1 − ϵV. In both cases, an individual knows that they must absorb the vaccine cost, EV. In the case of efficacious vaccinating, a positive utility ES is also gained by protecting their neighbours for the remainder of the influenza season, which serves a similar function to the w5 EH term stemming from the ‘No’ branch where an individual is considering future outcomes. In the case of inefficacious vaccination, an individual assumes that they may still become infected with a probability that increases with past and current disease incidence. This is represented by the term VN(t), the valence of the ‘No’ branch.
We model the NPI decision process using sequential SEU theory, a method similar to random walk SEU theory, but excludes the possibility that the preference state may start from a non-neutral initial value . We use sequential SEU theory because we assume individuals make social distancing decisions on a day-to-day basis, dependent only on the current state of infection amongst their respective neighbours, whereas in the case of vaccination, the tendency to vaccinate or not can be carried over from one season to the next. Each infectious individual decides whether or not to practice NPIs to protect their neighbours for the duration of their illness (Fig 1b), and each susceptible individual decides whether to practice NPIs to protect themselves from their infectious neighbours that day (Fig 1c).
On the ‘Yes’ branch for the infectious NPI decision (Fig 1b), we have the probability of efficaciously using NPIs, w7 = ϵNPI, or inefficaciously using them, w8 = 1−ϵNPI. If NPIs are efficacious, they receive a positive utility ES for saving susceptible neighbours from infection. However, if NPIs are inefficacious, they receive the valence VN(t) of the ‘No’ decision, on the branch associated with w8, representing that the outcome is the same as if they had never practiced NPIs to begin with. An individual believes that their use of interventions during their illness will be either fully effective or ineffective on all of their neighbours. Regardless of whether NPIs work or not, the infectious individual who practices NPIs pays a cost (ED)(NTot) for having to utilize NPIs to protect their NTot neighbours, where ED < 0 is the negative utility received (cost incurred) per neighbour. For the ‘No’ branch where the individuals decides not to practice NPIs, they may infect a neighbour with probability w5, receiving a negative utility (cost incurred) of EH < 0 due to feeling responsible for spreading the infection. On the other hand, they infect no neighbours with probability w6 = 1−w5, thereby receiving a utility of zero.
NPI decisions are made by susceptible individuals who seek to protect themselves from their infectious neighbours in a similar way (Fig 1c). On the ‘Yes’ branch, an individual believes their NPIs will be efficacious with probability w7, receiving utility zero. If the NPIs are not efficacious, they receive the valence VN(t) from the ‘No’ branch. In either case, they pay a cost ED in order to practice NPIs. On the ‘No’ branch, an individual who does not practice NPIs that day becomes infected with probability w9 = MC(NInf), and receive a negative infection utility EI. With probability 1−w9, they believe they will not become infected and receive utility zero. We do not apply social utilities EH and ES in the susceptible NPI decision process because we assume as a first-order approximation that their decision focuses on short term outcomes (NPIs are only effective for the duration of infection, and may have to be repeated several times in the season, whereas a one-time vaccination decision will protect their neighbours from infection for the duration of the season). Also, if an individual is vaccinated, their perceived probability of becoming infected that day is reduced by 1−ϵV. This reflects the fact that these individuals will believe themselves to have less chance of becoming infected than those who have not vaccinated. We note that infectious individuals practice NPIs on all of their neighbours, whereas susceptible individuals only practice them on their infectious neighbours. Infectious persons may stay home from work, and their hand washing benefits all susceptible persons with whom they come in contact with. In contrast, susceptible persons can be selective about avoiding infectious persons, or hand-washing after contact with infectious persons.
In our model, individuals are not aware of their own or their neighbours’ true susceptibility statuses. That is, they will make their intervention decisions based only on their own acquired knowledge which assumes everyone is susceptible at the beginning of each influenza season. This is in contrast to the true state of the system, which incorporates factors such as waning immunity. Moreover, the data we present on susceptible NPI rates only reports for those who are truly susceptible.
The vaccination and NPI decision-making processes are embedded in an agent-based simulation model of influenza transmission through a static contact network. The contact network consists of 10,000 nodes through which influenza is transmitted, and was constructed by sampling a subnetwork from a larger contact network derived from census data from Portland, Oregon [38–40]. Previous research has confirmed that the subnetwork is a good approximation to the full network .
We assume a Susceptible-Infectious-Recovered-Vaccinated-Susceptible (SIRVS) natural history. Individuals move from the susceptible state S to the infectious state I with probability per day, where Ninf is the number of infectious neighbours around the susceptible person on day t, and β is the transmission probability which varies seasonally according to . If either the susceptible person or the infected person has opted to practice NPIs that day, then NPIs are effective with probability ϵNPI, and that infected neighbour is not included in Ninf for the purposes of computing Pr(t, Ninf). Infectious individuals recover (move from state I to state R) in a number of days sampled from a Poisson distribution with mean λ days. Individuals who have been efficaciously vaccinated with probability ϵV are moved to the V state, 14 days after being vaccinated . Both recovered and vaccinated individuals become susceptible again at the beginning of each new season (day 285 of each year) with probabilities ρ and ω, respectively. In order to capture seasonal case importation, 5 randomly chosen susceptible individuals are made infectious every 10 days, from day 330 to 360.
Each day, the following sequence of events occurs: (1) each susceptible individual decides whether or not to practice NPIs on that day; (2) the following occur in a random order for each randomly chosen individual in the population: (i) If an individual is susceptible, they update their vaccination preference, (ii) if an individual is susceptible, they may become infected and make an infectious NPI decision, (iii) if an individual is infectious, they may recover.
To construct a baseline scenario, we calibrated the transmission probability β, amplitude of seasonality Δβ, and case importation rate η to the targets: (1) a cumulative seasonal incidence of approximately 15% to 20% in the absence of vaccination, and (2) infection prevalence that peaked, on average, between December and January of each year [24, 42, 54, 55]. These estimates come from North American (primarily, United States) populations. The calibrated value of Δβ was constrained on the interval (0.15, 0.3) .
We also calibrated the preference state threshold for vaccinating θV, per season memory decay rate s, and proportionality constant for the perceived chance of becoming infected in an influenza season κH to the targets: (1) vaccine coverage of 30% to 40% per season, with (2) the majority of vaccinations occurring in October and November. These targets provide disease dynamics very similar to seasonal influenza. The utilities EI and EV were set according to Ref. , based in turn on Ref. [42, 50, 55–59]. Finally, the social utilities ED, ES, and EH were calibrated to the targets: (1) an infectious individual practices NPIs with 87% probability, and (2) a susceptible individual practices NPIs with a 66% probability .
After obtaining this baseline scenario, we conducted three-point estimation Monte Carlo probabilistic sampling using triangular distributions to obtain sets of parameter values that yielded outputs within acceptable ranges. The triangular distributions were defined around the most uncertain baseline parameter values. Very broad ranges were used for the most uncertain parameters, to reduce model fitting issues due to having more parameters than calibration targets (generally, for each set of calibrated parameter values described above, there was one less target data point than the number of parameters to be calibrated) (Table 2). We repeatedly sampled parameter values from these distributions, and ran simulations using the sampled parameter sets. We discarded any parameter sets that yielded outcomes outside of a feasible range for vaccine uptake and NPI practice rates (Table 3). We accepted a larger range of NPI practice rates than for vaccine uptake, due to the greater uncertainties about the frequency of NPI practice for seasonal influenza . In total, 2250 simulations were tested, providing a target number of 100 parameter sets yielding feasible outcomes. All simulations ran for 50 seasons with an initial population of susceptible individuals having no preference towards vaccinating and no perceived probabilities of becoming infected. For our results, the first 20 seasons of each simulation were discarded to remove transient effects.
The data shows the average value per season over all 100 parameter sets, for vaccine coverage, infection incidence, probability of susceptible individuals practicing NPIs given that they encounter one or more infectious individuals on a given day (“susceptible NPI practice”), and probability of infectious individuals practicing NPIs while ill (“infectious NPI practice”).
When vaccination is first introduced to the population, vaccine coverage climbs rapidly and peaks in the first few years after the vaccine becomes available, as members of the population adopt vaccination to avoid infection (Fig 3). As a result, seasonal influenza incidence decreases, which in turn causes a decrease in the probability that susceptible persons practice NPIs if they have an infected neighbour. This occurs because the reduced infection incidence due to the vaccine reduces the perceived infection risk among susceptible individuals, and thus makes them less willing to practice NPIs. After the initial peak in vaccine coverage and the corresponding dip in NPI practice, the vaccine uptake, infection incidence, and NPI practice rates equilibrate. In contrast to susceptible NPI practice, the infectious NPI practice rate stays almost constant during the whole period, due to the relative stability of the utilities found in the decision branches regarding this decision (Fig 1b). For example, an infectious individual deciding to use NPIs will always look to protect all of their neighbours, and this does not depend strongly on population-level incidence of infection that season. This is in contrast to a susceptible individual’s decision whether to use NPIs, which depends on how many of their neighbours are perceived to be infected.
Example of a baseline scenario of our model where vaccination becomes available in season 10, causing a change in the vaccine coverage each season (blue), the seasonal infection incidence (green), the probability that a susceptible individual practices NPIs given that they encounter one or more infectious individuals on a given day (black), and the probability that an infectious individual practices NPIs while ill (red). Confidence intervals represent two standard deviations of outputs for the 100 parameter sets.
Interventions Can Interfere with One Another
Next, we conducted a univariate sensitivity analysis, evaluating the impact of changes in baseline parameter values corresponding to measures that public health might take in order to improve outcomes. Increasing the utility for saving others from infection (ES) causes a significant reduction in infection incidence (Fig 4a and 4b). It also causes a slight decrease in NPI practice by susceptible individuals, but this is outweighed by large increases in both vaccine uptake and NPI practice by infectious persons that are sufficient to cause a net decline in infection incidence. These results illustrate a tradeoff whereby vaccine uptake, NPI practice among infectious individuals, and NPI practice among susceptible individuals cannot be simultaneously increased by changing ES. A reduced incidence due to expanding any one of these interventions will reduce the perceived infection risk and make individuals incrementally more complacent about preventing infection, which in turn reduces the uptake rates for the other interventions. Hence, each intervention tends to interfere with the other. Our focus in the remainder of this paper is to determine the conditions under which the interference between the two intervention types is strongest, which model parameters are most subject to interference, and how to bring about the greatest net reductions in infection incidence, despite interference. How interference plays out over time has already been described (Fig 3).
Univariate sensitivity analysis for parameters ES, EH, and ED. The numbers on the horizontal axes correspond to multiples of the baseline values for ES, EH, and ED, hence 1.0 corresponds to the baseline value of the each parameter. (a), (c), (e) Average values across 30 seasons for vaccination coverage (blue) and incidence (green). (b), (d), (f) Average values across 30 seasons for NPI usage amongst susceptible individuals (black) and infectious individuals (red). Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
In contrast to significant reductions in infection incidence caused by increasing ES, increasing the cost for harming others (EH) causes only small net reductions in incidence, because a large increase in the NPI practice among infectious persons is strongly offset by a modest decline in vaccine uptake, while the rate of NPI practice by susceptible persons remains relatively constant (Fig 4c and 4d). Similarly, attempting to reduce incidence by decreasing the perceived cost of practicing NPIs (ED) also causes only a small net reduction in incidence, since the resulting increases in NPI practice among infectious and susceptible persons are again offset by reductions in vaccine uptake (Fig 4e and 4f).
Decreasing the cost of vaccination (EV)–for instance by making the vaccine more easily accessible–results in significant reductions in infection incidence, because the significant increase in vaccine uptake is only partly offset by the resulting decline in NPI practice (Fig 5a and 5b). Likewise, increasing the perceived cost of infection (EI) causes an increase in both vaccine uptake and susceptible NPI practice, although infectious NPI practice remains relatively unchanged. The effect is a significant net decrease in incidence (Fig 5c and 5d). In summary, increasing the utility for saving others from infection (ES), decreasing the perceived cost of vaccination (EV), or increasing the perceived cost of infection (EI), are more effective in reducing infection incidence than changing perceived harms (EH) or perceived cost of NPI (ED), despite interference.
Univariate sensitivity analysis for EV and EI. The numbers on the horizontal axes correspond to multiples of the baseline values for EV and EI, hence 1.0 corresponds to the baseline value of the each parameter. (a), (c) Average values across 30 seasons for vaccination coverage (blue) and incidence (green). (b), (d) Average values across 30 seasons for NPI usage amongst susceptible individuals (black) and infectious individuals (red). Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
Determining Which Interventions Interfere Most Strongly
In order to better understand how NPIs interfere with vaccine uptake, we compute the difference ΔV in vaccine uptake between the baseline scenario where individuals are free to practice susceptible and infectious NPIs versus a hypothetical scenario where they cannot practice either form of NPI. Similarly, to determine how vaccination interferes with susceptible (and infectious) NPI practice, we compute the difference ΔNS in susceptible NPI practice rates (similarly, ΔNI for infectious NPI practice rates) between the baseline scenario where individuals are free to choose vaccination versus a hypothetical scenario where vaccination is not available. We also compute the difference in seasonal incidence ΔI between the baseline scenario and all of these hypothetical scenarios.
Impact of interference on intervention uptake rates.
With respect to impacts of interference on intervention uptake rates, we observe that in general, across a broad range of utilities for ES, EH, ED, EV, and EI, NPI practice interferes significantly with vaccine uptake (ΔV usually ranging from 5% to 15%, Figs 6a, 6c, 6e, 7a and 7c). Vaccination also interferes significantly with susceptible NPI practice (ΔNS usually ranging from 5% to 15%, Figs 6b, 6d, 6f, 7b, and 7d), but not as much with infectious NPI practice (ΔNI is smaller, Figs 6b, 6d, 6f, 7b, and 7d). Infectious NPI practice is not as strongly affected because infectious NPI practice depends less on population prevalence than susceptible NPI practice does.
Univariate analysis for social parameters ES, EH, and ED determining the amount that vaccination and NPIs interfere with each other in each scenario. (a), (c), (e) Average values across 30 seasons for change in vaccination coverage (blue) and change in incidence (green) between hypothetical scenarios without NPI usage and the baseline scenarios. (b), (d), (f) Average values across 30 seasons for change in NPI usage amongst susceptible (black) and infectious (red) individuals and change in incidence (green) between hypothetical scenarios without vaccine usage and the baseline scenarios. Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
Univariate analysis for EV and EI determining the amount that vaccination and NPIs interfere with each other in each scenario. (a), (c) Average values across 30 seasons for change in vaccination coverage (blue) and change in incidence (green) between hypothetical scenarios without NPI usage and the baseline scenarios. (b), (d) Average values across 30 seasons for change in NPI usage amongst susceptible (black) and infectious (red) individuals and change in incidence (green) between hypothetical scenarios without vaccine usage and the baseline scenarios. Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
Interference of NPIs with vaccination is strongest at intermediate values of EV (Fig 7a). When EV is small, NPIs are not popular in the population and hence NPIs do not interfere strongly with vaccination. Thus, when NPIs are removed, there is little impact ΔV on vaccination levels. When EV is large, vaccination becomes too costly for the population to adopt it widely. Therefore, vaccination levels do not increase by significant amounts when NPIs are added or removed from the population. When vaccination is not present, we see that for small EV, NPIs are interfered with especially strongly. This is because high vaccination coverage seen in the baseline scenario disappears, and thus individuals choose to practice NPIs more often. For larger values of EV, NPIs are undermined by decreasing amounts since vaccination levels in the baseline scenarios are greatly reduced (Fig 7b). Similar reasoning can explain trends in ΔV, ΔNI and ΔNS as other utilities are varied (Fig 7c and 7d).
Impact of interference on influenza incidence.
Overall, when removing either vaccination or NPIs, we observe an increase in incidence, compared to baseline scenarios where both interventions are available (ΔI > 0; green lines in Figs 6 and 7). This occurs because having both interventions as an option is better than just having one of them.
However, the increase in incidence caused by removing vaccination (Figs 6b, 6d, 6f, 7b, and 7d) is much larger than the increase in incidence caused by removing NPIs (≈ 5–10% versus ≈ 1–2% Figs 6a, 6c, 6e, 7a, and 7c). Hence, vaccination appears to be a much stronger determinant of influenza incidence than NPI practice. When NPIs are introduced to the population, vaccine uptake declines sufficiently such that the net change in incidence ΔI is small, thus vaccination responds to the introduction of NPIs in a way that strongly mitigates the effectiveness of NPIs in reducing incidence (Figs 6a, 6c, 6e, 7a, and 7c). However, when vaccination is introduced to the population, the resulting decline in NPI practice is not sufficient to prevent significant changes in ΔI (Figs 6b, 6d, 6f, 7b, and 7d), thus NPIs do not strongly mitigate the effectiveness of vaccination. In other words, vaccination interferes strongly with the effectiveness of NPI practice, whereas NPI practice interferes weakly with the effectiveness of vaccination.
As a result of interference, efforts to boost NPI practice should be partially counter-productive due to the mitigating response of vaccine uptake, whereas efforts to boost vaccine uptake should be productive because the mitigating responses of NPI practice are not strong enough to prevent a significant decrease in influenza incidence. This trend is consistent across a broad range of parameter values.
Understanding What Drives Different Levels of Interference for Different Interventions
To understand the source of this asymmetry between the two interventions, we vary the NPI efficacy (ϵNPI) and the vaccine efficacy (ϵV) (Figs 8 and 9). As the NPI efficacy increases, the proportion of individuals practicing NPIs increases significantly (Fig 8b) and the vaccine uptake decreases in response, while the infection incidence also declines (Fig 8a). Similarly, when the NPI efficacy is very high, removing NPIs has a much larger impact on incidence than removing vaccination, the latter of which has almost no effect. But when the NPI efficacy is very low, the situation is reversed, and removing vaccination has a much bigger impact on incidence than removing NPIs (Fig 8c and 8d). These results show that feedback between interventions operates such that, if a less efficacious intervention is removed, the resulting increased uptake of the more efficacious intervention is sufficient to prevent a net increase in incidence. In contrast, if a more efficacious intervention is removed, the resulting increased uptake of the less efficacious intervention is not adequate to prevent an increase in incidence. Similar patterns hold when the vaccine efficacy (ϵV) is varied, for similar underlying reasons (Fig 9). However, a secondary factor working in favour of vaccination is that vaccination–if efficacious–protects individuals throughout the influenza season, whereas NPI practice needs to be efficacious every time there is an infection in a network neighbour, in order for an individual to avoid infection throughout the entire season.
(a), (b) Univariate sensitivity analysis for NPI efficacy, ϵNPI. Data shows average values across 30 seasons for vaccination coverage (blue), incidence (green), NPI usage amongst susceptible individuals (black), and NPI usage amongst infectious individuals (red). (c), (d) Determining the amount that vaccination and NPIs interfere with each other for various NPI efficacies. Average values across 30 seasons for change in vaccination coverage (blue), change in incidence (green), and change in NPI usage amongst susceptible (black) and infectious (red) individuals are shown. Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
(a), (b) Univariate sensitivity analysis for vaccine efficacy, ϵV. Data shows average values across 30 seasons for vaccination coverage (blue), incidence (green), NPI usage amongst susceptible individuals (black), and NPI usage amongst infectious individuals (red). (c), (d) Determining the amount that vaccination and NPIs interfere with each other for various vaccine efficacies. Average values across 30 seasons for change in vaccination coverage (blue), change in incidence (green), and change in NPI usage amongst susceptible (black) and infectious (red) individuals are shown. Confidence intervals represent two standard deviations of the mean of the 100 parameter sets across 30 simulated seasons.
The difference in vaccine uptake with and without NPIs is highest for intermediate values of ϵV. In general, this occurs because when intervention efficacy is very low, individuals will not adopt it, regardless of whether there is an alternative or not. Therefore, even if the alternative intervention is removed, individuals will tend not to increase adoption of the inefficacious intervention (Fig 9c, lowest values of ϵV). On the other hand, if an intervention is significantly more effective than the alternative intervention, or not very costly, then it will continue to be used by a large proportion of the population, and will not experience significant interference from the less effective alternative intervention which does not significantly change incidence (Fig 9c, highest values of ϵV).
In summary, these results show that, the more efficacious an intervention is, the less its effectiveness will be compromised by the other intervention, but the more it will compromise the effectiveness of other intervention. The central role of intervention efficacy also explains why the highest reductions in incidence occur when utilities that support vaccination only are changed (e.g. the vaccine cost is reduced, Fig 5a and 5b), or when utilities that support both vaccination and NPI practice are changed (e.g. when the payoff for saving others from infection is increased, Fig 4a and 4b, or the perceived cost of infection is increased, Fig 5c and 5d). In contrast, decreasing the perceived cost of social distancing ED has little impact on incidence (Fig 4e and 4f), since this NPI practice is interfered with by the mitigating response of vaccine uptake.
We have constructed a seasonal influenza transmission model that incorporates how behavioural decisions for both individual vaccinating decisions and individual NPI practice (hand-washing, social distancing) respond to changes in infection incidence. Our population was distributed across an empirically-based network, and parameter values were based either on literature [24, 41–52, 55], or were calibrated to typical influenza seasonal patterns using a probabilistic sampling approach.
These results illustrate how vaccine uptake and NPI practice interfere with one another. If vaccine coverage increases, the resulting change in transmission patterns causes a decrease in the practice of NPIs. This is especially true for susceptible individuals, since susceptible NPI practice is more sensitive to population incidence. Similarly, if NPI practice expands, vaccine coverage will decrease by a roughly similar amount.
Although susceptible NPI practice and vaccine coverage have similar impacts on each other’s uptake, the impact on incidence is highly asymmetrical between the two interventions: the effectiveness of NPI practice is strongly mitigated by the response of vaccine uptake, whereas the effectiveness of vaccination is only weakly mitigated by the response of NPI practice. This asymmetry is driven by the differing efficacies of the two types of interventions: the higher the efficacy of the intervention (ϵV, ϵNPI), the less its effectiveness in terms of reducing influenza incidence will be mitigated by the other intervention, but the more it will mitigate the effectiveness of other intervention.
Because influenza vaccine efficacy is generally higher than NPI efficacy, these effects have potentially important implications for influenza mitigation strategies. Efforts to boost NPI practice could be strongly counteracted by the resulting declines in vaccine coverage, hence boosting NPI practice could be counter-productive. However, boosting vaccine coverage can still be productive since the resulting response of NPI practice will not as strongly mitigate the effectiveness of expanded vaccine coverage. Because of the role of efficacy, increasing NPI efficacy through fostering better hand-washing techniques or respiratory etiquette might more useful than only increasing NPI uptake rates.
As a result of this asymmetry, we observed that increasing the utility for saving others from infection, ES was the most effective way of decreasing incidence because it supports both vaccination and NPI practice. From the standpoint of an advertising campaign, this would mean highlighting the fact that saving friends and family from becoming ill, both through vaccination and through NPIs, would be effective. In contrast, attempting to expand NPI practice without simultaneously encouraging vaccine uptake (for example through making hand sanitizer stations more widely available) could be counter-productive since NPI efficacy is not as high as vaccine efficacy. Similarly, increasing the perceived cost of infection, EI was also found to be an effective way to reduce incidence, since both vaccination and NPI practice are thereby supported.
The asymmetry also explains why decreasing the cost of vaccination, EV, was observed to be effective. The resulting expansion in vaccine uptake suppresses NPI practice to some extent, but because vaccine efficacy is higher than NPI efficacy, decreasing EV still causes net reductions incidence. Hence, reducing the perceived cost of the vaccine by expanding availability (through more seasonal influenza vaccine clinics) or decreasing its price will reduce influenza incidence, despite interference.
Our model makes several simplifying assumptions with respect to decision making. Firstly, we group all NPIs into two categories: those utilized by susceptible individuals, and those utilized by infectious individuals. In reality, however, an infectious individual may practice combinations, such as choosing to practice strict respiratory etiquette, but not staying home to isolate themselves. Both forms of NPI likely have differing efficacy, hence our grouping assumption could influence results. Similarly, we used a common cost parameter, ED, for all NPIs, but different forms of NPI would likely impose varying costs. We also allowed our population to have knowledge of both vaccine and NPI efficacies.
Additional factors that could impact the decision making processes are misinterpretations of influenza-like illnesses (ILIs) as being cases of influenza. Often, individuals may mistake other respiratory illnesses for influenza, artificially inflating perceived infection numbers and impacting perceived vaccine efficacy. Similarly, we assume only a single strain of influenza, when in reality, there are often multiple circulating strains. The model could be improved in future research by adding further heterogeneity such as age structure, making perceived efficacy depend on individual experience with interventions, and introducing a probability of ILI being mistaken for influenza, or vice versa . The contact network could also be modified to include family and work structures, which may in turn influence memories of previous infections and perceived infection risk. For example, individuals may weigh the fact that they have had a family member who was recently infected more so than if a casual contact like a co-worker recently fell ill. Moreover, individuals may be more inclined to practice NPIs around family members than their other contacts. Finally, personal infection history may be considered to be significantly more important than neighbour infection history when evaluating perceived risks, which is not accounted for in the model.
Being a highly parametrized model, there are several drawbacks associated with calibrating the model to empirical targets in order to obtain a baseline parameter set. We took parameter values directly from estimates in the available literature whenever possible, but (especially for NPIs) there is a dearth of information regarding intervention behaviour and impact for seasonal influenza , necessitating calibration. As a result, we had more calibrated parameters than calibration targets (see Methods), meaning that alternative baseline parameter sets could have matched the data almost as well as the baseline set that we used. Our adoption of very broad sampling intervals for the probabilistic uncertainty sampling partially addresses this since the broad intervals will include those parameter values, however, the resulting frequency distribution of outcomes could still vary depending on what baseline parameter set is used as the baseline for defining the triangular distributions. Future work could explore alternative parameterization methods, such as Latin hypercube sampling, which might help overcome this limitation.
In conclusion, interference stemming from feedbacks between interventions and disease dynamics can comprise the realized effectiveness of those interventions for reducing influenza incidence, depending on the clinical efficacy of the interventions in individuals. Health authorities and epidemiologists should further explore the potential for interference between different interventions for the same infectious disease, and formulate infection control strategies accordingly.
Conceived and designed the experiments: MAA CTB. Performed the experiments: MAA CTB. Analyzed the data: MAA CTB. Wrote the paper: MAA CTB.
- 1. Laxminarayan R, Mills AJ, Breman JG, Measham AR, Alleyne G, et al. Advancement of Global Health: Key Messages from the Disease Control Priorities Project. The Lancet. 2006;367:1193–1208.
- 2. Klein E, Smith DL, Gilligan CA. Economic incentives and mathematical models of disease. Environment and Development Economics. 2007;12:707–732.
- 3. Hethcote HW. The Mathematics of Infectious Diseases. SIAM Review. 2000;42(4):599–653.
- 4. Brewer NT, Chapman GB, Gibbons FX, et al. Meta-analysis of the relationship between risk perception and health behaviour: The example of vaccination. Health Psychology. 2007;26(2):136–145. pmid:17385964
- 5. Galvani AP, Reluga TC, Chapman GB. Long-standing influenza vaccination policy is in accord with individual self-interest but not with the utilitarian optimum. PNAS. 2007;104(13):5692–5697. pmid:17369367
- 6. Bootsma MCJ, Ferguson NM. The effect of public health measures on the 1918 influenza pandemic in U.S cities. PNAS. 2007;104(18):7588–7593. pmid:17416677
- 7. He D, Dushoff J, Day T, Ma J, Earn DJD. Inferring the causes of the three waves of the 1918 influenza pandemic in England and Whales. Proceedings of the Royal Society B. 2013;280.
- 8. Valle SD, Hethcote H, Hyman JM, Castillo-Chavez C. Effects of behavioral changes in a smallpox attack model. Mathematical Biosciences. 2005;195:228–251. pmid:15913667
- 9. Kelso JK, Milne GJ, Kelly H. Simulation suggests that rapid activation of social distancing can arrest epidemic development due to a novel strain of influenza. BMC Public Health. 2009;9(117). pmid:19400970
- 10. Centers for Disease Control and Prevention. Nonpharmaceutical Interventions (NPIs); 2012. http://www.cdc.gov/nonpharmaceutical-interventions/.
- 11. Bauch C, Manfredi P, d’Onofrio A. Behavioral Epidemiology of Infectious Diseases: An Overview. In: Manfredi P, d’Onofrio A, editors. Modeling the Interplay Between Human Behavior and the Spread of Infectious Diseases. Springer; 2013. p. 1–19.
- 12. Fine PEM, Clarkson JA. Individual versus public priorities in the determination of optimal vaccination policies. American Journal of Epidemiology. 1986;124:1012–1020. pmid:3096132
- 13. Bauch CT. Imitation dynamics predict vaccinating behaviour. Proceedings of the Royal Society B Biological Sciences. 2005;272(1573):1669–1675.
- 14. Bauch CT, Earn DJD. Vaccination and the theory of games. Proceedings of the National Academy of Sciences of the United States of America. 2004;101(36):13391–13394. pmid:15329411
- 15. Fu F, Rosenbloom DI, Wang L, Nowak MA. Imitation dynamics of vaccination behaviour on social networks. Database of Systematic Reviews. 2010;.
- 16. Perisic A, Bauch CT. A simulation analysis to characterize the dynamics of vaccinating behaviour on contact networks. BMC Infectious Diseases. 2009;9(77). pmid:19476616
- 17. Perisic A, Bauch CT. Social Contact Networks and Disease Eradicability under Voluntary Vaccination. PLOS Computational Biology. 2008;5(2).
- 18. Zhang H, Zhang J, Zhou C, Small M, Wang B. Hub nodes inhibit the outbreak of epidemic under voluntary vaccination. New Journal of Physics. 2010;12.
- 19. Reluga TC, Bauch CT, Galvani AP. Evolving public perceptions and stability in vaccine uptake. Mathematical Biosciences. 2006;204:185–198. pmid:17056073
- 20. Salathé M, Bonhoeffer S. The effect of opinion clustering on disease outbreaks. Journal of the Royal Society Interface. 2008;5:1508–1508.
- 21. Xia S, Liu J. A Computational Approach to Characterizing the Impact of Social Influence on Individuals’ Vaccination Decision Making. PLOS ONE. 2013;8(4).
- 22. d’Onofrio A, Manfredi P, Salinelli E. Vaccinating behaviour, information, and the dynamics of SIR vaccine preventable diseases. Theoretical Population Biology. 2007;71:301–317. pmid:17335862
- 23. Vardavas R, Breban R, Blower S. Can Influenza Epidemics Be Prevented by Voluntary Vaccination? PLOS Computational Biology. 2007;3(5). pmid:17480117
- 24. Wells CR, Bauch CT. The impact of personal experiences with infection and vaccination on behaviour-incidence dynamics of seasonal influenza. Epidemics. 2012;4:139–151. pmid:22939311
- 25. Francis DP, Heyward WL, Popovic V, et al. Candidate HIV/AIDS Vaccines: Lessons Learned from the World’s First Phase III Efficacy Trials. AIDS. 2003;17:147–156. pmid:12545073
- 26. Smith RJ, Blower SM. Could Disease-Modifying HIV Vaccines Cause Population-Level Perversity? The Lancet Infectious Diseases. 2004;4:636–639. pmid:15451492
- 27. Tully S, Cojocaru M, Bauch CT. Coevolution of Risk Perception, Sexual Behaviour, and HIV Transmission in an Agent-Based Model. Journal of Theoretical Biology. 2013;337:125–132. pmid:23988796
- 28. Reluga TC. Game theory of social distancing in response to an epidemic. PLOS Computational Biology. 2010;6(5). pmid:20523740
- 29. Funk S, Gilad E, Watkins C, Jansen VAA. The Spread of Awareness and its impact on epidemic outbreaks. Proceedings of the National Academy of Sciences of the United States of America. 2009;106:6872–6877. pmid:19332788
- 30. Gross T, D’Lima CJD, Blasius B. Epidemic Dynamics on an Adaptive Network. Physical Review Letters. 2006;96.
- 31. Shaw LB, Schwartz IB. Fluctuating epidemics on adaptive networks. Physical Review E. 2008;77.
- 32. Shaw LB, Schwartz IB. Enhanced vaccine control of epidemics in adaptive networks. Physical Review E. 2010;81.
- 33. Zanette DH, Risau-Gusman S. Infection Spreading in a Population with Evolving Contacts. Journal of Biological Physics. 2008;34:135–148. pmid:19669498
- 34. Glass RJ, Glass LM, Beyeler WE, Min HJ. Targeted social distancing design for pandemic influenza. Emerging Infectious Diseases. 2006;12(11):1671–1681. pmid:17283616
- 35. Chapman GB, Coups EJ. Predictors of Influenza Vaccine Acceptance Among Healthy Adults. Preventive Medicine. 1999;29(4):249–262. pmid:10547050
- 36. Busemeyer JR, Townsend JT. Decision Field Theory: A Dynamic-Cognitive Approach to Decision Making in an Uncertain Environment. Psychological Review. 1993;100(3):432–459. pmid:8356185
- 37. Shim E, Chapman GB, Townsend JP, Galvani AP. The Influence of Altruism on Influenza Vaccination Decisions. Journal of The Royal Society Interface. 2012;9:2234–2243.
- 38. Network Dynamics and Simulation Science Laboratory. Synthetic data products for societal infrastructures and proto-populations: Data set 1.0. Virginia Polytechnic Institute and State University; 2008.
- 39. Network Dynamics and Simulation Science Laboratory. Synthetic data products for societal infrastructures and proto-populations: Data set 2.0. Virginia Polytechnic Institute and State University; 2008.
- 40. Network Dynamics and Simulation Science Laboratory. Synthetic data products for societal infrastructures and proto-populations: Data set 3.0. Virginia Polytechnic Institute and State University; 2008.
- 41. Truscott J, Fraser C, Cauchemez S, Meeyai A, Hinsley W, Donnelly C, et al. Essential epidemiological mechanisms underpinning the transmission dynamics of seasonal influenza. Journal of the Royal Society Interface. 2011;.
- 42. Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, et al. Effectiveness and Cost-Benefit of Influenza Vaccination of Healthy Working Adults: A Randomized Controlled Trial. Journal of the American Medical Association. 2000;284(13):1655–1663. pmid:11015795
- 43. Demicheli V, Pietrantonj CD, Jefferson T, Rivetti A, Rivetti D. Vaccines for preventing influenza in healthy adults. Psychological Review. 2007;100(3):432–459.
- 44. Larson EL, hui Ferng Y, Wong-McLoughlin J, Wang S, Haber M, Morse SS. Impact of Non-Pharmaceutical Interventions on URIs and Influenza in Crowded, Urban Households. Public Health Reports. 2010;125(2):178–191. pmid:20297744
- 45. Sheehan JJ, Mott PJ, Sisk BW, Arbogast JW, Ferrazzano-Yaussy C, Bondi CAM. Alcohol-Based Instant Hand Sanitizer Use in Military Settings—A Prospective Cohort Study of Army Basic Trainees. Military Medicine. 2007;172(11):1170–1176. pmid:18062391
- 46. Earn DJD, Dushoff J, Levin SA. Ecology and evolution of the flu. Trends in Ecology and Evolution. 2002;17.
- 47. Longini IM, Hallorana ME, Nizama A, Wolffb M, Mendelmanc PM, et al. Estimation of the efficacy of live, attenuated influenza vaccine from a two-year, multi-center vaccine trial: implications for influenza epidemic control. Vaccine. 2000;18:1902–1909. pmid:10699339
- 48. Cox RJ, Brokstad KA, Ogra P. Influenza Virus: Immunity and Vaccination Strategies. Comparison of the Immune Response to Inactivated and Live, Attenuated Influenza Vaccines. Scandinavian Journal of Immunology. 2004;59:1–15. pmid:14723616
- 49. Ambrose CS, Yi T, Walker RE, Connor EM. Duration of Protection Provided by Live Attenuated Influenza Vaccine in Children. The Pediatric Infectious Disease Journal. 2008;27.
- 50. Lee PY, Matchar DB, Clements DA, Huber J, Hamilton JD, Peterson ED. Economic Analysis of Influenza Vaccination and Antiviral Treatment for Healthy Working Adults. Annals of Internal Medicine. 2002;137(4):225–231. pmid:12186512
- 51. Carrat F, Vergu E, Ferguson NM, Lemaitre M, Cauchemez S, et al. Time Lines of Infection and Disease in Human Influenza: A Review of Volunteer Challenge Studies. American Journal of Epidemiology. 2007;167.
- 52. Nichol KL, Tummers K, Hoyer-Leitzel A, Marsh J, Moynihan M, McKelvey S. Modeling Seasonal Influenza Outbreak in a Closed College Campus: Impact of Pre-Season Vaccination, In-Season Vaccination and Holidays/Breaks. PLOS ONE. 2010;161(5).
- 53. Centers for Disease Control and Prevention. Seasonal Influenza; 2013. http://www.cdc.gov/flu/.
- 54. World Health Organization. Influenza (Seasonal); 2014. http://www.who.int/mediacentre/factsheets/fs211/en/.
- 55. Keitel WA, Cate TR, Couch RB, Huggins LL, Hess KR. Efficacy of repeated annual immunization with inactivated influenza virus vaccines over a five year period. Vaccine. 1997;15:1114–1122. pmid:9269055
- 56. Nichol KL. Cost Benefit Analysis of a Strategy to Vaccinate Healthy Working Adults Against Influenza. JAMA Internal Medecine. 2001;5(3):749–759.
- 57. Meltzer MI, Cox NJ, Fukuda K. The Economic Impact of Pandemic Influenza in the United States: Priorities for Intervention. Emerging Infectious Diseases 5. 1999;5:659–671.
- 58. Bureau of Labor Statistics. Average Annual Pay by State and Industry; 2014. http://www.bls.gov/news.release/annpay.nr0.htm.
- 59. Keech M, Scott AJ, Ryan PJJ. The Impact of Influenza and Influenza like Illness on Productivity and Healthcare Resource Utilization in a Working Population. Occupational Medicine. 1998;48(2):85–90. pmid:9614766
- 60. Mitchell T, Dee DL, Phares CR, Lipman HB, Gould LH, et al. Non pharmaceutical interventions during an outbreak of 2009 pandemic influenza A H1N1 virus infection at a large public university april to may 2009. Clinical Infectious Diseases. 2011;52:138–145.