https://orcid.org/0000-0003-2940-647X
Figures
Abstract
Background
Protecting vaccines from freeze damage is a poorly addressed problem. We describe the effectiveness of the eLearning KeepCoool on cold chain maintenance in general practices.
Methods
For this intervention study, temperatures of vaccine refrigerators were logged at one-minute intervals. Personnel from practices with cold chain breaches was offered the eLearning. The primary outcome was the intervention’s effectiveness to achieve temperatures in the target range (2 to 8°C) in the sixth week (follow-up) compared to the first (baseline). Using continuous temperature data, a generalized additive model for location, scale and shape was estimated.
Results
The practice response rate was 38% (64 of 168). At baseline, 73% of the practices and 68% of the refrigerators (51 of 75) showed cold chain breaches. 47% of the practices (n = 22 with 24 refrigerators) participated in the eLearning (55 physicians and practice assistants). At follow-up, 17% of those refrigerators were in the target range continuously, 38% reached targets 95% of the time while always >0°C, and temperatures ≤0°C decreased by 63%. Based on 2 million temperature data, the average Euclidian distance based on regression showed a significant improvement (p<0.05).
Conclusion
The eLearning KeepCoool improved the practices´ vaccine cold chain. It is freely available at https://keepcoool.ukbonn.de.
Citation: Thielmann A, Schmitz M-T, Welchowski T, Weltermann B (2024) Effectiveness of the online-eLearning program KeepCoool at improving the vaccine cold chain in general practices. PLoS ONE 19(4): e0301847. https://doi.org/10.1371/journal.pone.0301847
Editor: Pasyodun Koralage Buddhika Mahesh, Ministry of Health, Sri Lanka, SRI LANKA
Received: May 9, 2023; Accepted: March 20, 2024; Published: April 16, 2024
Copyright: © 2024 Thielmann et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data set is available upon reasonable request to the ethics´ committee (ethik@ukbonn.de) as the data contain potentially identifying information.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The World Health Organization (WHO) considers protecting vaccines from freeze damage ‘one of the most poorly addressed problems in vaccine management’ [1]. Cold chain breaches are documented worldwide [2] and linked to disease outbreaks [3–5] or suspected to be linked [6]. In moderate climates, freeze exposure is the key problem which is especially dangerous to adsorbed vaccines (e.g. hepatitis B, tetanus, diphtheria, pertussis): aluminum-containing adsorbents form irreversible precipitates which decrease vaccines’ potency and may induce local irritation [7, 8]. For example, absorbed hepatitis B vaccines have a freezing threshold of -0.5°C [8]. To ensure vaccines potency, most vaccines are to be maintained within a temperature cold chain between +2°C and +8°C starting from the time of manufacture till preparation for patient application. Recently, new mRNA vaccines have increased the attention for the cold chain as some require -70°C for long-term stability and are allowed for storage at +2°C to +8°C for some weeks only [9].
Deficits in vaccine refrigerator management are documented worldwide. A systematic literature review showed freezing temperature exposure in 33.3% of refrigerators used for vaccine storage in ten wealthier countries [2]. Additional deficits were the lack of thermometers (6.9% to 91.9% of refrigerators) [10–17] and temperature logbooks (26% to 94%) [11, 14, 16]. Also, 20.3% to 52.6% of the practices incorrectly stored vaccines in door shelves [12, 13, 17]. In a 2014 survey among German general practitioners (GP), 16% reported cold chain breaches as error or near error, and only 51% reported to document refrigerator temperatures twice a day as recommended [18, 19].
Aiming at improving cold chain maintenance in general practices, we developed and evaluated the guideline-based eLearning program KeepCoool for GPs and practice assistants. The elearning was based on recommendations from several countries (US [20], UK [21], Australia [22], Canada [23], Scotland [24]). It provides learning content in five tutorials: temperature, refrigerator, storage, responsibilities, and monitoring. When starting, participants completed an 11-item knowledge questionnaire. The effectiveness of the elearning was evaluated in a prospective study with 64 general practices addressing knowledge and refrigerator temperatures. As published, the prevalence of refrigerators with temperatures in the target range was 32% at baseline [25]. Also, a high prevalence of additional storage deficits was documented, e.g., no use of bins (81.3%), no thermometers in the center of refrigerators (54.0%), vaccines boxes with contact to outer walls (46.3%), refrigerator unsuitable for vaccine storage (44.6%), and unwrapping of vaccines (31.1%) [26]. Access to the eLearning program KeepCoool significantly increased the knowledge of GPs and practice assistants: the mean number of correct answers was 5.6 of 11 answers at baseline and increased to 9.8 after program participation (p<0.001) [27].
Drawing on further data of the KeepCoool study, we here describe the effectiveness of the eLearning program KeepCoool to improve the vaccine cold chain measured by continuous temperature logging after six weeks. Secondary outcomes comprised chances in the prevalence of freezing temperatures, refrigerators used, and a Euclidian prediction model using continuous temperature data.
Methods
Study population and study design
The KeepCoool study was conducted in German general practices from a university teaching practice network (n = 185) from February to October 2018. Typically, these physician-owned practices are one to three physician practices with about 2 practice assistants per physician with larger group practices currently emerging. The practices serve patients covered by the statutory health insurance (about 90% of the population) and private insurances (about 10% of the population). Practices frequently have long lasting relationships with their patients.
Recruitment followed a structured approach: contact by phone and fax up to three times or until practices responded. A trained health services researcher (A.T.) and a trained study assistant visited all practices which had volunteered for participation. They had been involved in other practice studies and were familiar with the specifics of the setting as well as the topic. They informed the practice teams about the study and the study materials, obtained written informed consent, conducted the baseline survey, and placed the data logger in the refrigerators. Details are published [25, 26, 28].
The study was originally designed as a randomized, controlled trial with intervention arm (study arm 1) and waiting-list control arm (study arm 2) in practices with confirmed cold chain breaches [28]. In addition, the original design included a third arm to follow practices with temperatures in the target range (study arm 3: monitoring only). However, temperature measurements of the first 7 days (baseline) were far worse than expected: 68% refrigerators were outside the target range, and 15% had reached critically low temperatures <0°C for up to 7 days [25]. Subsequent ethical considerations motivated the change to a study with pre-post-design to avoid the use of vaccines with reduced potency and to assure patient safety. Thus, the study arms 1 and 2 were combined resulting in a total of two study arms: intervention arm (A) with practices having cold chain breaches at baseline. The monitoring only arm (B) with practices without cold chain breaches at baseline remained identical (former study arm 3). Thus, access to the eLearning program KeepCoool was offered to all practices with cold chain breaches at baseline (intervention arm A). For analyses, the practices in the intervention arm A were later grouped according to their participation or non-participation in the eLearning. We aimed to investigate if the participation in the eLearning improved refrigerator temperatures. Data logging remained identical for all practices. Temperature data logging was continued every minute until the follow-up practice visits after a minimum of five weeks.
For analysis, refrigerators served as their own control. Separately, refrigerators in the target range at baseline were monitored to evaluate if temperatures were maintained (monitoring only). The responsible ethics´ committee was informed on the protocol change. All practice sites originally included remained in the study.
eLearning intervention KeepCoool
The best practices-based eLearning KeepCoool provides learning content in five tutorials: temperature, refrigerator, storage, responsibilities, and monitoring. When starting, participants completed an 11-item knowledge questionnaire as basis for tailored learning. The eLearning used three didactic elements: a) personal address, b) targeting of the respective profession (physicians or practice assistants), c) individualized feedback tailored to a participant’s baseline knowledge. The program followed a consistent presentation with `basic information´, `tips for the practice´, and `expert knowledge´ for in-depth understanding. Options to download relevant literature and practice templates were available. After the eLearning, participants who successfully answered ≥ 7 of the same 11 questions (> 60% correct answers as required by the regional physician association) received a certificate. For details [27].
Continuous temperature logging, primary and secondary outcomes
Temperatures were measured using a data logger (testo 175T) with an accuracy of ±0.4°C in the operating range -5°C to +10°C (calibrated under a DIN EN ISO 9001:2008 certified quality assurance system). The device was equipped with a standard probe which measured ambient air temperatures inside the refrigerator at 1-minute intervals similar to prior studies [10, 12, 15, 29, 30]. We abstained from glycol probes [28] due to limited resources and an easier handling of the new device, which remained inside the refrigerators throughout the study with the display turned off and memory access locked [25]. For the analyses, the first 120 minutes after each data logger set-up were excluded to allow for probe acclimatisation.
Using data from all practices, the effectiveness of the intervention was determined using 7-day-temperature readings (10,080 recordings) of the 1st week (baseline period) compared to the 6th week (follow-up period). The primary outcome was the percentage of refrigerators with temperatures within the target range (2 to 8°C) in the follow-up period. Secondary outcomes comprised various parameters, e.g., the prevalence of freezing temperatures (see Table 2), structural changes of refrigerators, and a Euclidian temperature prediction model. For the three temperature zones (+2 to +8C; ≤0°C; ≤ -0.5°C) we calculated the hours in the respective zone (so called: cumulative time) and the longest period in the respective zone (so called: longest consecutive time). In addition, we noted if—and when—the practice assistants and the physicians finished the KeepCoool eLearning knowledge test and received the certificate.
Ethical approval, consent to participate and trial registration
The study complies with the ethical principles of the World Medical Association Declaration of Helsinki. Ethical approval was obtained from the Ethic Commission of the Medical Faculty of the University of Duisburg-Essen (reference number: 14-6118-BO). Participants provided written informed consent. The study was registered in the German Trial Register: DRKS00006561 (date of registration: 20 February 2015).
Statistical analyses.
A sensitivity power simulation was conducted with 1055 observations randomly drawn with replacement of the complete data set. Then the temperatures of the refrigerators were randomly drawn from the response distribution conditional on the estimated parameters from the original model M3. After data generation, model M1 was estimated with the same parameter specification as model M3. In addition, model M2 had the same specification, but without intervention main and interaction effects (null hypothesis). Those two nested models M2, M3 were compared with a likelihood-ratio test. The complete procedure was repeated independently 250 times and the intervention effect was detected in all simulated cases. This shows that the much larger total sample size of 2110282 is more than enough to achieve high levels of power to detect the intervention effect.
Practice, refrigerator, personnel, and temperature characteristics were analysed using descriptive statistics comparing the first week (baseline) with the sixth week (follow-up). In addition, all available temperature measurements observed (up to 20 weeks) were used for two analyses:
- To account for participants´ variations in accessing the eLearning, the periods before/after the first participant finished the test were compared regarding the percentage of time within the target range (2 to 8°C).
- Due to the high temperature variability between refrigerators and a target range of 2 to 8°C centred around 5°C, we chose a generalized additive model for location, shape and scale [31] to identify if a practice’s participation in the eLearning changed/improved temperature over time. This approach has the benefit to not only model a mean response, but also the variance depending on the co-variates. The trend over time was estimated with a p-spline basis [32]. Besides the main intervention effect all second and third order interactions between intervention and the other fixed effect variables (e.g., temperatures measured on weekends, physician participation, number of participants per practice, group versus solo practice) were included. Heterogeneity between practices and refrigerators were taken into account as random intercept effects. Due to the large sample size model selection of the distribution of the temperature was conducted without covariate effects based on the generalized Akaike information criterion [33]. As result the skewed power exponential type four distribution was chosen [34, 35]. The calculated average Euclidian distance compares the distance of the prediction interval to 5°C as median of the target range (2 to 8°C) for the time periods before and after the first participant per practice finished the test. The +5°C value was chosen because guidelines recommend to target 5°C when adjusting refrigerators. The t-test statistics was applied (p<0.05). The details are presented in the Supplement.
Statistical analyses were performed using IBM SPSS Statistics for Windows, version 25 (Armonk, NY: IBM Corp.) and R, version 4.0.4. Percentages and mean values are reported for valid cases. Models were estimated using R package gamlss (Version 5.3–2).
Results
Characteristics of participating practices and practice personnel
Of the 168 practices contacted, 64 participated (response rate: 38.1%) with 75 refrigerators. At baseline, 73.4% of the practices (47 of 64) showed cold chain breaches in ≥ 1 refrigerator and were offered the eLearning. This offer was accepted by 48.9% of the practices (n = 23 of 47) (`participating practices´), while 24 practices did not participate (`non-participating practices´). Follow-up data were available for 44 of 47 practices (one drop-out among participating practices, two among non-participating ones). As separate third group, practices without cold chain breaches at baseline were followed (monitoring only practices). The subpopulations were comparable for key characteristics (Table 1 and Fig 1).
Refrigerators and temperatures at follow-up
Overall, we observed improvements in refrigerators and temperatures, most of which were significantly better in the participants than the non-participants, although these practices did not reach the quality of the monitoring only group (Table 2).
The primary outcome (continuous temperatures within target range of 2 to 8°C at follow-up) was reached by 4 of 24 (+16.7%) refrigerators. Improvement was noted also for secondary outcomes: critically low temperatures (≤0°C) 62.5% less at follow-up; in the target range in 95% of the time while always >0°C improved by 20.8% (baseline: 16.7%; follow-up: 37.5%).
Among participants, one practice had a new refrigerator (unsuitable at baseline) and one had ordered two refrigerators (both unsuitable at baseline). In the non-participants, one refrigerator was defrosted, one practice exchanged their refrigerator with another existing one, one bought a new refrigerator (unsuitable at baseline).
Participation in eLearning and temperature changes
As published prior, the average knowledge score at baseline was 5.6 (SD ±1.9) of 11 correct answers with slight differences between physicians (5.9; SD ±2.3) and practice assistants (5.5; SD ±1.7) [27]. Average knowledge improved to 9.8 ± 1.2 with an increase of 4.2 points (p<0.001). In physicians, the mean knowledge score at follow-up increased by 4.6 points in physicians (p<0.001) and by 4.0 points in practice assistants (p < 0.001) [27].
On average, the first person per practice finished the test 7.4 days (SD 9.16, range 0 to 40, median 5) after receiving the login information. Within practices the duration between the first and last participant who finished the test was 7.9 days (SD 11.93, range: 0 to 40, median 2). Seven practices finished eLearning during the sixth week follow-up or later. For details see Fig 2.
Prediction model
Fig 3 shows the average Euclidean distance of the 95% prediction intervals (considering both, model uncertainty and response uncertainty) from the two points (5, 5) up to and after the first participant finished the eLearning for the participating practices of the intervention group. These two points represent the average value of the interval between 2 and 8 degrees centigrade that is considered a good range for storage of medical vaccines. The smaller the Euclidean distances, the better are the circumstances for medical storage. Euclidean distances were significantly reduced after the first participant finished, as indicated by a paired t-test (p-value = 0.0046).
The model shows if a practice’s participation in the eLearning changed/improved temperature over time given the covariates observed time, weekend status, participation of a physician, number of participating practice assistants and single/group practice. Refrigerators of participating practices in the intervention group were analyzed using N = 2,110,282 single measurements (mean per practice 95,921.91, SD 22,445.52). All p-values were significant after multiple comparison adjustment regarding the false discovery rate [36] level of 0.05 except that weekend measurement did not influence the scale of the distribution. The intervention therefore affected both location and scale of the refrigerator temperature.
Discussion
To our knowledge, KeepCoool is the first eLearning with proven effectiveness on vaccine storage knowledge [27] and cold chain management in the German speaking countries. Using the strict outcome of continuous cold chain maintenance in the 7-day follow-up period (2 to 8°C), 17% of refrigerators in the participating practices newly reached this target. Using more than two million temperature measurements over up to 4,5 months, our prediction model showed a significant improvement after participation in the eLearning KeepCoool.
In the international literature, few interventions with established effectiveness for improving the cold chain quality are available, none of which were based on eLearning. These interventions used various components either alone or combined: 1) written educational material, 2) introduction of thermometers with or without feedback on temperatures by either graphic display or telephone advice, 3) 1:1 onsite education with inspection of refrigerators. Direct comparison with KeepCoool is difficult due to methodological differences. With regard to reaching the target temperature range, the best result was observed in an Australian RCT of 50 primary care practices not yet using acceptable thermometers: onsite education with distribution of min/max-thermometers led to a fourfold increase of practices with refrigerator temperatures in the target range after 30 days [37]. Two other studies available provided long-term follow-ups after 6 months, 1 year, and 5 years: they documented a tendency for improved temperatures but also reported fluctuations over time in 25% of the practices with initially optimal temperatures [15, 29]. Similar fluctuations were observed in our monitoring only group (27.3%). In both foreign studies, the intervention comprised the distribution of national guidelines and serial feedback sessions with temperature graph and advice. In contrast to these interventions, our eLearning KeepCoool is much more cost-effective, yet additional reinforcement strategies are needed as practices with cold chain breaches improved significantly but only 13% of the practices were in the target range at follow-up compared to 72.7% in the monitoring only group.
In didactics, the one-size-fits-all approach is changing towards learner adaptive approaches [38]. One approach is tailored learning, which involves the individualization of information for each participants to reach better results regarding the desired behavior change. It is discussed that individualization increases the acceptability of information whereby information is more frequently remembered and discussed [39, 40]. We used this approach for KeepCoool as physicians, practice assistants and those in training differ regarding knowledge and experience. In addition to targeting the professional group and tailoring to baseline knowledge by providing feedback, we used personal address to individualize the learning experience and facilitate behavior change. Such educational approaches have become more widespread with technological advancements [38].
Strengths and limitations
The content of KeepCoool is based on best-practices and guidelines from several countries. The eLearning addressed all practice personnel. A selection bias cannot be excluded as practices participating in the elearning may differ from those who did not. Contamination between practices was unlikely as they are spread in a larger area and work independently, but cannot be fully excluded. Some practices participated in the eLearning beyond the period defined for the primary outcome. We addressed this in the Euclidian model with 2 million temperature data from up to 4.5 months. For the effectiveness, we chose a strict primary outcome according to recommendations for vaccine storage, yet slightly less strict outcomes might be as appropriate for modern vaccines (e.g., in target range 95% of the time while always above the freeze threshold). Data on changes of refrigerators or practice personnel were not obtained to not burden busy practice personnel. However, we assured that only personnel who had participated in the baseline evaluation were provided access to the elearning. The Euclidian distance used the +5°C median as guidelines recommend to target 5°C when adjusting refrigerators. No long-term monitoring, e.g. after 6 to 12 months, and room temperature monitoring were performed but recommended for future studies.
Overall, the eLearning KeepCoool showed high learning effectiveness [27] and significant improvements regarding the vaccine cold chain. The freely available eLearning allows for the nationwide scaling up to all personnel and practices vaccinating: https://keepcoool.ukbonn.de/.
Supporting information
S1 File. Includes both the supporting tables and figures of the manuscript.
https://doi.org/10.1371/journal.pone.0301847.s002
(PDF)
Acknowledgments
We would like to thank all scientists from the Institute of General Practice, University of Duisburg-Essen, and the Institute of Family Medicine and General Practice, University of Bonn, who supported the study. We also owe special thank the physicians and personnel of the participating practices for their participation in the study.
References
- 1.
World Health Organization: Aide mémoire for prevention of freeze damage to vaccines. WHO/IVB/07.09. Geneva 2007.
- 2. Hanson CM, George AM, Sawadogo A, Schreiber B: Is freezing in the vaccine cold chain an ongoing issue? A literature review. Vaccine 2017; 35(17): 2127–33. pmid:28364920
- 3. Boros CA, Hanlon M, Gold MS, Roberton DM: Storage at− 3 C for 24 h alters the immunogenicity of pertussis vaccines. Vaccine 2001; 19(25): 3537–42.
- 4. McColloster P, Vallbona C: Graphic-output temperature data loggers for monitoring vaccine refrigeration: implications for pertussis. Am J Public Health 2011; 101(1): 46–7. pmid:21088272
- 5. Onoja AL, Adu FD, Tomori O: Evaluation of measles vaccination programme conducted in two separate health centres. Vaccine 1992; 10(1): 49–52. pmid:1539460
- 6. McIntyre RC, Preblud SR, Polloi A, Korean M: Measles and measles vaccine efficacy in a remote island population. Bull World Health Organ 1982; 60(5): 767. pmid:6983927
- 7. Paul-Ehrlich-Institut: Ausflückungen in Adsorbatimpfstoffen. Bulletin zur Arzneimittelsicherheit—Informationen aus BfArM und PEI 2012(1): 12–6.
- 8.
World Health Organization: Temperature sensitivity of vaccines. Department of Immunization, Vaccines and Biologicals, World Health Organization 2006: 1–62.
- 9.
EMA: Product Information as approved by the CHMP on 28 May 2021, pending translations and endorsement by the European Commission: ANNEX I—SUMMARY OF PRODUCT CHARACTERISTICS.
- 10. Bell KN, Hogue C, Manning C, Kendal AP: Risk factors for improper vaccine storage and handling in private provider offices. Pediatrics 2001; 107(6): e100. pmid:11389298
- 11. Bishai DM, Bhatt S, Miller LT, Hayden GF: Vaccine storage practices in pediatric offices. Pediatrics 1992; 89(2): 193–6. pmid:1734382
- 12. Gazmararian JA, Oster NV, Green DC, et al.: Vaccine storage practices in primary care physician offices: assessment and intervention. Am J Prev Med 2002; 23(4): 246–53. pmid:12406478
- 13. Haworth EA, Booy R, Stirzaker L, Wilkes S, Battersby A: Is the cold chain for vaccines maintained in general practice? BMJ 1993; 307(6898): 242–4. pmid:8369691
- 14. Lee S, Lim H-S, Kim O, et al.: Vaccine Storage Practices and the Effects of Education in Some Private Medical Institutions. J Prev Med Public Health 2012; 45(2): 78–89. pmid:22509448
- 15. Lewis PR, Reimer RF, Dixon AJ: Evaluating the efficacy of vaccine storage in the general practice setting. Aust N Z J Public Health 2001; 25(6): 547–50. pmid:11824993
- 16. Thakker Y, Woods S: Storage of vaccines in the community: weak link in the cold chain? BMJ 1992; 304(6829): 756–8. pmid:1571683
- 17. Yuan L, Daniels S, Naus M, Brcic B: Vaccine storage and handling. Knowledge and practice in primary care physicians’ offices. Can Fam Physician 1995; 41: 1169. pmid:7647622
- 18. Thielmann A, Sikora M, Schnell U, Gesenhues S, Weltermann B: Impfkühlschrank-und Impfstoffmanagement in Hausarztpraxen: Eine repräsentative, Web-basierte Umfrage unter Hausärzten (KeepCool I). Das Gesundheitswesen 2015; 38: 1–6.
- 19. Weltermann BM, Markic M, Thielmann A, Gesenhues S, Hermann M: Vaccination Management and Vaccination Errors: A Representative Online-Survey among Primary Care Physicians. PloS one 2014; 9(8): e105119. pmid:25118779
- 20.
Centers for Disease Control and Prevention: Vaccine Storage & Handling Toolkit 2014.
- 21.
Salisbury D, Ramsay M, Noakes K: Immunisation against infectious disease. 3rd ed. London: TSO 2006.
- 22.
Australian Government: National vaccine storage guidelines. Strive for 5. 2nd ed. Canberra: Dept. of Health and Ageing 2013.
- 23.
Public Health Agency of Canada: National vaccine storage and handling guidelines: For immunization providers 2015. Ottawa: Public Health Agency of Canada 2015.
- 24.
Health Protection Scotland: Guidance on vaccine storage and handling. Glasgow: Health Protection Scotland 2013.
- 25. Thielmann A, Puth M-T, Kersting C, Birgitta Weltermann: Vaccine cold chain in general practices: A prospective study in 75 refrigerators (Keep Cool study). PloS one. pmid:31743356
- 26. Thielmann A, Puth M-T, Weltermann B: Visual inspection of vaccine storage conditions in general practices: A cross-sectional study of 75 vaccine refrigerators (Keep Cool study). PLoS ONE (2019) 14(12):e0225764.
- 27. Thielmann A, Puth M-T, Weltermann B: Improving knowledge on vaccine storage management in general practices: Learning effectiveness of an online-based program. Vaccine 2020; 38(47): 7551–7. https://www.sciencedirect.com/science/article/pii/S0264410X2031210X?via%3Dihub pmid:33041099
- 28. Thielmann A, Viehmann A, Weltermann BM: Effectiveness of a web-based education program to improve vaccine storage conditions in primary care (Keep Cool): study protocol for a randomized controlled trial. Trials 2015; 16(301). pmid:26169675
- 29. Gold MS, Martin L, Nayda CL, Kempe: Electronic temperature monitoring and feedback to correct adverse vaccine storage in general practice. MJA 1999; 171(2): 83–4.
- 30. Page SL, Earnest A, Birden H, Deaker R, Clark C: Improving vaccination cold chain in the general practice setting. Aust Fam Physician 2008; 37(10): 892. pmid:19002316
- 31.
Rigby RA, Stasinopoulos MD, Heller GZ, Bastiani F de: Distributions for modeling location, scale, and shape: Using GAMLSS in R: CRC press 2019.
- 32. Eilers PHC, Marx BD: Flexible smoothing with B-splines and penalties. Statistical science 1996; 11(2): 89–121.
- 33. Akaike H: Information measures and model selection: Proceedings of the 44th Session of the International Statistical Institute 1983(1): 277–91.
- 34. Fernández C, Osiewalski J, Steel MFJ: Modeling and Inference with υ-Spherical Distributions. Journal of the American Statistical Association 1995; 90(432): 1331–40.
- 35. Rigby RA, Stasinopoulos DM: Generalized additive models for location, scale and shape. Journal of the Royal Statistical Society: Series C (Applied Statistics) 2005; 54(3): 507–54.
- 36. Benjamini Y, Yekutieli D: The control of the false discovery rate in multiple testing under dependency. Annals of statistics 2001: 1165–88.
- 37. Jeremijenko A, Kelly H, Sibthorpe B, Attewell R: Improving vaccine storage in general practice refrigerators. BMJ 1996; 312: 1651–2. pmid:8664720
- 38. Lineberry M, Dev P, Lane HC, Talbot TB: Learner-adaptive educational technology for simulation in healthcare: foundations and opportunities. Simulation in Healthcare 2018; 13(3S): S21–S27. pmid:29346223
- 39. Hawkins RP, Kreuter M, Resnicow K, Fishbein M, Dijkstra A: Understanding tailoring in communicating about health. Health Educ Res 2008; 23(3): 454–66. pmid:18349033
- 40.
Kreuter MW, Farrell DW, Olevitch LR, Brennan LK: Tailoring health messages: Customizing communication with computer technology. Mahwah, New Jersey: Lawrence Erlbaum Associates 2000.