Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Predicting the effectiveness of the online clinical clerkship curriculum: Development of a multivariate prediction model and validation study

  • Naoto Kuroda ,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing – original draft

    naoto.kuroda@wayne.edu

    Affiliations Department of Pediatrics, Wayne State University, Detroit, Michigan, United States of America, Department of Epileptology, Tohoku University Graduate School of Medicine, Sendai, Japan

  • Anna Suzuki,

    Roles Conceptualization, Data curation, Writing – review & editing

    Affiliation Kyushu University School of Medicine, Fukuoka, Japan

  • Kai Ozawa,

    Roles Data curation, Resources

    Affiliation The Jikei University School of Medicine, Tokyo, Japan

  • Nobuhiro Nagai,

    Roles Conceptualization, Data curation, Resources

    Affiliation Faculty of Medicine, Shimane University, Izumo, Japan

  • Yurika Okuyama,

    Roles Data curation, Resources

    Affiliation Akita University School of Medicine, Akita, Japan

  • Kana Koshiishi,

    Roles Data curation, Resources

    Affiliation Asahikawa Medical University, Asahikawa, Japan

  • Masafumi Yamada,

    Roles Data curation, Resources

    Affiliation Faculty of Medicine, Shimane University, Izumo, Japan

  • Makoto Kikukawa

    Roles Conceptualization, Funding acquisition, Supervision, Writing – review & editing

    Affiliation Department of Medical Education, Kyushu University, Fukuoka, Japan

Abstract

Given scientific and technological advancements, expectations of online medical education are increasing. However, there is no way to predict the effectiveness of online clinical clerkship curricula. To develop a prediction model, we conducted cross-sectional national surveys in Japan. Social media surveys were conducted among medical students in Japan during the periods May–June 2020 and February–March 2021. We used the former for the derivation dataset and the latter for the validation dataset. We asked students questions in three areas: 1) opportunities to learn from each educational approach (lectures, medical quizzes, assignments, oral presentations, observation of physicians’ practice, clinical skills practice, participation in interprofessional meetings, and interactive discussions with physicians) in online clinical clerkships compared to face-to-face, 2) frequency of technical problems on online platforms, and 3) satisfaction and motivation as outcome measurements. We developed a scoring system based on a multivariate prediction model for satisfaction and motivation in a cross-sectional study of 1,671 medical students during the period May–June 2020. We externally validated this scoring with a cross-sectional study of 106 medical students during February–March 2021 and assessed its predictive performance. The final prediction models in the derivation dataset included eight variables (frequency of lectures, medical quizzes, oral presentations, observation of physicians’ practice, clinical skills practice, participation in interprofessional meetings, interactive discussions with physicians, and technical problems). We applied the prediction models created using the derivation dataset to a validation dataset. The prediction performance values, based on the area under the receiver operating characteristic curve, were 0.69 for satisfaction (sensitivity, 0.50; specificity, 0.89) and 0.75 for motivation (sensitivity, 0.71; specificity, 0.85). We developed a prediction model for the effectiveness of the online clinical clerkship curriculum, based on students’ satisfaction and motivation. Our model will accurately predict and improve the online clinical clerkship curriculum effectiveness.

Introduction

Given scientific and technological advancements, the medical education community’s expectations of online education are rising [13]. In addition, coronavirus disease 2019 (COVID-19) has necessitated the use of the online platform in medical education [48]. Globally, online clinical clerkship was proposed to replace face-to-face clinical clerkship during the COVID-19 pandemic (mid-COVID-19) [9]. Various opinions and practical innovations have been reported in relation to making online clinical clerkship a viable alternative to face-to-face clerkship [1012]. However, there are currently no scores that can predict the effectiveness of online clinical clerkship curricula. Developing such predictive scores would make online clinical clerkship more effective.

We aimed to develop and externally validate a prognostic score that represents students’ satisfaction and motivation, and can predict the effectiveness of the online clinical clerkship curriculum.

Material and methods

Survey design and data collection

This study followed the principles of the Declaration of Helsinki and was approved by Kyushu University’s ethics committee. To achieve our objectives, we conducted cross-sectional surveys. We developed the survey questions with a focus on content validity through an iterative process within the team, which included a medical educator. In addition, we administered the survey to a pilot group of 30 medical students. During a semi-structured interview after the survey was administered, these students were asked to provide feedback on the clarity of formulation of the items and on the format of the questionnaire.

The study consisted of two parts. First, we collected the survey responses for the derivation dataset. Through derivation analysis, we developed a scoring model for the online curriculum to predict medical students’ satisfaction and motivation. We then collected the survey responses for the validation dataset. Developed using the derivation dataset, we assessed our predictive model’s prediction performance via validation analysis.

The study participants were undergraduate medical school students in Japan. The survey that yielded the derivation dataset was conducted from May 29, 2020 to June 14, 2020. This period was chosen because the first declaration of a state of emergency due to COVID-19 in Japan (April 7–May 25, 2020) led to the widespread use of online clinical clerkship. We conducted the survey that yielded the validation dataset from February 26 to March 11, 2021. This period was chosen because the second declaration of a state of emergency due to COVID-19 in Japan (January 7–March 21, 2021) also led to the widespread use of online clinical clerkship.

Both surveys have identical content. The questionnaire was presented along with an informative letter summarizing the research purpose, as well as the informed consent form, written briefly and clearly in Japanese to avoid misinterpretation. The questionnaire took 3–5 min to complete.

The inclusion criteria were as follows: students (1) who switched from face-to-face to online clinical clerkship mid-COVID-19 and (2) completed all the survey questions. We included fifth- and sixth-year medical students in the derivation dataset and fourth- and fifth-year students in the validation study. This is because in Japan, medical students graduate in March, so sixth-year students often do not participate in clinical clerkships between January and March. The Japanese undergraduate medical education curriculum is six years period. It usually consists of four/three and half years of preclinical education and two/two and half years of clinical clerkship. Clinical clerkship has medical students participate as members of a medical team in a variety of clinical settings. Medical students can acquire clinical competencies through clinical clerkship.

Survey items

The questionnaire asked respondents to share information about their demographic information. If participants were medical students who switched from face-to-face to online clinical clerkship, we inquired as to the effectiveness of online compared to face-to-face clinical clerkship and, how frequently they had various experiences while completing their online clinical clerkship.

Exposure factors and other factors

We examined the extent of students’ exposure to factors associated with the effectiveness of online clinical clerkship. We decided which factors should be included in our survey based on previous literature and recommendations for medical education. The factors incorporated into this study included lecture duration [1315] and lecture frequency per week [13, 16, 17]; opportunities to take quizzes [18, 19], submit assignments [20, 21], give oral presentations [22, 23], observe physicians’ practice [10, 24], practice clinical skills [10, 24], participate in interprofessional meetings [2426], and interact with physicians [24, 27]; and the frequency of technical Internet-related problems [28]. For details regarding the options for each answer, see S1 Method

Measuring outcomes

Based on Kirkpatrick’s assessment model for assessing the usefulness of medical education and medical curricula [26, 2931], we used two outcome measures of the effectiveness of online clinical clerkship: satisfaction level (Level 1 in Kirkpatrick’s assessment model) and motivation level (Level 2a in Kirkpatrick’s assessment model). In questions about satisfaction and motivation, the answers were based on a 5-point Likert scale, ranging from 1 = the level/amount was much higher during face-to-face clerkship than it was during online clinical clerkship and 5 = the level was much higher during online clinical clerkship than it was during face-to-face clerkship.

Total number of respondents and the number of respondents included in the study (Fig 1)

thumbnail
Fig 1. A flowchart showing the selection of suitable participants for inclusion in this study.

To compile the derivation dataset, we collected responses from 2,640 Japanese medical students. Among the 2,640 respondents, 2,594 (98.3%) consented to participate in this study. Respondents with answers missing from their profile were excluded from the analysis (n = 127). In Japan, fifth- and sixth-year students are the equivalent years of practice, so we also excluded responses from students in years below the fifth year (n = 19). The next step was to categorize the responses according to the type of education the students received mid-COVID-19. Sixteen respondents were excluded for inappropriate responses. Of the 2,432 remaining respondents, 1,758 reported that they had experienced a face-to-face clerkship in a hospital pre-COVID-19 and then switched to an online clinical clerkship due to the COVID-19 crisis. Of the 1,758 respondents, those with missing responses regarding the content of the education to which they were exposed were excluded (n = 87). Finally, 1,671 responses were used for derivation analysis in this study.

https://doi.org/10.1371/journal.pone.0263182.g001

For the derivation dataset, we collected responses from 2,640 Japanese medical students. Among the 2,640 respondents, 2,594 (98.3%) consented to participate in this study. The sample size was calculated based on a 99% confidence interval (CI) and a 5% margin of error. The student population enrolled in their fifth and sixth years at Japanese medical universities totaled 18,195. The required sample size was 642, and in this study, the total number of respondents was 2,594, which is 4-fold larger than required. Respondents with missing answers were excluded from the study. Of the remaining respondents, we included those who indicated having experienced a face-to-face clerkship in a hospital prior to the first emergency declaration and then subsequently switching to an online clinical clerkship. Finally, 1,671 responses were used in the derivation analysis (Fig 1).

For the validation dataset, we collected responses from 132 Japanese medical students. Among the 132 respondents, 130 (98.5%) consented to participate in this study. Of the remaining respondents, we included those who indicated having experienced a face-to-face clerkship in a hospital prior to the second emergency declaration and then subsequently switching to an online clinical clerkship. Finally, 106 responses were used for the validation analysis (Fig 1).

To compile the validation dataset, we collected responses from 132 Japanese medical students. Among the 132 respondents, 130 (98.5%) consented to participate in this study. Respondents with answers missing from their profile were excluded from the analysis (n = 0). In Japan, fourth- and fifth-year students are the equivalent years of practice, so we also excluded responses from students in other years (n = 0). The next step was to categorize the responses according to the type of education the students received mid-COVID-19. No respondents were excluded due to an inappropriate response. Of the 130 remaining respondents, 107 reported that they had experienced a face-to-face clerkship in a hospital pre-COVID-19 and then switched to an online clinical clerkship due to the COVID-19 crisis. Of the 107 respondents, those with missing responses regarding the content of the education to which they were exposed were excluded (n = 1). Finally, 106 responses were used for validation analysis in this study.

Statistical analysis

We used SPSS v27 (IBM, Armonk, NY, USA). Statistical significance was set at a two-sided p-value of 0.05.

Using the derivation dataset, we performed a multivariate logistic regression analysis to identify the factors associated with each outcome measurement of the effectiveness of online clinical clerkship, per the outcomes defined in the Measuring Outcomes section. We categorized the outcomes into the following binary: equivalent/better than face-to-face clerkship or not. The predictive variables included (a) lecture duration, (b) lecture frequency, (c) quizzes, (d) assignment submission, (e) student oral presentations, (f) observation of practice, (g) clinical skills practice, (h) participation in interprofessional meetings, (i) interactive discussions with physicians, and (j) technical Internet-related problems. We categorized the answers to (a)–(h) into three categorical variables: more opportunities in face-to-face clerkship, same opportunities in online as in face-to-face clinical clerkship, and more opportunities in online clinical clerkship. We categorized (j) (technical Internet-related problems) into three variables: not at all, a little, and a lot. In this study, we did not incorporate variables related to medical students’ profiles because doing so would not have been in keeping with the study’s aim to develop a prediction model for the effectiveness of the online clinical clerkship curriculum.

Based on multivariate logistic regression analysis, we only incorporated associated factors (p < 0.1) into the final prediction model. Using multivariate logistic regression analysis, the final model determined point values for each associated variable and developed an integer-based estimation system for satisfaction and motivation, respectively.

By applying this model to the validation dataset, we assessed the prognostic instrument’s predictive accuracy, with discrimination. Discrimination (i.e., the degree to which a model differentiates between participants answering equivalent/better satisfaction/motivation in face-to-face clerkship) was calculated using concordance (c-) statistics, ranging from 0.5 (no discrimination) to 1.0 (perfect discrimination).

Results

Table 1 presents the participants’ demographic information and the proportion of respondents that selected each answer to every survey question.

thumbnail
Table 1. Summary of respondents’ answers in each survey.

https://doi.org/10.1371/journal.pone.0263182.t001

Identifying the factors associated with satisfaction/motivation level regarding online clinical clerkship

S1 Table shows the results of the multivariate logistic regression analysis for students’ satisfaction with online clinical clerkship. The positive factors associated with satisfaction were lecture frequency (estimate = 0.054; p = .010), quizzes (estimate = 0.291; p = .001), student oral presentations (estimate = 0.175; p = .046), observation of practice (estimate = 0.584; p = .001), clinical skills practice (estimate = 0.594; p = .002), participation in interprofessional meetings (estimate = 0.421; p = .003), and interactive discussion with physicians (estimate = 0.494; p < .001). The negative associated factor was technical problems (estimate = -0.155, p = .043).

S2 Table shows the results of the multivariate logistic regression analysis for students’ motivation with regard to online clinical clerkship. The positive factors associated with motivation were quizzes (estimate = 0.264; p = .002), student oral presentations (estimate = 0.313; p < .001), observation of practice (estimate = 0.427; p = .010), clinical skills practice (estimate = 0.657; p = .001), participation in interprofessional meetings (estimate = 0.497; p = .001), and interactive discussion with physicians (estimate = 0.338; p < .001). The negative associated factor was technical Internet-related problems (estimate = -0.270; p < .001).

Final prediction model

Having identified the associated variables via multivariate logistic regression analysis, we again performed multivariate logistic regression analysis, this time incorporating only the associated variables identified in the first analysis. In the prediction model for satisfaction, eight predictors remained in the final multivariable model after simplification: lecture frequency (estimate = 0.049; p = .007), quizzes (estimate = 0.321; p < .001), oral presentations (estimate = 0.192; p = .027), observation (estimate = 0.583; p = .001), practice (estimate = 0.586; p = .002), interprofessional meetings (estimate = 0.416; p = .004), interactive discussion (estimate = 0.492; p < .001), and technical problems (estimate = -0.158; p = .038) (S3 Table). In the prediction model for motivation, seven predictors remained in the final multivariable model after simplification: quizzes (estimate = 0.270; p = .001), oral presentations (estimate = 0.331; p < .001), observation (estimate = 0.422; p = .010), practice (estimate = 0.651; p = .001), interprofessional meetings (estimate = 0.497; p = .001)), interactive discussion (estimate = 0.358; p < .001), and technical problems (estimate = -0.248; p = .001) (S4 Table). Based on the results of this final multivariate logistic regression analysis, we assigned point values to these variables and developed an integer-based estimation system for satisfaction and motivation, respectively (Table 2).

Validation analysis to assess the scoring model’s prediction performance

We determined our predictive instrument’s discrimination by applying it to the validation dataset. Based on these scores, we created receiver operating characteristic (ROC) curves to predict whether students’ satisfaction/motivation was equivalent/better than in face-to-face clerkship, as shown in Fig 2.

thumbnail
Fig 2. The prediction models’ accuracy for satisfaction and motivation.

We applied the scoring prediction model developed using the derivation dataset to the validation dataset. Based on the model’s prediction performance in the validation dataset, we created receiver operating characteristic (ROC) curves for satisfaction and motivation. (A) A given ROC curve delineates predictive accuracy for satisfaction. (B) A given ROC curve delineates predictive accuracy for motivation. AUC: area under the curve.

https://doi.org/10.1371/journal.pone.0263182.g002

The c-statistics, which indicate the area under the curve to predict satisfaction and motivation, were 0.69 · (95% CI: 0.55–0.83, p = .005) and 0.75 · (95% CI: 0.60–0.90, p = .001), respectively. When the cut-off value for the satisfaction scores was set to 22, sensitivity was 0.50, and specificity was 0.89. When the cut-off value for the motivation scores was set to 17, sensitivity was 0.71 and specificity was 0.85.

Discussion

We developed a practical instrument to predict the effectiveness of online clinical clerkship based on medical students’ satisfaction and motivation. Although an instrument to predict the online clinical clerkship curriculum’s effectiveness has not yet been developed, such an instrument is needed, especially given the increase in demand for online clinical clerkship due to the COVID-19 pandemic.

As the results show, our model’s prediction performance was relatively high (satisfaction: 0.69, motivation: 0.75). Satisfaction and motivation are based on individual self-assessment, which means that different people can rate their satisfaction with and motivation for the same curriculum differently. Individual factors (such as sex and year in medical school) are reported to be related to satisfaction and motivation [30, 32]. Nevertheless, we did not incorporate individual components into the score in our prediction model. This was because our goal was to develop a scoring model that predicts the effectiveness of online clinical clerkship curricula. In short, our model’s prediction performance was high in spite of not incorporating individual demographic information.

Using our scoring system, educators will be able to both predict the effectiveness of and improve their online clinical clerkships. Our instrument will be of great help to educators in building a better online clinical clerkship curriculum. In our study, the best cutoff values for motivation and satisfaction were 22 and 17, respectively (Table 2). Educators can use these scores as guides to improve online clinical clerkship by enhancing the educational approaches in their curriculum, with the goal of exceeding the scores we obtained. For example, to maintain medical students’ satisfaction and motivation during online clinical clerkships in a manner that is comparable to face-to-face clerkship, educators might consider providing the students with the same opportunities to observe physicians’ practice and to practice their clinical skills as face-to-face clinical clerkship (at only these two variables, score for satisfaction:20, and score for motivation:18) (Table 2).

This study has several strengths. The assembled population is one of the largest study sample sizes for predicting the effectiveness of online clinical clerkship. Our instruments were developed using big data from national-level cross-sectional studies involving a total of 1,671 medical students in the derivation dataset and 106 students in the validation dataset. Another strength is that our scoring system is a practical instrument that can be applied to the online clinical clerkship curriculum. The predictors are well defined and easily measured, so that students’ online opportunities to experience each educational approach can be evaluated as equivalent to, or more or less frequent than in face-to-face clerkship. We also made technical troubles simple to evaluate in three frequency categories: never, occasionally, and often.

Our study has some limitations. First, the cross-sectional study retrospectively measures subjects’ exposure and outcome simultaneously. Thus, recall bias was unavoidable. Second, our prediction model is based on medical students’ satisfaction and motivation, which are self-assessed outcomes that are easily measured using a questionnaire. However, knowledge and skill acquisition, which are Level 2b in Kirkpatrick’s assessment model, are more objective outcomes [26, 2931]. Predictive instruments based on such outcomes, as opposed to satisfaction and/or motivation, might be more useful. Therefore, more research might be required to show the correlation with exam scores or patient outcomes before using this study as a tool to modify curricula. The third limitation is that qualitative factors on the part of the faculty were not considered in the current model because the current study was a quantitative survey. For example, the content and quality of lectures, the faculty’s skills of online facilitation might also affect the students’ satisfaction and motivation. Forth, in this study, we only included medical students in Japan. Future research should consider the applicability of these results to other countries. Online clinical clerkship would greatly depend on health care resources, Internet infrastructure in countries/communities, and whether students are familiar with the devices required to access online education [33, 34]. This indicates that our model would be difficult to apply in countries with limited healthcare resources and/or a lack of Internet infrastructure. Another limitation is that these data are based on online clinical clerkship during the COVID-19 pandemic. The impact of COVID-19 varies across countries and communities. Furthermore, online clinical clerkship post-COVID-19 pandemic may be of a different nature (e.g., a combination of face-to-face and online). We believe that our prediction model can be applied in other countries’ online clinical clerkship curriculum or post-COVID-19 pandemic. However, we need to evaluate our model’s validity in other countries and post-COVID-19 pandemic in future studies.

Conclusion

We successfully developed a scoring model to predict the effectiveness of the online clinical clerkship curriculum, based on students’ satisfaction and motivation. Our scoring model will accurately predict and improve the online clinical clerkship curriculum’s effectiveness.

Supporting information

S1 Method. Detailed description of the answer options for each survey question.

https://doi.org/10.1371/journal.pone.0263182.s001

(DOCX)

S1 Table. Multivariate logistic regression analysis to identify the factors associated with maintaining medical students’ satisfaction with online clerkship (Level 1 in Kirkpatrick’s assessment model).

https://doi.org/10.1371/journal.pone.0263182.s002

(DOCX)

S2 Table. Multivariate logistic regression analysis to identify the factors associated with maintaining medical students’ motivation during online clerkship (Level 2a in Kirkpatrick’s assessment model).

https://doi.org/10.1371/journal.pone.0263182.s003

(DOCX)

S3 Table. The final model using multivariate logistic regression analysis to predict medical students’ satisfaction with online clerkship (Level 1 in Kirkpatrick’s assessment model).

https://doi.org/10.1371/journal.pone.0263182.s004

(DOCX)

S4 Table. The final model using multivariate logistic regression analysis to predict medical students’ motivation during online clerkship (Level 2a in Kirkpatrick’s assessment model).

https://doi.org/10.1371/journal.pone.0263182.s005

(DOCX)

Acknowledgments

The authors are grateful to Dr. Eishi Asano at Children’s Hospital of Michigan, Detroit Medical Center, Wayne State University for the advice on statistical analysis. The authors are also grateful to all participants of this survey-based study.

References

  1. 1. Chen BY, Kern DE, Kearns RM, et al. From modules to MOOCs: application of the six-step approach to online curriculum development for medical education. Acad Med. 2019;94(5):678–685. pmid:30681454
  2. 2. Pickering JD, Henningsohn L, DeRuiter MC, et al. Twelve tips for developing and delivering a massive open online course in medical education. Med Teach. 2017;39(7):691–696. pmid:28485629
  3. 3. Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online. 2019;24(1):1666538. pmid:31526248
  4. 4. Daniel M, Gordon M, Patricio M, et al. An update on developments in medical education in response to the COVID-19 pandemic: a BEME scoping review: BEME Guide No. 64 published online January 26, 2021. Med Teach. 2021;43(3):253–271. pmid:33496628
  5. 5. Gibbs T. The Covid-19 pandemic: provoking thought and encouraging change. Med Teach. 2020;42(7):738–740. pmid:32608302
  6. 6. Gordon M, Patricio M, Horne L, et al. Developments in medical education in response to the COVID-19 pandemic: a rapid BEME systematic review: BEME Guide No. 63. Med Teach. 2020;42(11):1202–1215. pmid:32847456
  7. 7. Tolsgaard MG, Cleland J, Wilkinson T, Ellaway RH. How we make choices and sacrifices in medical education during the COVID-19 pandemic. Med Teach. 2020;42(7):741–743. pmid:32442052
  8. 8. Yang DY, Cheng SY, Wang SZ, et al. Preparedness of medical education in China: lessons from the COVID-19 outbreak. Med Teach. 2020;42(7):787–790. pmid:32468890
  9. 9. Rose S. Medical student education in the time of COVID-19. JAMA. 2020;323(21):2131–2132. pmid:32232420
  10. 10. Chao TN, Frost AS, Brody RM, et al. Creation of an interactive virtual surgical rotation for undergraduate medical education during the COVID-19 pandemic. J Surg Educ. 2021;78(1):346–350. pmid:32654999
  11. 11. Co M, Chu KM. Distant surgical teaching during COVID-19—a pilot study on final year medical students. Surg Pract. 2020;24:105–109. pmid:32837531
  12. 12. Jack MM, Gattozzi DA, Camarata PJ, et al. Live-streaming surgery for medical student education—educational solutions in neurosurgery during the COVID-19 pandemic. J Surg Educ. 2021;78(1):99–103. pmid:32747320
  13. 13. Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007;27(1):6–15. pmid:17385735
  14. 14. Mustafa T, Farooq Z, Asad Z, et al. Lectures in medical educaton: what students think? J Ayub Med Coll Abbottabad. 2014;26(1):21–25. pmid:25358209
  15. 15. Hashmi NR, Daud S, Manzoor I. Medical education: views and recommendations by final year MBBS students of a private medical college in Lahore. J Coll Phys Surg Pak. 2010;20(2):93–97. pmid:20378035
  16. 16. Najmi RS. Lecture as a mode of instruction in undergraduate medical education. J Pak Med Assoc. 1999;49(2):30–33. pmid:10513433
  17. 17. Manzoor I, Mumtaz A, Habib M, et al. Lectures in medical education: students’ views. J Ayub Med Coll Abbottabad. 2011;23(4):118–121. pmid:23472432
  18. 18. Karpicke JD, Roediger HL, III. The critical importance of retrieval for learning. Science. 2008;319(5865):966–968. pmid:18276894
  19. 19. Larsen DP, Butler AC, Roediger HL III. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ. 2009;43(12):1174–1181. pmid:19930508
  20. 20. Epp S. The value of reflective journaling in undergraduate nursing education: a literature review. Int J Nurs Stud. 2008;45(9):1379–1388. pmid:18325522
  21. 21. August E, Burke K, Fleischer C, et al. Writing assignments in epidemiology courses: how many and how good? Public Health Rep. 2019;134(4):441–446. pmid:31112451
  22. 22. Plack MM, Driscoll M, Marquez M, et al. Peer-facilitated virtual action learning: reflecting on critical incidents during a pediatric clerkship. Acad Pediatr. 2010;10(2):146–152. pmid:20206914
  23. 23. Singh K, Bharatha A, Sa B, et al. Teaching anatomy using an active and engaging learning strategy. BMC Med Educ. 2019;19(1):149. pmid:31096975
  24. 24. Reed T, Pirotte M, McHugh M, et al. Simulation-based mastery learning improves medical student performance and retention of core clinical skills. Simul Healthc. 2016;11(3):173–180. pmid:27093509
  25. 25. Mann KV, Mcfetridge-Durdle J, Martin-Misener R, et al. Interprofessional education for students of the health professions: the “Seamless Care” model. J Interprof Care. 2009;23(3):224–233. pmid:19387906
  26. 26. Peiris-John R, Selak V, Robb G, et al. The state of quality improvement teaching in medical schools: a systematic review. J Surg Educ. 2020;77(4):889–904. pmid:32057742
  27. 27. Alsoufi A, Alsuyihili A, Msherghi A, et al. Impact of the COVID-19 pandemic on medical education: medical students’ knowledge, attitudes, and practices regarding electronic learning. PLOS ONE. 2020;15(11):e0242905. pmid:33237962
  28. 28. Dost S, Hossain A, Shehab M, et al. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11):e042378. pmid:33154063
  29. 29. Kirkpatrick DL. Evaluation of training. In:, Craig RL, Bittel LR, eds. Training and Development Handbook. New York, NY: McGraw-Hill; 1967:87112.
  30. 30. Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach. 2008;30(2):124–145. pmid:18464136
  31. 31. Wong BM, Etchells EE, Kuper A, et al. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85(9):1425–1439. pmid:20543652
  32. 32. Çalışkan F, Mıdık Ö, Baykan Z, et al. The knowledge level and perceptions toward COVID-19 among Turkish final year medical students. Postgrad Med. 2020;132(8):764–772. pmid:32657235
  33. 33. Walsh S, De Villiers MR, Golakai VK. Introducing an e-learning solution for medical education in Liberia. Ann Glob Health. 2018;84(1):190–197. pmid:30873817
  34. 34. Barteit S, Neuhann F, Bärnighausen T, et al. Perspectives of nonphysician clinical students and medical lecturers on tablet-based health care practice support for medical education in Zambia, Africa: qualitative study. JMIR MHealth UHealth. 2019;7(1):e12637. pmid:30664475