Research questions

1. Why were pre-registration medical programs evaluated?
2. How do pre-registration medical programs get evaluated?
3. What aspects of pre-registration medical programs get evaluated?

Study selection

We used Covidence systematic review software to screen, collate and chart included studies (Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org, Australia). We completed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) flow diagram (Fig 1) [9].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flow diagram for study selection.

https://doi.org/10.1371/journal.pone.0305996.g001

Extraction, analysis, and screening

To extract and analyse data we followed Arksey and O’Malley’s process [7]. Title and abstract screening were conducted independently by two authors (SKG & JSP). Members of the team were assigned articles for full-text screening using Covidence Software (Melbourne VIC, Australia). Each article was screened by two members of our team (All authors). A third reviewer checked and made decisions on discrepancies (TJ). These decisions were cross-checked by two reviewers (JSP & SKG). A list of excluded studies and the reasons for exclusions are available in S2 File. Two authors (TJ & JSP) extracted data on article characteristics (including author, year, country of research, evaluation purpose, study purpose, study design and model used) and identified themes in line with the ACGME definition [1], including study purpose (apply/develop/review/validate), design, model used and whether an explicit model was referenced. In addition to this, as per Arksey and O’Malley’s guide for collating and summarizing information [7], two authors (TJ & JSP) identified qualitative patterns in the included studies. We grouped findings into themes inductively and in relation to our research questions. Disagreements were discussed with a third party (SKG) and then the wider research team. The extracted data is available in S3 File. All studies identified in the search is available in S4 File.

Research team positionality & reflexivity

The research team comprised medical professionals and nurses, social and biomedical scientists, all of whom currently work in medical education in Australia. Some of the research team have extensive experience in curriculum design (TJ, JB, MM, JSP) and program evaluation (JB, ML). We hold a perspective that the purpose of healthcare education is both to improve the quality of healthcare and to promote the professional self-development of healthcare professionals.

From 2261 identified studies, thirty-two were included for review using the inclusion criteria outlined in Table 1. These characteristics of the included studies are shown in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of the included studies.

https://doi.org/10.1371/journal.pone.0305996.t002

Why were pre-registration medical programs evaluated?

Included studies were primarily concerned with either proving efficacy of their program (i.e., demonstrate performance relative to a standard) (n = 21) or improving quality of programs (n = 11) (Table 2). No studies were concerned with comparing programs across universities. The stated aims of the 32 studies were to review existing literature (n = 9), develop a new evaluation model (n = 7), or to apply (n = 12) or validate (n = 4) an existing model (or part thereof) (Table 2). The stated drivers for evaluation were continuous quality improvement [10, 12, 14, 23, 31, 39], benchmarking to national and international standards [5, 12, 38] accreditation [10, 18, 39] and quality assurance [29, 31].

How do pre-registration medical programs get evaluated?

Twenty-two of the 32 studies (69%) explicitly identified an evaluation model they had used or would recommend (Table 2). The models most reported were Context-Input-Process-Product (CIPP) [19, 21, 25, 36, 39], Kirkpatrick [5, 33], World Federation Medical Education (WFME) [20, 29], and the Standard, by the Joint Committee on Standards for Educational Evaluation (JCSEE) [26]. One third of the included studies did not draw on an existing evaluation model.

Most studies were conducted via standardised student evaluation surveys [14, 17, 18, 22, 25, 30, 33, 36], whereas some studies reported that they used specific tools to evaluate the quality of curriculum content. For example, the Progress Test Medizin (PTM) [28] was used to evaluate quality within the clinical component of the curriculum. Steketee (2015) developed a curriculum mapping software to support and facilitate curriculum management [32]. As a component of curriculum evaluation, Findyartini (2015) assessed the validity of the collaborative Progress Tests conducted in three medical schools [13].

Tavakol et al (2010) highlighted the importance of economic analyses as a key component of program evaluation. It was suggested that cost-benefit analysis should be conducted on elements of the program including educator selection, technological tools in education, and the learning environment [35].

In another approach, Sullivan et al 2019 utilised multiple methods to evaluate their medical program, including pre-post surveys, focus groups and curriculum content analyses. To achieve this, the evaluation was funded as a standalone project to ensure the quality of a new medical program [33]. Even with this funding, they highlighted the difficulties of achieving a comprehensive program evaluation. Echoing the concerns of Mirzazadeh et al and Sullivan et al, Toosi et al highlight the time intensive nature of the CIPP model in addition to the omission of key stakeholders [36]. Fetterman described five tools for evaluation: culture of evidence, using a critical friend, encouraging a cycle of reflection and action, cultivating a community of learners and developing reflective practitioners [12].

Six (19%) studies utilised quantitative methodology [11, 13, 28, 34, 38, 39], three (9%) studies utilised qualitative methodology [10, 16, 29], seven (22%) used mixed methods [21, 22, 25, 26, 33, 36, 37], and eleven (34%) reported no methods because they were reflections and methods were not necessary [5, 12, 14, 15, 17, 18, 20, 23, 24, 32, 35].

What aspects of pre-registration medical programs get evaluated?

Studies included in this review concentrated on whole of medical program [5, 6, 10, 18, 21, 22, 26, 31, 34, 35], medical school curricula [5, 13, 24, 33, 39], evidence-based medicine teaching [24], or clinical programs within a broader medical program [30, 37]. Within these broad purviews, studies evaluated various facets: student learning outcomes, graduate outcomes [5, 13, 14, 28], student attitudes [5], learning environment and facilities [5, 39], human resources [39], student mistreatment [5], program metrics such as attrition [5], core curricular content [16, 18, 33, 39], knowledge retention [28], competency-based medical education (CBME) program [23, 37], clinical teaching [30] and systematic course evaluation [15]. Outcomes that were evaluated were largely learning outcomes [33], student performance [12, 18, 21], student experience [5, 12, 14, 18, 21, 25], educator workload [14], social accountability [39] and cost of program [21, 35].

Meta-evaluation of medical programs involves the systematic assessment and analysis of the evaluation processes and findings of multiple evaluations conducted within the medical education domain. Instead of evaluating the medical program itself, meta-evaluation examines the evaluations performed on the program, focusing on their methods, quality, and outcomes. Four (13%) of the included papers concentrated on meta-evaluation, within the aspects of: continuous quality improvement [10]; assessment [13]; or whole-of-program [26, 29]. Ruhe et al. conducted a meta-evaluation by evaluating the McGill Evaluation model against JCSEE standards and highlighted the importance of achieving a balance between utility, feasibility, propriety, accuracy and evaluation accountability [26]. Akdemir et al. reviewed continuous quality improvement (CQI) by evaluating American and Canadian undergraduate accreditation systems and identified pros and cons of CQI [10].

Most studies used student evaluation surveys [14, 17, 18, 22, 25, 30, 33, 36] to evaluate quality, and one explored links between the program and patient outcomes [27]. Only five studies explored faculty experiences, perspectives, or educational quality [14, 20, 30, 36, 39]. For programs that implemented a content review [24, 33, 36], there was significant time and resource investment undertaken, which they noted as unsustainable.

Overall, current literature is heavily focused on evaluation of the curriculum and are mainly conducted via student-targeted survey-based methods, with restricted stakeholder involvement. Few studies reported patient perspective [27] or economic perspective as markers of quality [21]. There is very limited evaluation on staff wellbeing or workload [14]. Only four studies discussed burden of evaluation (economic, cognitive and administrative) [10, 13, 26, 29]. None of the studies identified in this review reported on diversity, equity and inclusion, or digital capability/citizenship (an individual’s ability to use digital systems) [40] strategies of medical programs.

What are the costs of medical program evaluation?

Four evaluation models featured in pre-registration program evaluations: CIPP, Kirkpatrick, WFME and SJCSEE. This might suggest that they are the most popular ‘cleaning’ products available. But an important question is: to which extent does such popularity reflect their quality and value (for money)? CIPP is a generic evaluation model which can be adapted for use across a wide range of applications including but not limited to accreditation standards, curricular innovations, assessments, resources, finances, or student experience [21, 25, 26]. Drawing on the CIPP model, Mirzazadeh et al. additionally collected information about how much time and cost was associated with undertaking the evaluation. Findings from their cost analysis highlight sustainability as an important consideration for medical schools when using CIPP as the evaluation model [21]. Thus, considering value for money, the CIPP model (Table 2), was reported as more time intensive than other models [21, 36]. So, while that product may enable a more sparkling end-result, the cost is time.

What are the strengths and weaknesses of different models/frameworks used in medical program evaluation?

The Kirkpatrick model for evaluation concentrates on establishing learners’ reactions, learning, behaviour, and their impacts on others. The included studies that utilised Kirkpatrick or the modified Kirkpatrick models reported largely on learner outcomes [5, 6, 11, 33]. However, two of the included studies concluded that although they used the Kirkpatrick model, they found it too linear and hierarchical, noting it did not consider a comprehensive perspective to understand why programs succeed or fail [6, 11]. In fact, in their review of the use of Kirkpatrick for program evaluation, Allen et al. state “the Kirkpatrick Model […] should not be the gold standard for program evaluation” [6]. Thus, since Kirkpatrick is used to assess outcomes of learning at four predetermined levels (noted above), its use is limited to the areas of curriculum or educational interventions [5, 33].

In contrast, WFME is a more prescriptive evaluation guide which includes components of curriculum, assessment, students, staff and other generic program components such as mission and values, quality assurance and governance. Therefore, WFME has been used for wider program evaluation and quality assurance [29]. JCSEE has been mainly used for wider program meta evaluations due to the focus on utility, feasibility, propriety, accuracy and accountability of evaluations, thus limiting its use in primary evaluations of core program areas [26]. While one might argue that WFME and JCSEE are standards, and not models, we found that the standards were used as a de facto model in the absence of any other prescribed approach to program evaluation. Perhaps this uncovers another phenomenon in program evaluation: that program evaluation occurs under the auspices of accreditation, and the documentation of such is seldom represented in published literature due to its sensitive nature.

What should be the driver of medical program evaluations?

Our results raise additional questions for the medical education community. Is the purpose of program evaluation to identify the quality or success of a program? Is it to meet the requirements of accreditation bodies? We found evaluations were largely accreditation driven, reactive, episodic, and student focused. The implicit understanding that emerged from the included studies is that quality of a program can be assessed through students’ knowledge and reported satisfaction with their education. These are core measures that should be part of program evaluation. However, as Allen et al. point out, these do not explain contributing factors nor the complex interplay of curriculum content, environment, and the context of learning [6]. We agree. The current literature largely omits these considerations: satisfaction of other stakeholders, learner and staff wellbeing, equity and diversity. Few studies undertook meta-evaluation. We would encourage those in program evaluation to incorporate such considerations, which we outline further below.

What is missing in the evaluation of pre-registration medical programs?

Staff and learner wellbeing is a significant contributor to, and determinant of program success, especially due to the increasing psychological distress, workload and burnout identified among higher education communities due to ineffective academic environments exacerbated by lack of appropriate policies [41, 42]. Markers of quality within this domain might include equitable access to support and significance given to staff/learner wellbeing within the program [43, 44]. The learning environment and culture impacts on how learners engage and perform [45]. Existing models can be utilised to consider program, faculty and universities key performance indicators (KPIs) [43].

Diversity, Equity and Inclusion (DEI) are important as markers of quality, since a diverse and equitable learner/staff body helps to model to students the needs of the heterogenous healthcare industry [46]. The included studies have not focused on inclusion or empowerment in their evaluations. The evaluation literature offers a limited picture of how medical programs are promoting DEI. Another reason for a scarcity of DEI data in evaluation literature may be the hesitancy of institutions to publish data that could be interpreted as negative or insufficient. We call for program evaluators to include evaluation of DEI strategies. For evaluators who want to include such aspects, we recommend they underpin evaluations with realism and transparency [47, 48].

None of the included studies evaluated their programs in terms of student digital capability and citizenship [40]. In the context of the recent boom in Artificial Intelligence, and accrediting bodies requiring digital capability, we also call for medical programs to evaluate strategies they have in place for promoting digital citizenship.

Meta-evaluation is important to answer the question: is the evaluation process working? We recommend medical programs undertake periodical meta-evaluation of the evaluation model, evaluation process and evaluation burden (cost, cognitive and administrative). As Tavakol et al. suggest, evaluations should ensure there is a return on investment for the medical program delivered [35]. Meta-evaluation was the focus of four articles [10, 13, 26, 29], in which financial, cognitive and administrative burden of evaluation has not been covered. In contrast, when studies concentrate on meta-evaluation of the financial burden [21, 35], aspects of evaluation process, cognitive and administrative burden are largely omitted. It is vital to compromise between conducting effective evaluation and evaluation burden, in a resource limited health/ higher education system, where time spent on evaluation could compete with time spent in teaching and clinical care.

Who should be involved in the evaluation of pre-registration medical programs and when?

In terms of stakeholders, we suggest that qualified practicing healthcare professionals and consumers of healthcare are also key stakeholders who can offer important insights into the quality of a given medical program. Quality improvement science suggests that there is a continuum of stakeholder engagement: non-participating, symbolic, and engaged participation [49]. This continuum proposes that there are varying levels of stakeholder engagement, that range from information sharing to collaborative patient engagement models, such as community-based participatory research. Stakeholders with different levels of engagement offer different perspectives on the quality of the program. Medical programs might source broader stakeholder engagement from university academics or professional staff (internal and external to the program e.g. accrediting bodies), adjunct clinical professors, hospital-based clinicians involved in education delivery, simulated participant programs and healthcare consumers (patients and their carers).

How should we implement program evaluation?

In terms of the logistics of program evaluation, authors who used the CIPP model highlighted the time-intensive nature of comprehensive program evaluation. This implores the question of how often program evaluation should occur and what should trigger a more comprehensive review? None of the included papers described a timing cycle for review, nor did they identify thresholds to trigger ad hoc reviews. We propose that accreditation should be the key driver for timing of reviews and should be considered the “proactive” model of program evaluation, as many require annual reporting. If this is not practicable, a 3-yearly cycle may be more appropriate to accommodate for changing science or practice relevant to pre-registration medicine programs. As for reactive models, educators might consider key issues such as: 1) consistent, significant worsening student performance outcomes, 2) consistent, widespread student dissatisfaction over a period, 3) resourcing constraints (including physical environments, educational tools and staffing), and/or 4) significant shifts in evidence or knowledge of fields of science.

The current focus of pre-registration medical program evaluation literature is limited to one or two facets of a program. Evaluations are often accreditation-driven, giving utmost prominence to student satisfaction at the expense of other stakeholders. The main areas frequently evaluated are curriculum, learning outcomes and graduate outcomes, learning environments, program metrics, student performance and student experience. Evaluation of staff, well-being, DEI, digital citizenship, finances, and meta evaluation are quite overlooked (dusty) areas within the literature and need attention. Use of broad stakeholder involvement as highlighted above, is encouraged for any evaluation.

Strengths and limitations

We searched a wide range of literature sources on evaluation published since 2000. This allowed us to look at a range of medical education literature and retrieve articles from various countries. However, this also meant that we did not include unpublished evaluation research, which is likely where significant amounts of program evaluation data are reported. We found designing a search strategy for program evaluation quite difficult, due to the variety of methods and terminology used for program evaluation. This challenge meant crafting a search strategy that was focused on the inclusion of the terms “program” and “evaluation”, and thus, our search may not have identified papers that used different terms. Additionally, we restricted our search to English language journal articles and thus may have excluded relevant research in languages other than English.