Introduction

Employers are sceptical about hiring health professionals whose qualifications are obtained through e-learning [1]. Clinical incompetence and poor quality of assessments are often used to support arguments against the employment of such health professionals [2]. These narratives are palatable to the regulators of health professions, who often argue in line with protecting the public from poorly trained and incompetent health professionals [1–3]. However, technological advancements and the recent COVID-19 pandemic have effectuated the rapid uptake of e-learning and distance education into mainstream programmes in the health professions. Strategies need to be in place to evaluate and guarantee quality in these programmes. The focus of this review is on the theoretical components of training health professions inferences are made, however, on the need for integrating clinical practice on e-learning platforms, albeit being a challenge in LMICs.

An educational programme is evaluated against standards to determine its quality [1, 3, 4]. The literature describes various models and frameworks used for assessing the quality of e-learning programmes. The quality matters (QM) model is a prominent model for evaluating e-learning quality and has generated widespread interest through several studies that support its impact [4]. The QM model applies eight standards: the course overview, learning objectives, assessment and measurement, resources, materials, learning engagement, course technology, learner support, and accessibility in evaluating the quality of e-learning programmes. The context, inputs, processes and product (CIPP) model was initially developed by Stufflebeam [5]. Fishbain et al. [6] operationalised it in describing the process of evaluating online programmes, considering the structure, complexity, and cultural differences. Furthermore, essential elements in evaluating the quality of e-learning programmes such as technological infrastructure, institutional support, course design, support of instruction and learning, assessment, educators, administration, and curricular structure have been reported [4, 7, 8]. Fishbain et al. [6] make a futuristic recommendation related to evaluating e-learning programmes for the health professions by including patient outcomes. The above models and frameworks present a plethora of approaches used to evaluate e-learning programmes reflecting various areas of chiasm and divergence. However, the operationalisation of these models and frameworks within undergraduate e-learning programmes in health professions education is complex.

Undergraduate education programmes in the health professions integrate clinical and theoretical components [1]. The clinical detail adds an absorbing layer of complexity in evaluating and maintaining quality in an e-learning programme. The clinical environment where students learn is often distant from the online educator, who relies on the clinicians to support students to develop clinical competence [1, 9]. Kolb and Kolb [10] argue that classroom activities must align with the clinical experience to enhance learning and avoid cognitive dissonance, which interferes with learning and hinders competence development [11]. Thus, students enrolled in undergraduate e-learning in the health professions programmes may experience cognitive dissonance in the clinical setting due to differing resources, approaches to healthcare, disease profile, student support and online educators who may apply different standards. The literature confirms the influence of cognitive dissonance on the perceptions of the quality of educational programmes [12, 13]. Therefore, more specific standards and approaches in determining and ensuring quality in e-learning nursing education may be needed in health professions programmes to learn and demonstrate skills and competencies [1]. However, at the time of Delva et al. ‘s [1] study, there were no reported standards specific to evaluating the quality of undergraduate nursing programmes and no reported entity in the United States responsible for the oversight and continuous evaluation of the quality of online nursing programmes. The unavailability of explicit strategies for evaluating e-learning undergraduate programmes in the United States may be similar in other countries, including Low- and middle-income countries (LMICs). The uniqueness of LMICs is that their challenges are dissimilar from developed countries. Hence context-specific standards will enhance the quality of e-learning.

E-learning provides greater flexibility concerning time, space, language, content, and administrative power than traditional face-to-face learning [14]. Technological advances in various areas, including education in the health professions, and the integration of e-learning and online distance education, is inevitable. However, there is a need to understand the determinants of quality standards in e-learning programmes in the health professions underpinned by the complexity brought on by educators, and students, variations in resources, regulations, and clinical outcomes. This article presents an integrative review that synthesised evidence related to standards used to determine quality in undergraduate e-learning programmes in the health professions from LMICs. We argue that low-resource settings contribute to the discourse related to evaluating quality in undergraduate e-learning programmes in the health professions, which could guide the development of contextually appropriate interventions. Such interventions need to enhance quality and should ensure the regulators and the public of the competence of health professionals trained through e-learning programmes. The purpose of the current review was to synthesise available literature related to standards for evaluating undergraduate e-learning programmes in the health professions in LMICs. The outcomes of this review will be standards that will be used in LMICs to evaluate the quality of e-learning. These standards will then be used to evaluate the quality of e-learning in undergraduate e-learning nursing programmes in LMICs.

Materials and methods

This section seeks to present the approach used in the integrative review for evaluating undergraduate e-learning programmes in the health professions in LMICs. The purpose of an integrative review is to summarize what is known about a topic and communicate the synthesis of literature to a targeted community [16, 17]. An Integrative review synthesizes research and draws conclusions from diverse sources on a topic and provides a more holistic understanding of a specific phenomenon with a focus on current state of evidence of a particular phenomenon [17]. A systematic review differs from an integrative review because it has a single narrow focused clinical question formulated in a PICO format and may include Meta analysis and requires protocol registration [16].

Data extraction

An author-generated custom-made Google form was used to extract data from the included articles. The form was piloted on two articles by both authors and refined to their satisfaction. The characteristics of the articles were author name, the title of the study, and country, but also the purpose of the study, population group, the study design and results or outcomes which were also identified from the included articles. The evaluation models, data collection and data analysis methods, as well as recommendations and limitations, were identified (Table 1). The first author extracted data from the included articles, and the second author checked the work and resolved discrepancies through discussions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Data extraction table.

https://doi.org/10.1371/journal.pone.0281586.t001

Data analysis

This integrative review was underpinned by the contemporary framework for integrative reviews, which was applied throughout the process of analysing data [17]. Whittemore and Knafl [17] suggest a multi-step process for analysing data for an integrative review. Frequencies related to study characteristics were tallied, while an inductive thematic approach was used to identify standards for evaluating e-learning as presented by the included articles (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Data analysis table.

https://doi.org/10.1371/journal.pone.0281586.t002

The identified standards were further synthesised through pattern coding to reflect broader themes and domains related to evaluating e-learning in undergraduate programmes in the health professions.

Results

Selecting the articles

A two-step process involving three reviewers and guided by an inclusion criterion was applied in selecting articles to be included in this review. The two authors and an expert in the education of health professions from a low-income country screened and selected the articles. The study had the following, inclusion criteria focused on the evaluation of undergraduate e-learning programmes in the health professions, should have been published between January 2010 and December 2020 and had to be from LMICs. The literature was excluded from the review based on the following criteria. Literature not related to health sciences, literature on post-graduate programmes, Literature not accessible from the university of the Free State library and Literature dated before 2010. Integrative reviews summarise the latest research, and ten years may be sufficient to provide information that would be useful for future use. The purpose of the criteria was to identify the most appropriate articles to enable us to develop valid and reliable answers to research questions. The primary search strategy identified 610 studies, from which 205 duplicates were excluded through automatic and physical deduplication. Of the remaining 295 records, 269 were eliminated for various reasons, including inappropriate context, different subjects, including inappropriate context, different subjects, articles addressing post-graduate education, articles not from LMICs, article abstracts that the librarian could not get full articles and articles not written in English. Full texts articles were sought from 24 remaining records, of which 16 were eliminated as they did not meet the inclusion criteria. A total of eight articles met the criteria and are included in this review (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA (modified) flow diagram.

https://doi.org/10.1371/journal.pone.0281586.g001

Study characteristics

Based on the studies’ characteristics, three articles from Iran were selected. The other articles were from India, Jordan, and Malaysia. There was a paucity of articles from Africa and South America, where many countries are classified in the low- and middle-income brackets. Most of the studies (n = 3) were published in 2020 and 2021. One article reported on an undergraduate nursing programme [18], while the other three journals reported on an undergraduate medical programme [19]. Only two studies mentioned using models to underpin their evaluation processes, namely, the SWOT analysis [20], the Kirk Patrick model [33] and the force field analysis [21]. Most of the studies applied mixed methods research [18, 21, 22], while others used quantitative methods [18, 23] and qualitative methods [20]. Seven domains, each with a specific theme, were generated inductively after engaging with the individual themes from each included article [32]. Seven domains, each with a specific theme were generated inductively after engaging with the individual themes from each included article. The seven domains are presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Domains in the evaluation of e-learning.

https://doi.org/10.1371/journal.pone.0281586.t003

As expected, a relatively low number of articles were included in this review. This finding could be attributed to the low uptake of e-learning programmes in low-resource settings versus the resource-intensive nature of e-learning programmes [24–26]. Brooks et al. [27] further explain that the scepticism around the quality of health profession graduates from e-learning or distance learning programmes hampers national investments in e-learning programmes, with educational institutions preferring face-to-face programmes. In the same vein, research outputs from low-resource settings are often low; hence there are limited publications in general [2].

Discussions

The purpose of this review was to synthesise available evidence related to standards for evaluating undergraduate e-learning programmes in the health professions in LMICs. The results of the integrated review of standards presented characteristics of the eight studies. They then focused on the seven domains related to the evaluation of undergraduate e-learning programmes in the health professions in LMICs.

The seven evaluation-related domains were curriculum planning, proficiency of the educator, learner proficiency and attitude, infrastructure for learning, support, and evaluation (data in S1 Text). There are similarities between the domains from this review and some of the popular standards used in high-income countries [28]. The similarity in these standards may be due to models and theories related to evaluation that are often transferred from high-income countries to low-income counties under the guise of confirmed validity [20, 22]. In addition, there are reported similarities among undergraduate e-learning programmes in the health professions between high- and low-income countries, supporting the findings from this review on the similarity of evaluation standards.

The domain related to curriculum planning encompasses various aspects of the structure of e-learning programmes. Babadi and Ehraz [23] reflect on the curriculum course plan and course design, educational outcomes and goals, and course content, while Sowan and Jenkins [15] describe assignments as part of curriculum planning. Wasfy et al. [32] describe course specifications linked to the methodology of online teaching and the method of assessment with well-formulated policies and procedures for online courses as desired requirements for e-learning programmes. This domain for evaluating undergraduate e-learning programmes in the health professions in LMICs is aligned with the course overview and standards of learning objectives as defined by the QM model [1]. The domain further aligns with the curricular structure of Capacho et al. [7] and the course design [8]. Prideaux [29] notes three levels of a curriculum: planned, prescribed, or implemented curriculum, which the former is often operationalised in educational programmes through specific outcomes, goals, designs and even assessments. Therefore, it seems common that the structure of the programme–specifically the curriculum plan for an e-learning programme must be included as part of the standards for evaluating undergraduate e-learning programmes in the health professions.

The uptake of e-learning in undergraduate education in the health professions in LMICs has been hampered by the resource-intensive nature of such programmes [25, 30]. The second domain from this review relates to the infrastructure for learning. Several authors described the necessary infrastructure for learning as part of the evaluation standards for their e-learning programmes [20, 22, 23, 31]. This infrastructure included equipment, information technology (IT) and network infrastructure, an independent learning management system and skilful human resources. In addition, other authors focused on the ease of accessing and using a specific infrastructure for learning [20, 22, 23]. Raghuveer and Nirgude [22] further mention the need for a variety of content material to be included and for compliance with principles and intellectual property. Wafsy et al. [32] describe the need for financial resources allocated for online learning. Infrastructure for e-learning is aligned with resources, materials and course technology as described by the QM model, with technology as described by Capacho et al. [7] and with technological infrastructure Kumar et al. [4]. As e-learning is based on the integration of technology in an educational programme, the standards for evaluating e-learning should have a specific focus on technological resources and use within a programme. Insufficient resources for learning, typical in resource-limited settings, compromise the quality of e-learning programmes.

The proficiency of educators is reported as a domain in this study and focuses on the educators facilitating e-learning. Raghuveer and Nirgude [22] reported that educators must be proficient and competent in facilitating e-learning. Authors of all included studies evaluated the online teaching and delivery skills of educators [18–23]. Moreover, the requirement of educators related to e-learning, including their perception and attitude, was included as part of needs assessments. Additionally, an evaluation of professionalism and ethics during the delivery of content was reported as well as the ability of educators to match students’ interests with digital tools. The role of educators in implementing e-learning programmes successfully cannot be overemphasised, as educators have the potential to influence the programme negatively. As part of educator development, continuous training on skills for online teaching is essential [32]. Feedback provided by educators should be constructive and timely to foster efficient implementation by learners [33]. The proficiency of the educator domain identified in this study is similar to that reported by Capacho et al. [7], while the QM model and Kumar et al. [4] are silent regarding the educator’s proficiency. In LMICs, educators are often not well versed in advancements in educational technology and may be pressured to copy and paste face-to-face teaching modalities into the online or e-learning space. Strategies need to be designed and implemented that focus on determining educator needs, developing the educator, and continuously monitoring and evaluating educators on their effectiveness in the online space.

The learners enrolled in an e-learning programme must be proficient in learning through digital means and possess the appropriate attitudes [23]. In this review, the domain of learner proficiency and attitude reflected standards used to evaluate learners in the e-learning space. The sub-themes were active and collaborative learning experiences, personalised learning, focus on learning needs, and the interest and motivation of learners to learn through electronic means. Jebraeily et al. [20] further focused on the process of improved self-learning and problem-solving skills. The learner proficiency domain was not reported in the other popular models for evaluating e-learning [4, 7]. Learners are expected to have some degree of proficiency specifically related to the technology used in their learning. Babadi and Ehraz [23], as well as Sowan and Jenkins [18], state that orienting learners to the learning technologies and subsequent support or maintenance of proficiency is an essential element in student learning. Learner proficiency should be an indispensable standard where institutions must focus on the competence of learners regarding learning technologies. Furthermore, Raghuveer and Nirgude [22] mention the role and value of attitudes towards e-learning, as learners should have an interest in the learning approaches. Moreover, a programme for learner training and a plan for academic counselling that is clear, manageable, and executed is required [32]. Just as in the case of educators, the literature by Babadi and Ehraz [23] specifies that the uptake, use and usefulness of such technology are poor when the learners’ interest in the technology is limited, thus compromising the quality of e-learning. Therefore, standards for e-learning should evaluate learner interest in addition to learner proficiency.

Support is understood as specific strategies, techniques and approaches that assist the attainment of a specific outcome. In this review, the included articles reflected the application of support as part of the standards used in evaluating the quality of e-learning programmes. Learners and their educators are expected to be supported during the teaching and learning activities [18, 20, 21]. The support should be from trained technical teams [32]. The support is focused on learners, specifically relating to technical support [21] and instructor support [18]. Jebraeily et al. [20] further mention the role of administrative support and the need for support gleaned from institutional plans and policies that direct the nature and type of support for learners and educators. These findings are aligned with other models for describing quality in e-learning programmes. The QM model refers to learner support and accessibility as a standard for evaluating the quality of e-learning support. Capacho et al. [7] relate to support in general, while Kumar et al. [4] relate to support of instruction and learning and institutional support. All e-learning programmes should support learners, and in the evaluation of quality, the nature and type of support need to be made explicit and should be accessible for learners and educators. The support should facilitate quality e-learning within the expected contextual remits.

The last domain, which focuses on the evaluation, provides institutions, programmes, and educators with information on the engagement with learners and the value of their e-learning programmes. In evaluating the quality of e-learning programmes, studies included in this review reflected evaluation elements using various parameters. Raghuveer and Nirgude [22] relate to the pattern of computer and internet usage, while Mousavi et al. [21] comment on issues related to programme [effectiveness, safety, convenience, and student satisfaction [18]]. The integration of internal reviewers, to monitor online learning materials and processes, and the data generated from the reviews drives decisions for continuous improvement [32]. Only Kumar et al. [4] applied standards aligned with evaluation, namely effectiveness of learning and satisfaction of learners and educators. The standards related to evaluation emphasise an interplay of process and outcome monitoring [21, 22]. However, there are gaps related to the longitudinal outcome of undergraduate e-learning programmes in the health professions.

Studies included in this review reported a wide array of standards and indicators of quality within undergraduate e-learning programmes in the health professions [18–21, 23]. However, none of the included articles described evaluation standards related to teaching and learning of clinical skills within undergraduate e-learning programmes in the health professions. The lack of standards for evaluating teaching and learning clinical skills is a significant gap within the literature from LMICs [25]. The generic standards used in undergraduate e-learning programmes in the health professions often miss the intricacies of clinical education [1]. There is a need for the development and integration of standards for evaluating undergraduate e-learning programmes in the health professions that include the evaluation of clinical education. These standards would support education institutions in determining the quality of their programmes and in professing that their graduates are as competent as graduates from face-to-face programmes.

Standards for undergraduate e-learning programmes in the health professions should be distinct from face-to-face programmes. The adoption of standards of face-to-face programmes in undergraduate e-learning is detrimental to the development of e-learning in LMICs [26]. The need for standards for evaluating undergraduate e-learning programmes in the health professions is vital in ensuring quality health professionals. Further research should focus on quality standards in teaching and learning clinical skills in e-learning programmes.

Conclusion

Learners graduating from e-learning and online programmes are currently viewed with scepticism, especially in terms of their clinical competence. However, the integration of digital technology in undergraduate education in the health professions has become inevitable. In most low-resource settings, face-to-face programmes remain superior and dominant as regulators and programme directors struggle to define and apply standards that comprehensively assess e-learning programmes’ quality. Delva et al. [1] noted that undergraduate e-learning programmes in the health professions must be comparable to face-to-face programmes. Still, caution should be taken regarding adopting face-to-face standards for online settings.

In this review, we synthesised standards that have been used for evaluating e-learning programmes in the health professions in LMICs to improve the quality of e-learning programmes. Only eight articles met the inclusion criteria–predominantly from the Middle East, North Africa and South-East Asia. Most of the standards described by the included articles aligned with popular models for evaluating e-learning programmes, with a few exceptions. A gap in clinical teaching and learning standards in undergraduate e-learning programmes in the health professions was evident in all the included articles.

Further research in this field should focus specifically on developing, expanding, and testing standards for evaluating the quality of e-learning that integrate teaching and learning of clinical skills. Such standards should allow evaluators to access clinical teaching sites, the educators within clinical teaching sites, and the nature and quality of clinical practice.

The contribution of this review is to identify themes used in evaluating the quality of undergraduate e-learning programmes in the health professions. The themes are curriculum planning, proficiency of the educator, learner proficiency and attitude, infrastructure for learning, support, and evaluation. When developing contextually appropriate interventions, it would be valuable to include teaching and learning of clinical skills.

Limitations

Limitations of the review are, the search string and inclusion criteria, which may have excluded some studies. Literature sourced from was limited to English-language publications within LMICs. Abstracts and full texts in foreign languages were excluded. Articles only from the University of the Free state were accessed and this might have limited the scope. The risk of bias in each of the articles was not assessed. We acknowledge not conducting a methodological quality review might affect the study results and the potential of overestimating or underestimating findings, can inadvertently affect the quality of the study. However, rigorous screening and consensus discussion mitigated the issue.

Supporting information

Acknowledgments

Prof. Ruth Albertyn: Critical reading

Jackie Viljoen: Language editing

Annamarie du Preez: Search string development