https://orcid.org/0009-0001-6312-4006
Figures
Abstract
Introduction
The shift to digital clinical placements for physiotherapy education due to COVID-19 prompts a need for evaluation of current evidence. Existing studies highlight benefits of digital technology in clinical placements, but lack of a systematic review focused on pre-registration physiotherapy students is a key gap. This systematic review will address this gap by synthesizing the evidence for digital clinical placements for pre-registration physiotherapy students internationally.
Methods and analysis
This systematic review is designed using the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) statement and Cochrane Handbook – it is registered on PROSPERO (CRD42024571696). Search terms will be adapted to each database, including EMBASE, MEDLINE, PROSPERO, ERIC, and CINAHL. Key journals, forward citation tracking, references of included studies, and professional organization websites will also be searched. The search will include studies published since database inception to 31/05/24. There will be no limit to study design or language. Studies that report on perceptions, experiences, barriers, facilitators, learning outcomes, and modes of assessment of digital clinical placements for pre-registration physiotherapy students will be included. Meta-aggregation will be used to synthesize themes from findings which enables the generation of themes without the need to re-interpret data and the loss of study specific context.
Ethics and dissemination
Ethics approval is not required. The results of this study will be written up for publication in relevant peer-reviewed scientific journals and contribute to a developing area of research. Results will also be presented at national or international conferences, events for the physiotherapy profession, or education events.
Citation: McConnell J, Rushton A, Noblet T, Pacey V, Mistry J, Lai J, et al. (2025) Perceptions, experiences, barriers, facilitators, learning outcomes, and modes of assessment of digital clinical placements for pre-registration physiotherapy students internationally: a systematic review protocol. PLoS ONE 20(2): e0319024. https://doi.org/10.1371/journal.pone.0319024
Editor: Catherine Doody, University College Dublin - National University of Ireland: University College Dublin, IRELAND
Received: August 27, 2024; Accepted: January 24, 2025; Published: February 20, 2025
Copyright: © 2025 McConnell et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
To become a physiotherapist, pre-registration physiotherapy students need to gain practical experience in clinical settings working with patients under the supervision of experienced professionals, commonly termed clinical educators, clinical instructors, or preceptors. Traditionally, this practical experience is gained through in-person clinical placements which enables hands-on training and exposure to real-world situations. However, with the COVID-19 pandemic and advancements in digital technology, the traditional approach to clinical placements was disrupted and digital clinical placements have emerged as a viable and appropriate alternative [1–4]. As digital clinical placements increasingly become more prevalent in the education of pre-registration physiotherapy students, it is necessary for physiotherapy educational programs to develop physiotherapists with understanding, skills, and knowledge in digital technologies across practice settings as called for by World Physiotherapy [5]. This review will explore the current evidence on digital clinical placements for pre-registration physiotherapy students. But first, it is necessary to operationalize the term “digital clinical placements” to ensure clarity in this review because there are various terms and concepts used internationally and throughout the literature.
Defining digital clinical placements
Several terms are used in literature to describe the concept of digital alternatives as a different way to offer clinical placements or to supplement them. For example, Sam et al. [6] and Houghton et al. [7] describe “digital clinical placements” as a scenario where in-person placements for medical students are offered with asynchronous or synchronous clinical education materials and resources (e.g., problem-based learning, online case studies, educational videos, Q&A’s, live webinars, and self-directed tutorials) delivered through a digital format. Quail et al. [8] describe the supplementation of in-person placements with simulation through Virtual Learning Environments (VLEs) and Standardised Patients (SPs) as “virtual clinical placements.” VLEs are computer generated applications created to achieve a specific learning outcome and SPs are human actors trained to respond in specific ways to students participating in a simulation (e.g., following a script or expressing specific symptoms) [8]. Robinson et al. [9] describes Simulated Learning Environments (SLEs) as both “virtual and digital patient placement” – SLEs use SPs and problem-based learning. Twogood et al. [1] and Vrzic et al. [3] also use the term “virtual clinical placement” to describe physiotherapy students participating in live patient consultations through digital communication (i.e., telehealth). Heneghan et al. [2] use the term “telehealth e-mentoring” to describes post-graduate physiotherapists using telehealth alongside their mentors to deliver consultation and rehabilitation. There is incongruent use of terms in the literature, as such, operationalizing a term will distinguish these varied descriptions of similar but distinct concepts to facilitate straightforward understanding and clarity for this review. A shared understanding enhances the coherence of academic and professional discourse and aids in the development of shared knowledge bases. As Podsakoff et al. [10] stated, “… concepts serve as the fundamental building blocks of theory, allowing us to organize complex phenomena with a common language that, when done well, facilitates communication between researchers. (p. 168)” For this paper, “digital clinical placements” will be described as healthcare delivered a/synchronously by pre-registered healthcare professionals (HCPs) under the supervision of registered HCPs to human patients entirely or partially via information and communication technology. This definition allows for clarity in defining inclusion criteria and the search strategy of this review.
The state of evidence
A preliminary search using Google Scholar, Cochrane Library, and Physiotherapy Evidence Database (PEDro) identified two reviews on digital clinical placements for pre-registration physiotherapy students with limited physiotherapy perspective and methodological rigour. First, Bridgman et al. [11] conducted a synthesis of published literature on the educational perspectives of allied health students on clinical placements using telehealth. Their findings suggest that telehealth placements offer financial, accessibility, and educational support benefits, however, considerable work is needed to address challenges in the overall experience [11]. Second, Romli et al. [12] aimed to determine if healthcare students acquired sufficient competence through alternative clinical placements such as telehealth, simulation, and case-based learning. Their systematic review suggests alternative clinical placements are either more effective or just as effective as traditional placements [12]. Both Bridgman et al.’s and Romli et al.’s reviews are limited in their capture of the physiotherapist perspective. A single paper with physiotherapy perspective is included in each review, 1 of 3 in the Bridgman article and 1 of 24 in the Romli article, this raises concerns for the generalizability of their findings towards physiotherapy.
Additionally, findings from Bridgman et al.’s and Romli et al.’s reviews should be accepted cautiously because of appraisal ratings of critically low and low [13], respectively. Both reviews used search strategies that included data bases only and raise concerns about the rigour of their methods. Further methods of searching such as forward citation tracking, searching reference lists, or manually searching specific journals and professional organizations websites would improve the design by increasing the likelihood of capturing all relevant data [14]. Romli et al.’s work remains important as the attainment and assessment of competency is an important aspect of education. Clinical competency is a precursory requirement of entry-to-practice for pre-registration physiotherapy students [15]. Therefore, evidence surrounding attainment and assessment of competency is valuable for informing physiotherapy educational programs. Both reviews underscore the need for a more rigorous review that is specific to physiotherapy to produce relevant and applicable findings. Therefore, a high-quality systematic review is required to synthesise the current evidence in the evolving field of digital clinical placements for pre-registration physiotherapy students.
Within this evolving field several primary studies [1,3,16–19] highlight the opportunities and challenges associated with digital clinical placements for pre-registration physiotherapy students. Students reported initial skepticism about the efficacy of telehealth in meeting learning objectives, particularly for manual therapies and hands-on skills [16]. Students also reported positive perceptions regarding telehealth. For example, Ross et al. [16] and Davies et al. [19] highlighted that students found telehealth beneficial for addressing accessibility and for follow-up consultations. A majority of students also advocated for a blended model combining in-person and telehealth [19]. While these studies provide valuable insight into the perceptions of physiotherapy students, generalizability is limited because the findings of each study focus on perceptions of students from only one university each – University of Queensland in Brisbane, Queensland, Australia and Macquarie University in Sydney, New South Wales, Australia.
As for clinical educators, some in Brisbane, Queensland, Australia believe digital clinical placements provided a unique opportunity for students to develop clinical reasoning, communication, and planning skills [17]. However, these clinical educators also believed that hands-on skills were less developed, and client outcomes were varied depending on client engagement and technological proficiency [17]. In regards to the clinical educator context of digital clinical placements, the same clinical educators from Brisbane reported challenges in efficient supervision, technical issues, and adapting educational models for digital [17]. These clinical educators also noted that telehealth allowed flexibility in placements, including remote supervision, though it required careful client selection and session preparation [17]. Another challenge noted by a clinical educator at a pain management clinic in Kingston, Ontario, Canada was navigating supervision without clear guidelines for digital healthcare [3]. One group in Australia and New Zealand aimed to address this same concern. This group created a guide on how to interpret the Assessment of Physiotherapy Practice (APP), the assessment tool used on clinical placements for physiotherapy students, to better evaluate competencies in the telehealth context [18]. A strength of this expert group was that it included the perspectives of several clinical educator leaders representing several education institutions and their respective clinical educators. This guide can be used to inform clinical educators assessing student skills within a digital clinical placement [18], however the applicability outside of the Australia and New Zealand context should be carefully assessed.
For the institutional context, there appears to be agreement on opportunities and challenges with the added perspective of sustainability and scalability of digital clinical placements. Twogood et al. [1] reported digital clinical placements had strengths in accessibility for students, particularly those with health risks and living abroad, and in opportunities to develop digital care skills essential to the evolving healthcare landscape. Twogood et al. [1] also reported challenges in technical issues and limited opportunities for hands-on clinical practice. Additionally, Twogood et al. [1] reported administrative challenges in scheduling and communication. Despite these challenges, digital clinical placements increased clinical placement capacity by over 400% compared to pre-COVID periods [1]. Twogood et al. [1] note the opportunity for scalability, but recognize the sustainability of this model may require adaptation for different contexts. Digital clinical placements are a vital innovation in physiotherapy education. Despite challenges, they offer a viable opportunity to expand placement options and equip students with essential digital healthcare skills. This systematic review will synthesize current evidence to provide insights that guide future integration of digital clinical placements into physiotherapy education.
Aim
To investigate the perceptions, experiences, barriers, facilitators, learning outcomes, and modes of assessment of digital clinical placements for pre-registration physiotherapy students internationally
Objectives
- To identify and synthesize perceptions and experiences associated with digital clinical placements for pre-registration physiotherapy students from the perspective of physiotherapy educational programs, pre-registration physiotherapy students, and physiotherapy clinical educators.
- To identify and synthesize barriers and facilitators associated with digital clinical placements for pre-registration physiotherapy students.
- To identify and synthesize learning outcomes and modes of assessment currently being used for digital clinical placements for pre-registration physiotherapy students.
Methods
Positionality
Researcher ontological, epistemological, and methodological stances shape the design, conduct, and interpretation of studies. Our ontological perspective lies between realism and relativism, acknowledging that while an external reality may exist, our understanding of it is inevitably influenced by individual experiences and social contexts. This position allows us to recognize the shared realities within digital clinical placements for pre-registration physiotherapy students while also appreciating the diversity of individual and contextual experiences. Epistemologically, our stance aligns with interpretivism, with elements of positivism and constructivism. This means we will prioritize understanding the subjective meanings and experiences of individuals engaged in digital clinical placements, while also acknowledging the value of systematically synthesizing findings to identify patterns and broader truths. We embrace a constructivist approach in appreciating that knowledge can be co-constructed between researchers and participants, yet we remain open to positivist ideals when aiming for rigor and reproducibility in the synthesis of evidence. Methodologically, we have chosen a qualitative systematic review with meta-aggregation for data synthesis. This approach aligns with our philosophical stance by enabling us to systematically identify, appraise, and synthesize qualitative evidence. Meta-aggregation offers a structured way to collate findings without losing the original studies interpretation, ensuring the synthesis reflects both the shared and unique aspects of the phenomenon under study. We are aware that our professional background and experiences influence how we approach and interpret the data. As professors, PhD students, and physiotherapists researching digital clinical placements for pre-registration physiotherapy students, we bring insights from both academic and practical sides of the field. We strive to remain reflexive throughout the research process, continuously interrogating our biases and assumptions to ensure they do not unduly influence the synthesis of the findings. Our aim is to conduct research that is rigorous, meaningful, and grounded in the lived realities of those it seeks to represent.
Design
This systematic review is designed using a combination of the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement [20] and Cochrane Handbook [14]. It is registered with the International Prospective Register of Systematic Reviews (PROSPERO CRD42024571696).
Eligibility criteria
Eligibility criteria were created using a modified PICOS framework to ensure sensitivity during the search process [21]. Inclusion criteria (Table 1) were created to ensure the search is as broad as possible. From the framework, Comparator was not included as it is not applicable to the research aim and objectives.
Information sources
Information sources will be searched from their inception to the 22nd of January 2025. The sources of information that will be searched include:
Databases.
EMBASE, MEDLINE, PROSPERO, Educational Resource Information Centre (ERIC), and Cumulative Index to Nursing and Allied Health (CINAHL) databases will be included in the search strategy.
Specific journals.
Key journals will be manually searched and include Physiotherapy, Physiotherapy Canada, Physical Therapy & Rehabilitation, Journal of Physiotherapy, and Physiotherapy Theory and Practice.
Professional organizations.
Relevant articles available on professional organizations’ (e.g., physiotherapy associations and special interests’ groups) websites will be searched manually. This will be achieved by entering relevant phrases adapted from the search strategy into Google and then manually searching the first 100 results [22].
Forward citation tracking.
Forward citation tracking will be used as a method to identify newer publications that have cited a particular article or other scholarly work. Included studies will be entered in Google Scholar to conduct forward citation tracking. While similar to Web of Science and Scopus, Google Scholar will be used as it can capture unique citations including non-English sources [23].
Search strategy
The search strategy was created in collaboration with a research librarian at University of Western Ontario Libraries. Three concepts were chosen to ensure the search strategy is as broad as possible. A complete search strategy is demonstrated in Table 2. Database specific subject headings (i.e., MeSH terms used in the MEDLINE OVID platform) will be combined with concepts where possible. The MEDLINE OVID platform search strategy is presented in S1 Appendix. The search strategy will be adapted for each information source.
Study records
Data management.
Results from all searches will be imported to, stored, and managed on Covidence [24]. The reference, abstract, and full text file will be stored on Covidence for screening. Duplicates will be removed prior to commencement of the selection process.
Selection process.
Two researchers will independently and simultaneously review titles and abstracts of the identified articles using Covidence’s structured workflow [25]. The predetermined eligibility criteria (Table 1) will be used to separate the articles into one of three categories: eligible, not eligible, and potentially eligible. Articles that are obviously not related to the objective based on their title and abstract will be excluded. If an article is unable to be excluded based on title and abstract, then it will be categorised as potentially eligible. After screening abstracts and titles, full-text copies of potentially relevant articles will be obtained, and eligibility ascertained using the same process. If there is disagreement between researchers regarding selection of studies, then a third researcher will be consulted. The selection process will be presented visually per PRISMA 2020 Guidelines [20]. The Kappa coefficient will be used to evaluate the inter-rater reliability between reviewers [26].
Data collection process.
Two researchers will independently extract qualitative data using a study-specific data extraction grid within Microsoft Excel. If data is unavailable in the full-text of included studies, then the original authors will be contacted to ask for a copy of the original study’s data. In the event of disagreement or lack of consensus between two researchers, then a third researcher will be consulted to make the decision for inclusion.
Data items
The data extraction grid will include key data items: citation details (title of study, authors, year of publication, and country), study design, population, purpose/phenomenon of interest, aims and objectives of the study, philosophical positioning, methodology/methods, and key findings. Key findings will be identified by the themes presented in the results of included studies. For example, this may include the subsection titles in the results section or may include the authors explicitly stating the themes of their study. These themes will then be used for meta-aggregation, a data synthesis method that does not use interpretive analysis of extracted data from included studies, instead it enables a summary of the original authors’ findings across included studies to determine common or competing themes [27]. The ‘Data synthesis’ section explains further how meta-aggregation avoids re-interpretation.
Quality assessment
Two researchers will independently use the Quality Assessment with Diverse Studies (QuADS) tool [28] to assess methodological and reporting quality of included studies. The QuADS is a tool for the assessment of quantitative, qualitative, and mixed-methods studies that has an inter-rater reliability with a Kappa coefficient of k = 0.66 [28]. The QuADS and will provide a structured approach to quality assessment that is user friendly for the authors of this study. The QuADS scores will be reported in a table. If there is a disagreement between the two researchers, then a third researcher will be consulted. The quality assessment ensures transparency and confidence in the quality of the included studies’ findings.
Data synthesis
Meta-aggregation will be used to synthesize extracted data. Meta-aggregation is a three-step synthesis model that avoids re-interpretation of data from included studies and aims to accurately and reliably present the findings of the authors from the included studies [27]. Avoiding re-interpretation ensures the results remain with the specific contextual factors that informed the original authors’ interpretation of the data. The first stage is a detailed extraction of findings from the results section of included studies and will be presented in a table. The second stage is categorization of the findings by identifying a common theme between at least two findings per category [27]. Categories will be based on similarity of concepts between the findings and developed using a consensus process of at least two researchers. The third stage involves developing a synthesized finding of at least two categories [27] that will aim to capture the collective meaning of the categories. Two researchers will develop the synthesized finding through a consensus process with a third researcher having final input. The findings will be illustrated as shown in the example in Fig 1.
Confidence in cumulative evidence
GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) will be used to assess quality and strength of evidence found in this study. GRADE-CERQual is a structured method for determining the confidence in evidence used for qualitative data synthesis and usefulness of the findings [29]. GRADE-CERQual assesses four components in each synthesis finding: methodological limitations, coherence, adequacy of data, and relevance [29]. Methodological limitations will be assessed using the QuADS tool as described in the Quality assessment section [30]. Coherence, adequacy of data, and relevance will be assessed with guidance from key papers outlining how to apply GRADE-CERQual to qualitative evidence synthesis [29–34]. Coherence is how clear and compelling the fit is between primary research data and a synthesized review finding. Coherence will be assessed using three steps: 1) collect and consider the necessary information related to coherence (i.e., clear patterns across primary research data that supports the review finding), 2) assess the body of data that contributes to each review finding and decide whether there are concerns about coherence, and 3) make a judgement about the seriousness of the concerns and justify this judgement [31]. Adequacy of data is a measurement of data richness and data quantity supporting a review finding. Adequacy of data will be assessed using three steps; 1) collect and consider the necessary information related to adequacy (i.e., the amount and depth of data that relates to the objectives of this review), 2) assess the body of data that contributes to each review finding and decide whether there are concerns about adequacy, and 3) make a judgement about the seriousness of the concerns and justify this judgment [32].
Relevance will be assessed in five steps; 1) clarify the review question and context, 2) decide on the appropriateness and implication of the sampling strategy, 3) gather information about relevance in the included studies, 4) assess the body of data that contributes to each review finding and decide whether there are concerns about relevance, 5) make a judgement about the seriousness of the concerns and justify this judgement [34]. Table 3 is an example of the table that will be created to facilitate the review findings for each objective. The assessments will be assigned one of four rankings: no or very minor concerns, minor concerns, moderate concerns, or serious concerns to each finding of this study [33]. A no or very minor concerns ranking would indicate that the review finding is likely be an appropriate representation of the phenomenon of interest and a serious concerns ranking would indicate that the review finding is very unlikely to be an appropriate representation of the phenomenon of interest [33]. An overall confidence in the evidence rating will be assigned as high, moderate, low, or very low. A high ranking would indicate that more research is unlikely to affect the confidence of the review findings and a very low ranking would indicate there are concerns that are very likely to reduce confidence of the review findings [33].
Publication and dissemination of results
The results of this study will be published in an international peer-reviewed scientific journal and contribute to a developing area of research. Results will also be presented at national or international peer-reviewed conferences, events for the physiotherapy profession, and education events.
Implications of this study
The authors of this review anticipate that the findings will provide insight into the perceptions and experiences of pre-registration physiotherapy students, clinical instructors, and physiotherapy educational faculty involved in digital clinical placements. This review will provide insight to the barriers and facilitators to the success of digital clinical placements. This review will provide insights regarding current learning outcomes and modes of assessment being used for digital clinical placements for pre-registration physiotherapy students. The authors expect the findings will be specific and relevant to educational institutions and research groups interested in digital clinical placements for pre-registration physiotherapy students, as well as clinicians and clinical sites supporting students on clinical placements. This review will identify gaps in the evidence and suggest future opportunities for research. The findings may underpin curriculum decision-making for educational programs conducting digital clinical placements.
References
- 1. Twogood R, Hares E, Wyatt M, Cuff A. Rapid implementation and improvement of a virtual student placement model in response to the COVID-19 pandemic. BMJ Open Quality. 2020;9(4):e001107. pmid:33148602
- 2. Heneghan NR, Jagodzinska J, Tyros I, Johnson W, Nazareth M, Yeung E, et al. Telehealth e-mentoring in postgraduate Musculoskeletal physiotherapy education: a mixed methods case study. Musculoskeletal Sci Practice. 2021;56:102448. pmid:34416558
- 3. Vrzic A, Vader K, Law M, Miller J. Implementing a virtual care clinical placement for a physiotherapy student in a hospital-based chronic pain programme. Physiotherapy Canada. 2022;74(2):210–3.
- 4. Tuckson RV, Edmunds M, Hodgkins ML. Telehealth. New England J Med. 2017;377(16):1585–92.
- 5.
Physiotherapy W. Report of the World Physiotherapy/INPTRA digital physical therapy practice task force. UK: World Physiotherapy London; 2019.
- 6. Sam AH, Millar KR, Lupton MGF. Digital clinical placement for medical students in response to COVID-19. Acad Med. 2020;95(8):1126. pmid:32744817
- 7. Houghton N, Williams L, Baptista A, Thakerar V, Dharmarajah A. Digital clinical placements: Student perspectives and preparedness for placements. Clinical Teacher. 2023;20(1):e13558. pmid:36599293
- 8. Quail M, Brundage SB, Spitalnick J, Allen PJ, Beilby J. Student self-reported communication skills, knowledge and confidence across standardised patient, virtual and traditional clinical learning environments. BMC Med Educ. 2016;16(1):73. pmid:26919838
- 9. Robinson KE, Allen PJ, Quail M, Beilby J. Virtual patient clinical placements improve student communication competence. Inter Learn Environ. 2020;28(6):795–805.
- 10. Podsakoff PM, MacKenzie SB, Podsakoff NP. Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organizational Res Method. 2016;19(2):159–203.
- 11. Bridgman K, Erickson S, Furlong L, Bird A. Allied health student engagement in telehealth clinical placements: a rapid review. Int J Allied Health Sci Practice. 2022;20(2):1.
- 12. Romli MH, Wan Yunus F, Adam SK, Salihan S. How Are alternative clinical placements performed compared to traditional clinical placements during the COVID-19 pandemic? Sought through a systematic review and meta-analysis. Med Sci Edu. 2024;34(4):927–47.
- 13. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. pmid:28935701
- 14. Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, et al. Searching for and selecting studies. Cochrane Handbook Systematic ReviewIntervent. 2019:67–107.
- 15. Mori B, Daly A, Norman KE, Wojkowski S. The development of the Canadian physiotherapy assessment of clinical performance (ACP) 2.0–alignment with the 2017 competency profile. Physiotherapy Canada. 2022;76(1):111–20.
- 16. Ross MH, Whitehead A, Jeffery L, Hill A, Hartley N, Russell T. Allied health students’ experience of a rapid transition to telerehabilitation clinical placements as a result of COVID-19. Australian J Clinical Educ. 2021;10:1.
- 17. Ross MH, Whitehead A, Jeffery L, Hartley N, Russell T. Supervising students during a global pandemic: clinical educators’ perceptions of a student-led telerehabilitation service during COVID-19. Internat J Telerehabilitation. 2022;14(1):e6464. pmid:35734382
- 18. Nahon I, Jeffery L, Peirsis C, Dunwoodie R, Corrigan R, Francis-Cracknell A. Responding to emerging needs: Development of adapted performance indicators for physiotherapy student assessment in telehealth. Australian J Clinical Educ. 2021;9:17.
- 19. Davies L, Lawford BJ, Chan C. Physiotherapy students’ attitudes toward the use of telehealth in clinical practice: a cross-sectional survey. Health Science Report. 2024;7(4):e2067. pmid:38650730
- 20. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Review. 2021;10(1):89. pmid:33781348
- 21. Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14(1):1–10.
- 22. Briscoe S, Abbott R, Lawal H, Shaw L, Coon JT. Feasibility and desirability of screening search results from Google Search exhaustively for systematic reviews: A cross‐case analysis. Res Synthesis Method. 2023;14(3):427–37. pmid:36633509
- 23. Martín-Martín A, Orduna-Malea E, Thelwall M, López-Cózar ED. Google Scholar, web of science, and scopus: a systematic comparison of citations in 252 subject categories. J Informetrics. 2018;12(4):1160–77.
- 24.
Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation.
- 25. Kellermeyer L, Harnke B, Knight S. Covidence and rayyan. J Med Library Association JMLA. 2018;106(4):580.
- 26. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. pmid:23092060
- 27. Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evidence-Based Healthcare. 2015;13(3):179–87. pmid:26262565
- 28. Harrison R, Jones B, Gardner P, Lawton R. Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies. BMC Health Serv Res. 2021;21(1):144. pmid:33588842
- 29.
Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Springer; 2018. p. 1–10.
- 30. Munthe-Kaas H, Bohren MA, Glenton C, Lewin S, Noyes J, Tunçalp Ö, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Impl Sci. 2018;13(1):9.
- 31. Colvin CJ, Garside R, Wainwright M, Munthe-Kaas H, Glenton C, Bohren MA, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence. Impl Sci. 2018;13(1):13.
- 32. Glenton C, Carlsen B, Lewin S, Munthe-Kaas H, Colvin CJ, Tunçalp Ö, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data. Impl Sci. 2018;13(1):14.
- 33. Lewin S, Bohren M, Rashidian A, Munthe-Kaas H, Glenton C, Colvin CJ, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Impl Sci. 2018;13(1):10.
- 34. Noyes J, Booth A, Lewin S, Carlsen B, Glenton C, Colvin CJ, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data. Impl Sci. 2018;13(1):4.