Objectives

This systematic review aims to summarize and evaluate the measurement properties of existing instruments that evaluate health professions students’ EBP competence including EBP knowledge, skills, and attitudes. This study will build on Landsverk et al. and describe in detail steps on how to adapt the COSMIN methodology for EBP instrument quality assessment using the updated version of the COSMIN guidelines [9,10]. Landsverk et al were the first to employ the COSMIN guidelines, which was a ten-step process at the time [9,10]. The updated COSMIN guidelines for systematic reviews have moved to an eight-step process.

Specifically, this systematic review will answer the following guiding research question: In health professional students (P), what instruments (I) demonstrate valid and reliable measurement properties for assessing EBP competence (O) including the subscales of knowledge, skills (including intention to practice, behaviours, competency, and patient benefits), and attitudes (including confidence, self-efficacy, beliefs and/or motivation)?

Criteria for considering studies for this review

1. 1) Formulate your research question

The systematic review was registered with PROSPERO under the identifier CRD42024564908 (Appendix A in S1 Appendix).

1. 2) Formulate eligibility criteria

A complete list of inclusion and exclusion criteria are listed in Appendix B in S2 Appendix, but we will include a general overview of the inclusion and exclusion criteria below.

Electronic searches

1. 3) Develop a literature search

The search strategy will aim to locate published studies on the measurement of EBP competence and its subscales. A three-step search strategy will be utilized in this review. First, an initial limited search of MEDLINE (EBSCO) and CINAHL (EBSCO) will be undertaken to identify articles on the topic. This search will identify several reviews that are like the current review objective but did not focus on students [9,1,11,13,14]. Inclusion and exclusion criteria from these reviews relevant to students will inform the criteria for this review. Additionally, we have identified key test articles that we have confirmed meet our search criteria and study EBP instruments intended for health professions’ students [15,16].

This review will search MEDLINE, CINAHL, SPORTDiscuss, and PsycINFO, all via EBSCO. SPORTDiscuss will locate any articles not caught by MEDLINE and CINAHL. PsycINFO will locate any articles that investigate instruments in social professions where EBP may also be utilised. ERIC will not be included due to the lack of relevance to our research question.

Data collection and analysis

1. 4) Conduct a literature search

Articles that meet our search criteria will be uploaded into Covidence software (Covidence, n.d.). All the reviewers will be blinded to the inclusion and exclusion decisions of the others and complete screening on their own time to reduce bias risk. Two reviewers (all authors) will screen the titles and abstracts of the articles for inclusion and exclusion criteria prior to moving onto the full-text screening phase. All conflicts will be resolved by a third reviewer (one of the authors) who will not be involved in title and abstract screening to reduce bias. For articles that are written in non-English languages, we will bring a collaborator to assist with the screening process. The articles that make it into the full-text screening phase will be uploaded to Covidence in their full-text format and will be screened for inclusion criteria by four reviewers (all authors). Votes from two reviewers will be necessary to include or exclude articles. Conflicts will be resolved via consensus from all four reviewers. We will use the following conflict resolution criteria and code it as such in Covidence:

1. No Full text article available
2. Not primary source journal article
3. Not relevant
4. EBP concepts missing or peripheral
5. Sample is not student health profession
6. Lacks psychometric property or instrument validation discussion
7. Knowledge evaluation only

Selection of studies

1. 5) Organize the studies from your articles

Mokkink et al. recommend an initial data extraction process to determine which instruments to include in the review or not (3, pg. 2931). Our review will follow this step, generally, but we have outlined a more detailed process guided by the Appendix D in S4 Appendix template. Two members of our research team (DS and ML) will extract the relevant data from each individual study to evaluate instruments that measure EBP competence (or its subscales) in a student population. The template has one tab each and we will outline instructions of how we will use it:

Data extraction and management

The data collected in this tab will include the EBP measurement instrument that is being studied, the authors, the EBP subscale construct(s) being studied (KSA), a study classification/purpose (i.e., to validate, translate, or originally develop), the country/language of the EBP instrument, if it is a uni-dimensional (i.e., EBP construct) or multi-dimensional (EBP subscales), the student population discipline, and the sample size.

We will pilot test this data extraction method with our authors (all four authors) using the template with four studies. Once the initial pilot testing is complete, two authors (DS and ML) will extract all data independently using their own template that is populated with the studies that have been included. If there is a discrepancy in data extraction between the authors, the two will meet to resolve the conflict. If they are unable to resolve the conflict, a third or fourth author (LL or BE) will break the tie.

1. 6) Evaluate the quality of the EBP competence measurement instrument

Mokkink et al. built on their previous work from 2018 whereby the quality evaluation was broken into six sub-steps (3, pg. 2933). Again, due to differences in the purpose from Mokkink et al. (PROMs evaluation for patients) and our review (student EBP competence instrument evaluation), we will expand on where we align and where we differ in more detail below [10]. We have created a template (Appendix E in S5 Appendix) that will be used for sub-steps 6.1, 6.2, and 6.3. In brief, Steps 6.1, 6.2, and 6.3 are related to extracting data, completing a psychometric measurement property rating of the evidence against the nine criteria for good measurement properties, and completing a risk of bias assessment on an individual study level. Steps 6.4, 6.5 and 6.6 are similar, but the psychometric measurement property rating and risk of bias assessment are done based on stratifying data by EBP measurement instrument. All sub-steps in Step 6 will be done by two independent raters (DS and ML). Where there is disagreement on the rating, discussion will ensure consensus. If they are unable to resolve the conflict, a third or fourth author (LL or BE) will break the tie.

1. Step 6.1 – Extract data on study populations, methods, and results

Assessment of risk of bias in included studies

1. Step 6.2 – Assess the methodological quality of the studies – is there risk of bias?

There is an overlap in the psychometric measurement property rating -aka from Mokkink et al. as “criteria for good measurement quality” – and the ROB assessment [10]. Therefore, we will collect data for Step 6.2 and 6.3 in the same template (Appendix E in S5 Appendix). Step 6.2 is a ROB assessment, and we will generally follow the same criteria set out by Mokkink et al., Appendix 1, the ROB Checklist 3.0 [10]. It consists of 10 boxes, each representing measurement properties of a EBP competence measurement instrument, plus the EBP instrument development item. The ROB rating scale consists of the following classifications: very good; adequate; doubtful; and inadequate. The variables that go into these classifications are outlined in detail in Mokkink et al.’s Appendix 1 and we will follow those classification criteria, with a few exceptions that we will specify below as well as in our template (Appendix E in S5 Appendix).

There is a summary tab in Appendix E in S5 Appendix that will capture the independent ratings for each individual study, broken down by boxes that are in their own separate tabs. Each rater (DS and ML) will read the paper and provide their independent assessment for each criterion outlined in detail in the specific box. If there is consensus, one author will translate the overall rating into the “methodological quality” assessment in the “Ind. Study ROB & Psych. Rating” tab. If they are unable to resolve the conflict, a third or fourth author (LL or BE) will break the tie.

1. Step 6.3 Rate the study results against criteria for good measurement properties

We will use the same template (Appendix E in S5 Appendix) as was used in Step 6.2. Just like Step 6.2, there is a summary tab (entitled “Ind. Study ROB & Psych. Rating”) where psychometric property measurement rating data will be entered based on the mean assessment of each individual box tab. The psychometric property measurement rating is the same as the “criteria for good measurement properties” in Mokkink et al.’s Table 1, with a few minor exceptions (3, pg. 2935). The criteria that guide the rating are at the bottom part of the tabbed spreadsheet. In other words, the ROB assessment criteria are at the top of Box 2, and the psychometric measurement rating scale is at the bottom. This same process will be followed for Boxes 2–10 inclusively. It is important to remember that Box 1 is only rated for ROB assessment. The rating criteria for each of the nine psychometric measurement properties are otherwise the same as what Mokkink et al. have recommended (3, Table 1). The rating scale that Mokkink et al. employed will be converted to a numerical value so that summarizing more than one article is easier to consolidate and understand. Therefore, the + = sufficient will be assigned a value of 2 points, the - = insufficient will be assigned a value of zero, and the? = indeterminate will be assigned a value of 0 or 1 depending on the rater’s judgement.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Conversion of numerical values from overall methodological quality to standard GRADE definitions.

https://doi.org/10.1371/journal.pone.0347078.t001

Measures of treatment effects, units of analysis, data synthesis, assessment of reporting biases

1. Step 6.4 Summarize the results

The previous steps in Step 6 have been related to extracting data, the ROB assessment and rating the study against good measurement property criteria for individual studies. There may be more than one study on a specific EBP measurement instrument. In these cases, we will combine data in a new template (Appendix F in S6 Appendix) and summarize the results. We will simply copy and paste data from the summary tab (entitled “Ind. Study ROB & Psych. Rating”) in Appendix E in S5 Appendix, and we will stratify the data by EBP measurement instrument so that we can summarize the results for each EBP measurement instrument. Therefore, we will sort by the EBP measurement instrument in Appendix E in S5 Appendix, then cut and paste into Appendix F in S6 Appendix. There is also a tab for single study EBP instruments in Appendix F in S6 Appendix and data from Appendix E in S5 Appendix can also be transferred. Clearly, if there is only one study, the mean values obtained from Appendix E in S5 Appendix will be the value that is used for Step 7 below.

1. Step 6.5 Rate the summarized results against criteria for good measurement properties

The same process outlined in Step 6.4 will move the “psychometric measurement property rating” over from Appendix E to Appendix F in S5 and S6 Appendix after the data has been sorted by EBP instrument. Once again, where appropriate, there may be more than one study on a specific EBP measurement instrument. In these cases, we will combine data in a new template stratified by EBP measurement instrument. Single-study EBP instruments will be captured on a different tab, and mean values will be taken from that study only. The EBP instruments that have more than one study will use the mean score across all studies. The overall mean score will be calculated on all nine psychometric measurement properties, and this score will feed into Step 7.

1. Step 6.6 Grading the quality of the evidence – summary ROB assessment

The same process outlined in Step 6.4 will move the “methodological quality assessment” over from Appendix E to Appendix F S5 and S6 Appendix and the data will be sorted by EBP instrument. Once again, where appropriate, there may be more than one study on a specific EBP measurement instrument. In these cases, we will combine data in a new template stratified by EBP measurement instrument. Single-study EBP instruments will be captured on a different tab, and mean values will be taken from that study only. The EBP instruments that have more than one study will use the mean score across all studies. Mokkink et al. recommend the GRADE approach in this step to “grade the certainty of evidence” (3, pg. 2936). However, they only explain how to GRADE the first two boxes (EBP Instrument Development and Content Validity). Our approach will be very different because we will attempt to factor in all ten boxes in our grading of the certainty of evidence for EBP instruments. We will use both the individual box mean scores, and the mean scores across all 10 boxes because we value all boxes in the final assessment. The overall mean score will be calculated on all 10 boxes, and this score will feed into Step 7. We agree with Mokkink et al.’s stance that without strong values in both Box 1 (EBP Instrument Development) and Box 2 (Content Validity), the evidence from the rest of the boxes (i.e., Box 3–10) is significantly less important and will therefore result in a lower grade in Step 7.

1. 7) Drawing conclusions

We will use the original GRADE categories as a basis to draw conclusions on the summary of the evidence across 10 boxes and perhaps multiple studies that have contributed to the amassed evidence. We will link the numerical values from the previous sub-steps in Step 6 to definitions offered by Guyatt et al. in Table 1 [18]. It should be noted that the steps in this component are a significant departure from the guidelines from Mokkink et al. [10]. (Mokkink et al. mostly focused their GRADE guidelines on Box 1 (EBP instrument development) and Box 2 (content validity). Our guidelines attempt to consider all Boxes (i.e.,1–10) and their ROB assessment, albeit in numerical values, that can be summarized based on the values from each box. Additionally, the process that Mokkink et al. outlined requires a lot of time extracting psychometric data and rating of those psychometric measurement properties. However, Mokkink et al. do not use those data in Step 7, but we have decided it is important to factor that into this final step [10]. As a result, we will use the guidelines below to assign a methodological quality standard or label based on the definitions, explanations, and guidelines in Table 1.

1. 8) Comprehensive reporting

This protocol has been developed based on the updated COSMIN guideline for systematic reviews of PROMs including use of the COSMIN Risk of Bias checklist and criteria for good measurement properties [10]. The completed review will be reported according to the PRISMA-COSMIN for outcome measurement instruments (OMIs) 2024 reporting guideline [19], in line with PRISMA 2020 extensions for systematic reviews of outcome measurement instruments. We will aim to publish the full systematic review in PLOS One.