Systematic review.

Existing feedback literacy instruments in medical education and other health professionals will be reviewed to comprehensively examine the current definitions and metrics used to assess feedback literacy. This review will use guidelines specified in the medical education context [32]. Instruments with similar constructs that measure any features of feedback literacy [10] will be included to provide a deeper understanding of such literacy.

Initially, a pilot search will be performed to select keywords that return relevant research articles. Based on the PICO (Population-Intervention-Comparison-Outcome) format, terms and synonyms related to healthcare education students, feedback literacy, psychometric properties, and instruments will be employed to perform a systematic search of six electronic databases (i.e., Scopus, Medline, Web of Science, CINAHL Complete, Education Research Complete, and Psychology and Behavioral Sciences Collection). The full search string for each database is provided in S1 Appendix. The references and citations of these articles will be manually screened for additional potential articles [32, 33]. Periodic searches will be conducted to identify newly published studies and ensure that the review accounts for all recent publications.

Identified studies will be evaluated using the following set of eligibility criteria:

1. included health professions education of any level (undergraduate and/or postgraduate)
2. instrument measuring student feedback literacy or any of its features (regardless of the feedback provider, e.g. peers, patients, teachers) including adaptations, revalidation in a different context and replication studies
3. focusing on students’ role in the feedback process i.e., behaviours and/or attitudes of students towards the feedback process
4. reporting any psychometric properties listed in the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) risk of bias checklist [34]
5. original study published in peer-reviewed journals
6. published in any year

Studies were further excluded if:

1. the instrument does not emphasize feedback literacy or its specific features (such as combination with other unrelated constructs or only mentions feedback literacy briefly)
2. developed instruments that do not focus on the students’ roles in the feedback process, for example, feedback providers opinions on the feedback process
3. non-English publications
4. non-primary research articles, such as reviews, opinion pieces, editorials, or perspectives
5. gray literature (e.g., theses, dissertations, unpublished reports, conference presentations)
6. no available full text

After removing duplicates, two authors will independently screen the retrieved records based on their titles, abstracts, and keywords. Subsequently, the full texts will be screened. Any differing opinions will be resolved through discussions between the authors. Consequently, relevant data, such as the description, utility, constructs, and psychometrics of each included instrument, will be extracted. The methodological quality of the studies will be independently appraised by two authors according to the COSMIN Risk of Bias Checklist [34]. This checklist was selected because of its comprehensive and robust methodology applied in its development. It covers a wide range of psychometric properties thoroughly, including content validity, structural validity, hypotheses testing, cross-cultural validity, internal consistency and test-retest reliability. Findings for the systematic review will be synthesized narratively because it has been successfully used in previous systematic psychometric reviews [35, 36], and it allows analysis between the various conceptualizations of feedback literacy. For example, narrative synthesis allows flexibility to compare between the concepts of Carless and Boud’s feedback literacy [10] and the Bing-You et al. FEEDME Culture [37]. Thus, narrative synthesis will be used for its inclusiveness, unlike other synthesis approach such as meta-analysis, which restricts a synthesis to one single concept used in all studies included. The narrative synthesis will be guided by Popay et al.’s [38] recommendations. Cohen’s kappa will be calculated to establish inter-rater reliability during the full-text screening and quality appraisal stages [39, 40]. To ensure ease of communication of findings by complying with a standard, this systematic review will be reported according to the PRISMA guidelines [41]. The outcomes will then guide in developing interview protocols for medical students, educators, and patients.

Thematic analysis.

Interview transcripts will be thematically analyzed using NVivo as this approach is flexible and accessible [49]. As an example, medical students, educators, and patients will be asked the prompt, “Please describe how medical students should receive feedback.” Accordingly, triangulating the different experiences between three types of interviewees (source triangulation) may provide a comprehensive description of medical student feedback literacy. Codes will be developed based on the collected data [31, 49] by one researcher. This inductive approach allows for a novel and comprehensive description of medical students’ feedback literacy that is not confined to existing concepts. This researcher will read the interview transcripts multiple times to become familiar with them. Recurring phrases or keywords will be highlighted to identify possible patterns in the data and cross-check those that appear among medical students, educators, and patients. Subsequently, codes will be generated to capture notable patterns, which will be checked to reduce overlap and redundancy, and subsequently grouped into potential themes using an iterative approach to refine each theme. These themes will be named and clearly defined. Next, these will be examined across the entire dataset to ensure that the conclusions drawn are firmly rooted in the data. The themes will be organized and presented as a thematic map. Additionally, at least two researchers (other than the coder) will review the themes to enhance the credibility of the findings [50]. Finally, the themes will be analyzed to generate measurable domains of medical students’ feedback literacy.

Item generation.

The systematic review and thematic analysis of the interviews will be used to produce a list of items. To ensure clarity and enhance the content validity, these will be based on the recommendations from previous research [42, 51, 52]. The words used to develop the items will be guided by the interviewees as much as possible to avoid overinterpretation and to ensure that the items represent the intact ideas of medical students, educators, and patients. Reverse-worded items will be included to minimize response sets [42].

For instance, thematic analysis may ultimately suggest modifying the general item “I have realized that feedback from other people can enhance my self-reflection on how I can systematically improve my learning” [53] to suit the medical education context to “I realize that feedback from supervisors enables me to reflect on how I can improve my clinical skills.”

Thematic analysis may also suggest using a new item to suit the local culture, such as, “I accept criticism on my weaknesses that considers the ways (e.g., religion) I was brought up” or “I accept criticism on my weaknesses that is not delivered in front of my peers.”

Participants will be asked to rate their responses using a 7-point semantic differential scale ranging from 1 (do not correspond at all) to 7 (correspond exactly). This 7-point semantic differential scale is chosen to improve the reliability of the instrument (as opposed to a 5 or 6-point scale) [54] while maintaining conciseness (as opposed to an 8, 9 or 10-point scale) [42], and allow a participant to express a true neutral stance (as opposed to an even-numbered scale) [42]. The Flesch-Kincaid reading ease score will be examined to ensure the items are clear and concise for easier interpretation [55, 56].

Expert validation.

To guarantee the item’s relevance and representation of the target construct, expert validation will be performed according to the best practices recommended by Yusoff [57]. A panel of experts will review the list of generated items after they are provided with instructions, definitions, and ratings to facilitate the evaluation process. A target number of ten experts will be recruited to assist in this step because the minimum recommended number is six [57], which will account for the possibility of dropouts. A total of four types of experts will be included. They will consist of clinicians who represent the end users of the medical training’s outcomes. Experts representing those in cultural sensitivity in Malaysian society will also help vet sensitivity, accessibility, and inclusivity. We will include experts in feedback literacy and questionnaire development.

The expert validation form will be distributed via email. The panel of experts will be asked to evaluate the items and provide comments to improve the instrument before independently rating the items. If needed, virtual or physical meetings will be conducted for additional consultation and explanations. Based on their scores on the content validation index (CVI), items will be refined or discarded. With ten experts, a CVI score of at least 0.78 should be achieved to ensure content validity [58].

Cognitive interviews.

The first draft of the questionnaire will be validated by experts and then tested with a group of medical students to strengthen comprehensibility. Cognitive interviews allow the exploration of the reasons why medical students might find the questionnaire to be comprehensible or lack thereof. The execution of cognitive interviews will follow the recommendations of Scott et al., who provide a step-by-step approach with illustrative examples [59]. Semi-structured individual interviews will be conducted with Malaysian medical students, who are the target population of the survey, face-to-face or virtually, and a researcher will be trained to conduct them. The interviews will be audiotaped and then transcribed verbatim.

An interview protocol will be designed based on the drafted items and piloted with two to three medical students to test its feasibility and implementation of the interview protocol [60]. The cognitive interview protocol might be revised after the pilot test to ensure that the questions elicit the needed response from medical students regarding the comprehensibility of the questionnaire items. During the interviews, the think-aloud approach will be adopted to ask interviewees to verbalize their thought processes while answering the questionnaires [61]. In addition, probing questions will be used to guide the focus on specific parts and ensure clarity among the interviewees [62]. Interview transcripts will be analyzed to identify areas for improvement and the questionnaire modified based on these findings. The cognitive interviews will be repeated until no new problems are identified and the questionnaire is deemed comprehensible and comprehensive [62].

National survey.

To extend its accessibility to participants nationwide, the medical student feedback literacy scale will be administered online. Approximately 3000 medical students are studying in Malaysia [63]. According to Krejcie and Morgan’s [64] calculations, the minimum sample size is 341. An online sample size calculator (http://www.raosoft.com/samplesize.html) (5% margin of error, 95% confidence interval) verified that a minimum of 341 participants is required. To accommodate the planned factor analysis, a larger sample size may be required to achieve a 7:1 ratio of items to participants, depending on the number of items developed [34]. Medical students in Malaysia will be recruited via social media posts and student societies. Recruitment will be periodically performed until a sufficient sample size is obtained.

This national survey will ask participants about their age, sex, ethnicity, institution, and year of study. The data will demonstrate whether the sample reflects different backgrounds to represent the Malaysian population. Recruitment will be repeated until the sample accounts for the various backgrounds. The researchers will be prepared to recruit a larger sample to guarantee representation.

The adequacy of the sample size should also fulfill the requirements of statistical analyses. This protocol will follow a generic recommendation that at least 200 respondents are required [65]. In addition, for exploratory factor analysis (EFA), the recommended respondent-to-item ratio is 1:10 [66], or a minimum of 1:5 [67, 68]. For confirmatory factor analysis (CFA), the recommended respondent-to-item ratio is 1:10 [69].

Data collected from the national survey will be analyzed for construct validity, internal consistency, hypothesis testing, and test-retest reliability. All data will be managed and analyzed using IBM SPSS Statistics 28.

Construct, convergent, and discriminant validity.

To guarantee a comparable and representative sample, data for EFA and CFA will be collected from the same national survey. The samples will be randomly split into two subsamples prior to data analysis to undergo either EFA or CFA [70].

The EFA will be performed to establish a hypothesis for the factor model among Malaysian medical students. Furthermore, EFA will be performed according to the guidelines described by Hair et al. [71]. A principal component analysis (PCA) with varimax rotation will be performed. The Bartlett’s test measure of sampling adequacy should be significant at p = 0.05, and the Kaiser-Meyer-Olkin (KMO) should be above 0.7 [72]. The initial PCA will yield the proposed factors. Those with eigenvalues greater than 1 in the scree plot will be considered [73]. Next, items with an average communality above 0.6 [74] will be considered. In addition, items with a factor loading above 0.5 in one factor [71] and without cross-loading in another factor above 0.5 [75] will be retained for subsequent analyses.

To further assess whether the hypothesis proposed by the EFA is plausible, the items retained from the EFA will undergo CFA. The CFA will be performed according to the guidelines developed by Hair et al. [71]. Indices such as the ratio of chi-square and degrees of freedom, goodness-of-fit index (GFI), Tucker-Lewis index (TLI), comparative fix index (CFI), adjusted goodness-of-fit index (AGFI), and root mean square error of approximation (RMSEA) will be evaluated. A model will be deemed to have a good fit if the chi-square to degrees of freedom ratio is less than three, the RMSEA is less than 0.08, and the other indices have values greater than 0.9 [71]. Items not meeting these criteria will be excluded from subsequent analyses.

In addition, the structure of the model will be assessed for convergent (i.e., the extent to which an indicator of a specific construct converges) and discriminant validity (i.e., distinct constructs are truly unique and measure phenomena that other constructs do not). To achieve convergent validity, the factor loadings and average variance extracted (AVE) and construct reliability should be equal to or higher than 0.5 and 0.7, respectively [71]. No cross loadings should occur between indicators to support discriminant validity, and the AVE values should be higher than the squared correlation estimates [71].

Internal consistency.

Based on the factors produced via the EFA, the internal consistency between these factors will be determined to evaluate whether the scores are consistent throughout the instrument [31]. To guarantee internal consistency, a Cronbach’s alpha of more than 0.70 will be targeted [71]. Items should demonstrate a coefficient above 0.5 and have a corrected item-total correlation above 0.2 within a factor [72, 76].

Hypotheses testing.

The Implicit Theory of Intelligence Scale [77] will be administered simultaneously during the national survey. The purpose is to further support the construct validity of the instrument through hypothesis testing. Such testing will support the structure of a new construct by correlating it with a similar construct [34].

Dweck’s Mindset Theory [78] postulates that individuals’ mindsets range from a growth to fixed mindset. People with the former believe they can enhance their abilities by receiving and applying feedback while those with the latter tend to believe that their abilities are predetermined. The proposed hypotheses are that a higher level of feedback literacy will be correlated with a growth mindset, and a lower level with a fixed mindset. Pearson’s correlation coefficient higher than 0.70 will be targeted to establish a strong correlation between the two constructs [79].

Test-retest reliability.

Test-retest reliability will be evaluated to examine the stability of our measurements over time [31]. Medical students at the Universiti Malaya will be recruited to complete the instrument twice. A sample of 50 medical students will be used to obtain heterogeneity within the sample [80]. To prevent recall and minimize the effect of behavioral changes on feedback literacy, the time interval will be two weeks [81, 82]. Students will be asked to fill in their student identification numbers to match their responses; however, to preserve their anonymity, these will be replaced by running numbers. Based on the McGraw and Wong convention, the reliability score will be calculated with a two-way mixed effect, single measurement, absolute agreement intraclass coefficient (ICC) [80, 83]. A minimum value of 0.75 will be targeted to achieve good test-retest reliability [80, 84].

International survey.

After the national survey is completed, the questionnaire will be disseminated online to medical students from other countries to gather evidence of its cross-cultural validity. According to Hofstede’s cultural dimensions, the survey will be replicated in countries with similar dimensions [85] (e.g., Singapore) and dissimilar (e.g., Canada) to Malaysia to allow comparison with a range of cultural contexts. We will apply for ethical approval at the respective institutions, and the sample size will be determined according to the number of medical students in the country and to achieve a 1:10 ratio of items to participants for factor analysis.

Cross-cultural validity.

Cross-cultural validity will be investigated using multi-group CFA (MGCFA). This method has been chosen for its ability to assess all hierarchical factorial invariances and both uniform and nonuniform item biases [86]. MGCFA will be performed following Gregorich’s recommendations [87]. Dimensional, configural, metric, and scalar variances will be examined to support cross-cultural validity. Dimensional and configural invariance will be determined by comparing the number of common factors and their associated items, respectively [87]. Metric invariance will be investigated by fixing equality constraints on the matching factor loadings and concurrently fitting the factor model to sample data from each group [87]. Scalar invariance will be assessed by fixing equality constraints on the corresponding factor loadings and item intercepts and fitting the common factor model to the sample data from each group simultaneously [87]. Differences in the fit indices of 0.01 or less support the metric and scalar invariance hypotheses, respectively [86]. If the metric invariance hypothesis is supported, comparisons of correlations and mean patterns are valid across cultural samples. Meanwhile, a supported scalar invariance hypothesis allows for the comparison of instrument scores across cultural samples.

If different cultural models exhibit partial factorial invariance or non-invariance, items and factors that do not fit the invariance criteria will be analyzed to determine their causes. Misfit items, factors, and samples will be excluded to assess whether this results in improved invariance indicators [86]. Based on the results, the models will be assessed for cultural construct bias and extreme response styles across the samples [86, 87].