https://orcid.org/0000-0002-5700-4315
Figures
Abstract
Purpose
To adapt the Scale for the Evaluation of Staff Patient Interactions in Progress Notes to Spanish and to test the psychometric properties.
Design and methods
The study was conducted in two phases: (1) Adaptation of the instrument to Spanish following the Standards for Educational and Psychological Testing. (2) Psychometric study in a sample of mental health nurses.
Citation: Pérez-Toribio A, Moreno-Poyato AR, Lluch-Canut MT, Nash M, Cañabate-Ros M, Myklebust KK, et al. (2023) Transcultural adaptation and assessment of psychometric properties of the Spanish version of the Scale for the Evaluation of Staff-Patient Interactions in Progress Notes. PLoS ONE 18(3): e0281832. https://doi.org/10.1371/journal.pone.0281832
Editor: Santiago Gascón, Universidad de Zaragoza, SPAIN
Received: September 12, 2022; Accepted: February 2, 2023; Published: March 28, 2023
Copyright: © 2023 Pérez-Toribio et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: ARMP, JRM, TLLC, MPLL: ARMP, MCR, JRM, TLLC, MPLL: PI21/00605 Instituto de Salud Carlos iii, Ministerio de Ciencia e Innovación, Spain Co-funded by the European Union The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In mental health nursing practice, the therapeutic relationship between the nurse and the patient is of special importance [1]. Therefore, it is essential that the quality and extent of nurse-patient interactions that occur in the context of the relationship are properly described in the nursing documentation [2].
The therapeutic relationship is known to be the cornerstone of mental health nursing [3, 4]. This is characterized by the interpersonal relationship that is established between the nurse and the patient, a relationship based on trust between the two parties, and focused on therapeutic assistance [5]. Through the therapeutic relationship, the nurse helps the patient to develop and work individually towards their wellbeing, thus empowering the patient in their recovery process [6]. Establishing an appropriate therapeutic relationship is associated with better health outcomes for patients, improving the effectiveness of interventions in inpatient mental health care and enhancing the well-being and experience of patients as well as nurses [7, 8]. For this reason, it is important to make nurses aware of the importance of maintaining a quality relationship with patients [9]. A relationship based on a solid bond that enables understanding of the patient’s needs and helps the relationship process to reach an agreement on the objectives and interventions to be carried out with the patient [10].
However, it is important to note that any nurse-patient interaction can be therapeutic, while at the same time it has the potential to be nontherapeutic if conducted incorrectly [11]. The literature identifies the most important barriers to establishing the therapeutic relationship as perceived by nurses: a) excessive administrative tasks that limit the time available for other activities [12], b) the lack of private space [5], c) lack of leadership and support from supervisors [13]. In addition, the biomedical model also constitutes a barrier to the therapeutic relationship [14]. This model conditions the language used in mental health nursing, a particularly important language that should be of great use for building and maintaining a therapeutic relationship [15].
Nurses are the staff members who have the most interactions with patients throughout the day [16]. Indeed, nurses represent the largest number of people working in mental health units [17]. Therefore, what happens in the interactions must be described in the nursing documentation of the clinical history [17]. Evidence suggests that the nursing records can be an effective way to improve nursing practice [18]. However, it has been shown that the quality of written mental health nursing documentation needs improvement to ensure continuity and patient safety [19]. Many nursing interventions that take place in the context of the therapeutic relationship are often not recorded in clinical courses [20], but they are more focused on problems and symptoms [21]. In light of the above, it is evident that there is a need for instruments to evaluate the quality of nurses’ clinical notes, especially in the context of records in which clinical information obtained in nurse-patient interactions is collected.
In order to assess the quantity and quality of interactions between nurses and patients described in clinical notes in mental health units, Myklebust and Bjørkly [2] developed and reliability tested the Scale for the Evaluation of Staff-Patient Interactions in Progress Notes (SESPI). This scale enables the evaluation of nurses’ clinical notes in relation to the quality of the patients’ experience, the approaches taken by the nurses and whether this approach meets the emotional needs of the patients [2].
Supporting the need and relevance of having an instrument to assess the quality of nursing clinical notes in acute mental health services in Spain, and given the usefulness of this instrument for assessing clinical notes in the original context, and whereas that no other tools have been found that assess the quality of nursing clinical notes and are suitable for acute mental health units, the aim of this study was to perform the cross-cultural adaptation of the SESPI scale to Spanish and to study its psychometric properties.
Methods
Design
A psychometric study of cross-cultural adaptation and validation of the SESPI Scale into Spanish was performed. The study was conducted in two phases: in the first phase, the SESPI scale was adapted to Spanish and in the second phase the psychometric properties of the Spanish version were analysed (Fig 1).
Instrument
The SESPI scale assesses the quality of the patient experience, the approaches taken by nurses and whether this approach meets the emotional needs of the patients [2]. This scale consists of the following four steps:
- Step 1 aims to know whether the patient’s experience is described in the clinical note. It is answered dichotomously (Yes/No), and only an affirmative answer enables one to proceed to the next step.
- Step 2 evaluates whether the patient’s experience in the interaction with the nurse has been positive or negative. Depending on the degree of satisfaction or dissatisfaction of the patient’s response, the answer is -II, -I, +I, +II.
- Step 3 identifies and categorizes the nurse’s response to the interaction described in the clinical note. There are four possible responses: a) The nurse’s response is not described in the clinical note; b) The nurse’s response is known, but not the patient’s reaction; c) We know the nurse’s response and the patient’s reaction, but not the patient’s feelings; d) The nurse’s response, the patient’s reaction and the feelings produced by this intervention are noted. Only this response enables one to move on to the next step where the clinical note is evaluated regarding to attunement.
- In step 4 this attunement is evaluated positively or negatively according to whether it was unsuccessful or successful. Again, the response is -II, -I, +I, +II depending on how unsuccessful or successful the attunement was.
Regarding the psychometric results of the original scale, the Cronbach’s alpha for the entire instrument was 0.977 and the ICC was 0.770 [2].
Phase 1 procedure
The process of cultural adaptation of the Spanish version of the SESPI scale was carried out following the Standards for Educational and Psychological Testing [22]. Before beginning the translation, permission was requested from the authors for its adaptation to Spanish. The English version of the instrument was translated into Spanish by two translators, whose mother tongue was Spanish and who were fully fluent in English, providing two versions of the SESPI instrument (v1 and v2), which were synthesized to a single version (v3) by a committee of experts formed by four nurses specialized in mental health and a nurse specialized in psychometrics. This version was sent to two new translators who did not know the original version, whose mother tongue was English and who were fluent in Spanish, so that they could carry out the back-translation into English. The two versions obtained (v4 and v5) were compared with the original SESPI by the same committee of experts, resulting in version 6, which was revised together with the original authors.
A pre-test was then carried out on a sample of 19 nurses with experience in recording information in clinical records. To assess the comprehension and clarity of the items, as well as the format and time required to complete the instrument, four clinical notes were provided. The clinical notes that were selected were written by nurses in which the patient’s experience was present in the situation described and which narrated episodes in which the staff and patients had the opportunity to interact.
Phase 2 procedure
For the reliability testing of the psychometric properties of the Spanish version of the SESPI-Sp instrument, 12 clinical notes were purposively selected from patient records from three different hospitals, representing the totality of possible responses to the scale. The clinical notes were collected in a single coded and anonymized document (S1 File). The selected clinical notes tried to represent the totality of the possible responses contained in SESPI, therefore, a heterogeneous sample was used trying to fulfil that purpose.
To facilitate data collection, an electronic adaptation of the SESPI scale was developed using the Microsoft forms platform. Thus, the research team sent each participating nurse an e-mail with the evaluator code, the document with the clinical notes to be evaluated, and the link to the form containing the SESPI scale, to be completed as many times as there were clinical notes to be evaluated.
Participants
Following the recommendations by Bujang and Baharum [23], for the 12 observations (clinical notes) that had to be analysed by the evaluators in our study (statistical power of 80%, alpha = 0.05 and an effect size difference of ICC = 0.20, 20 nurses were required. Similarly, the criterion used in the original SESPI reliability test was also taken into account [2] accepting a detection of an effect size difference of 0.20 (ICC = 0.70 vs. 0.50) as being acceptable for testing instrument development.
The invitation and selection of the nurse evaluators were made according to the following criteria: nurses from the mental health hospitalization area, with experience in collecting and writing information in medical records, with at least two years of experience in mental health nursing. The nurses were selected by purposive sampling. Finally, 29 nurses participated.
Statistical analysis
This study used the same statistical analysis paradigm for testing reliability as was used for the original SESPI [2]. Two tests were used to analyse the reliability of the instrument. The Kuder-Richardson 20 test was used for the dichotomous scale in step 1. The Cronbach’s alpha test was used to estimate the internal consistency of the scores for the total SESPI scale and for steps 2, 3 and 4. Likewise, the intraclass correlation coefficient (ICC) was chosen to calculate the inter-rater reliability of the distribution of scores for the 12 excerpts across the four steps of coding and for steps 1, 2, 3 and 4 separately. Statistical analyses were conducted using the Statistical Package for Social Science (SPSS) Version 27.
Ethical considerations
The study was approved by the IRB Bioethics Committee (2021/9835) and the other participating hospitals. The authorization and participation of the authors of the original version of the instrument was requested for its cross-cultural adaptation. All participants signed the informed consent form. To maintain the anonymity of participants, the principal investigator previously assigned an evaluator code to each nurse, sent via e-mail along with the form using the Microsoft forms platform in which they entered their evaluations of the SESPI-Sp.
Results
Phase 1: Adaptation of the SESPI scale
The translation process was carried out systematically and involved discussions of semantic, idiomatic, and conceptual equivalence at each stage. Regarding the content of the instrument, most of the steps, categories and examples were simple and therefore easy to translate. Only a few words required adaptation to bring the meaning closer to the Spanish context in which SESPI was to be used. The translations of these words in “v.1 and v.2” were easily agreed upon by the expert committee. However, the most complex adaptation had to do with the term "excerpts" since it has several accepted translations in Spanish, but in the context of acute hospitalization units in Spain, "nota del curso clínico" or "registro" is normally used to define an "extract" of a clinical note.
Before finalizing the Spanish version of SESPI, a pilot test was performed in which 19 nurses participated and evaluated four clinical notes. The nurses who participated in the pre-test of the SESPI-Sp required an estimated average time to evaluate the four notes that ranged from 5 to 10 minutes and confirmed that it was easy to complete. After completing the SESPI-Sp, participants were asked to report any items with an unclear meaning. All such reported comprehension problems were discussed by three authors (APT, AMP, JRM) and used to inform the modifications of the SESPI-Sp. One of the most common comments was that the format and wording did not help to understand the steps. In this sense, the format of the tool was partially modified in an attempt to clarify how to correctly complete and use the resources made available, keeping the content and structure intact so as not to detract from the semantic essence. Thus, on the 1st and 2nd pages of the SESPI-Sp, instructions on how to respond to each of the four steps and how to categorize the responses were incorporated. Another slight modification to the format was to unify each of the categories belonging to the four SESPI steps in a single table. The purpose of this modification was to achieve a more intuitive view of the steps and categories and thus reduce the confusion generated in the pre-test. In the table for each category, a brief explanation of the interpretation of this step was included, which in the original SESPI is included before the examples of situations. For each of the steps, the original SESPI presents examples that help the user of the tool to identify the corresponding category according to what is written in the clinical note. These examples are written after the brief explanation of each of the step categories. The cross-culturally adapted version places these examples at the end of the tool to have them available for consultation in case of doubt, yet separate from the categorization section, to avoid distracting the user. These modifications to the format do not change the meaning or usefulness of the tool. The original authors were aware of these modifications and gave their approval. The suggestions and proposals for improvement that they provided in their feedback were considered and were included in the final version, which was named SESPI-Sp (S2 File). Fig 2 shows the structure of SESPI and SESPI-Sp.
Phase 2: Psychometric properties of the SESPI-Sp
Participant characteristics.
Table 1 shows the characteristics of the 29 participating nurses who served as evaluators in the study. The age of the nurses ranged from 26 to 59 years, with a mean age of 35.6 years (SD = 8.7). Almost 60% of the nurses were female. Their experience in mental health was a mean of 10.9 years (SD = 9.1). More than half had the official title of mental health nurse specialist and had a doctoral or master’s degree. Their experience writing clinical notes ranged from 3 to 38 years, with a mean of 10.6 years (SD = 8.5).
Inter-rater and internal consistency reliability of SESPI estimated for the entire instrument.
The Cronbach’s alpha for the total instrument was 0.976. As in the original version, this value indicates that the raters’ scores had a very high consistency for the four steps of the SESPI.
Inter-rater and internal consistency reliability for each step of the SESPI.
Step 1 was completed using a dichotomous yes or no scale. The reliability analyzed using the Kuder-Richardson-20 coefficient was 0.949. The ICC for step 1 was 0.942 (95% CI: 0.082–0.995).
Steps 2 to 4 were evaluated using a four-point ordinal scale. Table 2 shows the values of Cronbach’s alpha and the intraclass correlation coefficient for steps 2, 3 and 4. Values above 0.941 were obtained for Cronbach’s alpha and the ICC for the total and for each of the steps analysed.
Discussion
This study aimed to carry out a cross-cultural adaptation and test the reliability of the Spanish version of the SESPI instrument. The purpose of this instrument is to evaluate the quality of nurse-patient interactions recorded in clinical notes in mental health hospitalization services.
Regarding the adaptation process, it should be noted that it was carried out swiftly and here were hardly any discrepancies among the participants during both the translation and back-translation processes; only some terms required adaptation to the Spanish context. This fact confirms the equivalence in meaning and the reliability of the semantic content of the Spanish version of the instrument [24]. Another important aspect that should be emphasized is that during the adaptation process the structure of the tool required a slight modification in terms of where the indications and examples were provided, although this did not lead to changes in the sense and meaning of the content of the instrument. The fact that this was the result of the recommendations of the participating clinical nurses, that the expert committee evaluated it positively and, moreover, that it received the approval of the original authors, guarantees the level of reliability of the adapted version of the instrument [24, 25].
In relation to the psychometric results, it should be noted that the reliability of the Spanish version of the SESPI showed excellent results regarding internal consistency and interobserver reliability. In the case of the total Cronbach’s alpha test it was 0.976, a very similar result to that of the original study testing the SESPI’s reliability [2]. In the case of the ICC, the result for the total scale was 0.975, indicating very good interobserver reliability [26]. In fact, for some authors, above 0.81 is considered almost perfect [27]. This result was substantially higher than that obtained in the original SESPI version [2], this finding could be due to a larger number of observations and observers in this study. For an in-depth study of the results of the different stages that could be answered with the instrument, 12 clinical notes were selected that included the different options that could be presented and were evaluated by 29 nurses with different profiles of clinical experience in mental health. In this sense, the results obtained for each stage of the instrument were also very acceptable, obtaining Cronbach’s alpha values above 0.94 for all the steps, along the same lines as the original version [2].
Strengths and limitations
Regarding the strengths of the study, in the first place, standardized translation and cross-cultural adaptation procedures were used [22]. SESPI has the great advantage of being a short questionnaire that requires only a short time for evaluation. This study reviewed numerous clinical notes from acute mental health units in several hospitals in Spain. However, for the psychometric study, 12 clinical notes were selected, in order to represent the different possible responses to the SESPI and to study the consistency of the responses in detail. The lack of scales for evaluating the quality of the clinical notes translated and validated in Spanish did not prevent us from reproducing the process of evaluating the psychometric properties so that it would be similar to that of the original study. The participants were recruited from a group of nurses in the mental health field and the selection process considered that the profiles of the nurses were representative of the group. However, the nurses who acted as evaluators showed a high education level and motivation to collaborate in this study. This fact could have conditioned the results. The findings may not be directly generalisable to nurses and staff with different educational backgrounds. A scale’s reliability is not a fixed quality, but depends on the specific group of raters and their training in the use of the scale [28]. Clearly, when the SESPI is used in research projects, new reliability tests are required for each project. This does not mean that our reliability testing of the SEPSI is superfluous. Any development of a new test or instrument requires reliability and validity testing. This is the first study measuring clinical nursing notes using the Spanish version of the SESPI-Sp instrument, new studies are required, with larger samples, to validate our findings.
Implications for practice
The process of translation, cross-cultural adaptation, and evaluation of psychometric properties indicated that SESPI-Sp is an instrument capable of providing valid and reliable measures to assess the quality of clinical notes in acute mental health inpatient services in the Spanish context. Therefore, this study has special relevance for management, clinical practice, teaching and research. With this tool, a critical examination of nursing documentation and the identification of gaps in nursing writing is achieved. Firstly, its use allows managers to be aware of the quality of the records. In addition, it is relevant for clinical practice, as being able to properly record interventions following specific steps, nurses will be more aware of the steps they need to follow to correctly record patients’ progress notes. In this sense, it can be an important tool for nurses’ decision-making processes, especially for planning goals and interventions in the context of patient interactions. In addition, knowing how to critically evaluate records can generate further knowledge related to the quality of the nursing record in the patient’s clinical notes. The availability of this instrument may be useful for designing training programs for nurses oriented towards the problem areas described by nurses and confirmed by patients’ expectations. Finally, new lines of research are opened that may generate further knowledge related to the quantity and quality of the nursing record in the patient’s clinical notes.
Conclusions
This study has enabled a Spanish adaptation of an instrument to assess the quality of nurse-patient interactions recorded in clinical notes, and to study its psychometric properties. The SESPI-Sp is a simple, easy to administer and reliable tool for assessing aspects of the quality of mental health care. The results of the reliability tests performed with the SESPI-Sp confirm its usefulness for quantitative research as was the case with the original tool. The SESPI can be used to measure the quantity and quality of nurse-patient interactions described in nurses’ clinical notes. The SESPI-Sp measures patient interactions from the perspective of attunement, including the quantity and quality of patient experiences described, staff approaches, and whether the approaches succeeded in meeting patients’ needs. The SESPI-Sp can provide data to evaluate nursing documentation qualitatively in relation to nursing interventions delivered in the context of interactions with patients in mental health units. Thus, contributing to the improvement of the quality of nursing care in mental health.
Supporting information
S2 File. Scale for the Evaluation of Staff Patient Interactions in Progress Notes.
https://doi.org/10.1371/journal.pone.0281832.s002
(PDF)
References
- 1. Lessard-Deschênes C, Goulet MH. The therapeutic relationship in the context of involuntary treatment orders: The perspective of nurses and patients. J Psychiatr Ment Health Nurs. 2021;(September):1–10.
- 2. Myklebust KK, Bjørkly S. Development and reliability testing of the Scale for the Evaluation of Staff-Patient Interactions in Progress Notes (SESPI): An assessment instrument of mental health nursing documentation. Nurs Open. 2019;6(3):790–8. pmid:31367401
- 3. McAndrew S, Chambers M, Nolan F, Thomas B, Watts P. Measuring the evidence: reviewing the literature of the measurement of therapeutic engagement in acute mental health inpatient wards. Int J Ment Health Nurs. 2014 Jun 17;23(3):212–20. pmid:24103061
- 4. Wright KM. Exploring the therapeutic relationship in nursing theory and practice. Ment Heal Pract. 2021 Jul 27.
- 5. Moreno-Poyato AR, Montesó-Curto P, Delgado-Hito P, Suárez-Pérez R, Aceña-Domínguez R, Carreras-Salvador R, et al. The Therapeutic Relationship in Inpatient Psychiatric Care: A Narrative Review of the Perspective of Nurses and Patients. Arch Psychiatr Nurs. 2016 Dec;30(6):782–7. pmid:27888975
- 6. Zugai JS, Stein-Parbury J, Roche M. Therapeutic alliance in mental health nursing: An evolutionary concept analysis. Issues Ment Health Nurs. 2015;36(4):249–57. pmid:25988275
- 7. McAllister S, Robert G, Tsianakas V, McCrae N. Conceptualising nurse-patient therapeutic engagement on acute mental health wards: An integrative review. Int J Nurs Stud [Internet]. 2019;93:106–18. Available from: https://doi.org/10.1016/j.ijnurstu.2019.02.013 pmid:30908958
- 8. Moreno-Poyato AR, Delgado-Hito P, Suárez-Pérez R, Leyva-Moral JM, Aceña-Domínguez R, Carreras-Salvador R, et al. Implementation of evidence on the nurse-patient relationship in psychiatric wards through a mixed method design: study protocol. BMC Nurs [Internet]. 2017 Dec 11;16(1):1. Available from: http://bmcnurs.biomedcentral.com/articles/10.1186/s12912-016-0197-8 pmid:28096737
- 9. Tolosa‐Merlos D, Moreno‐Poyato AR, González‐Palau F, Pérez‐Toribio A, Casanova‐Garrigós G, Delgado‐Hito P. Exploring the therapeutic relationship through the reflective practice of nurses in acute mental health units: A qualitative study. J Clin Nurs. 2023 Jan 24;32(1–2):253–63. pmid:35075705
- 10. Moreno‐Poyato AR, Rodríguez‐Nogueira Ó. The association between empathy and the nurse–patient therapeutic relationship in mental health units: a cross‐sectional study. J Psychiatr Ment Health Nurs. 2021 Jun 3;28(3):335–43. pmid:32657511
- 11. McAllister S, McCrae N. The therapeutic role of mental health nurses in psychiatric intensive care: A mixed-methods investigation in an inner-city mental health service. J Psychiatr Ment Health Nurs. 2017;24(7):491–502. pmid:28407389
- 12. Thompson L, McCabe R. The effect of clinician-patient alliance and communication on treatment adherence in mental health care: a systematic review. BMC Psychiatry. 2012 Dec;12(1):87. pmid:22828119
- 13. Bowers L, Nijman H, Simpson A, Jones J. The relationship between leadership, teamworking, structure, burnout and attitude to patients on acute psychiatric wards. Soc Psychiatry Psychiatr Epidemiol. 2011 Feb 16;46(2):143–8. pmid:20082064
- 14. Myklebust KK, Bjørkly S, Råheim M. Nursing documentation in inpatient psychiatry: The relevance of nurse–patient interactions in progress notes—A focus group study with mental health staff. J Clin Nurs [Internet]. 2018 Feb 4;27(3–4):e611–22. Available from: https://onlinelibrary.wiley.com/doi//10.1111/jocn.14108 pmid:29048775
- 15. Bradley P, Lowell A, Daiyi C, Macklin K, Nagel T, Dunn S. It’s a little bit like prison, but not that much: Aboriginal women’s experiences of an acute mental health inpatient unit. Int J Ment Health Nurs. 2021 Aug;30(4):917–30. pmid:33630404
- 16. Bridges J, Griffiths P, Oliver E, Pickering RM. Hospital nurse staffing and staff–patient interactions: an observational study. BMJ Qual Saf [Internet]. 2019 Sep;28(9):706–13. Available from: https://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2018-008948 pmid:30918050
- 17. Myklebust KK, Bjørkly S. The quality and quantity of staff-patient interactions as recorded by staff. A registry study of nursing documentation in two inpatient mental health wards. BMC Psychiatry. 2019 Dec 14;19(1):251. pmid:31412803
- 18. Urquhart C, Currell R, Grant MJ, Hardiker NR. Nursing record systems: effects on nursing practice and healthcare outcomes. Cochrane database Syst Rev. 2018 May 15;5(5):CD002099.
- 19. Instefjord HH, Aasekjær K, Espehaug B, Graverholt B. Assessment of quality in psychiatric nursing documentation—a clinical audit. BMC Nurs. 2014;13(1):1–7.
- 20. Pérez‐Toribio A, Moreno‐Poyato AR, Lluch‐Canut T, Molina‐Martínez L, Bastidas‐Salvadó A, Puig‐Llobet M, et al. Relationship between nurses’ use of verbal de‐escalation and mechanical restraint in acute inpatient mental health care: a retrospective study. Int J Ment Health Nurs. 2022 Apr 26;31(2):339–47. pmid:34837275
- 21. Buus N, Hamilton BE. Social science and linguistic text analysis of nurses’ records: a systematic review and critique. Nurs Inq. 2016 Mar;23(1):64–77. pmid:26109278
- 22.
Frey BB. Standards for Educational and Psychological Testing. In: Frey BB, editor. The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation [Internet]. 2455 Teller Road, Thousand Oaks, California 91320: SAGE Publications, Inc.; 2018. Available from: https://methods.sagepub.com/reference/the-sage-encyclopedia-of-educational-research-measurement-and-evaluation/i19872.xml
- 23. Bujang MA, Baharum N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: A review. Arch Orofac Sci. 2017;12(1):1–11.
- 24. Hernández A, Hidalgo MD, Hambleton RK, Gómez-Benito J. International test commission guidelines for test adaptation: A criterion checklist. Psicothema. 2020;32(3):390–8. pmid:32711675
- 25. Muñiz J, Elosua P, Hambleton RK, International Test Commission. [International Test Commission Guidelines for test translation and adaptation: second edition]. Psicothema. 2013;25(2):151–7.
- 26. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. pmid:27330520
- 27. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977 Mar;33(1):159. pmid:843571
- 28. Streiner DL, Kottner J. Recommendations for reporting the results of studies of instrument and scale development and testing. J Adv Nurs. 2014 Sep;70(9):1970–9. pmid:24684713