Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

KOJI AWARENESS, a self-rating whole-body movement assessment system, has intersession reliability and comparability to external examiner rating

Abstract

KOJI AWARENESS is a newly developed self-rating whole-body movement assessment system that includes 11 domains and 22 tests. The primary aim of this study was to investigate the intersession reliability of KOJI AWARENESS, and the secondary aim was to determine whether a fixed bias existed between self-rating and external examiner rating. Fifty university students rated their movement ability in two separate sessions; an external examiner also rated the students’ movement ability. Participants were blinded to their scores at the first session as well as the external examiner’s rating scores. The primary analysis included examining the intersession reliability of the total score with intraclass correlation coefficients (ICCs). ICC values were interpreted as follows: insufficient, < .7 and sufficient, ≥ .7. To achieve the secondary aim, Bland–Altman analysis was performed. ICC for the intersession reliability was .86 with a 95% confidence interval (CI) of .77 to .92 and a minimum detectable change (MDC) of 5.15. Bland–Altman analysis revealed fixed bias as the 95% CIs of the mean difference between the two different rating scores (−3.49 to −2.43 and −3.94 to −2.98 in the first and second sessions, respectively) did not include 0 in the data of each session. However, no proportional bias was identified because no statistically significant Pearson’s correlation (P > .05) was noted between the means of the two methods and the mean difference between the two different methods in each session. This study identified that KOJI AWARENESS has sufficient intersession reliability among relatively young and healthy participants. External examiner rating tended to have lower total scores than self-rating; however, the bias was below the MDC and seemed not to be clinically significant.

Citation: Takasaki H, Kanayasu S (2024) KOJI AWARENESS, a self-rating whole-body movement assessment system, has intersession reliability and comparability to external examiner rating. PLoS ONE 19(8): e0308973. https://doi.org/10.1371/journal.pone.0308973

Editor: Holakoo Mohsenifar, Iran University of Medical Sciences, ISLAMIC REPUBLIC OF IRAN

Received: May 28, 2024; Accepted: August 3, 2024; Published: August 15, 2024

Copyright: © 2024 Takasaki, Kanayasu. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Maintaining movement ability throughout life is essential for health and wellness. Self-monitoring is necessary to maintain movement ability. In particular, an assessment tool to monitor whole-body movement ability is an efficient self-monitoring strategy. Functional limitations in a specific part of the body can overload other parts and consequently affect whole-body postural control [1]; therefore, to identify problems that are not captured by the functional assessment of individual parts of the body, whole-body movement assessment would be beneficial.

There are several performance-based outcome measures (PBOMs) on whole-body movement assessment, including the Selective Functional Movement Assessment (SFMA) and Functional Movement Screen (FMS). However, these PBOMs require experienced examiners. In 2022, a self-assessment PBOM on whole-body movement ability without instruments was developed and named as KOJI AWARENESS. It incorporates movement tests of 11 components, with individuals marking their own movement score. This assessment is designed to be applied to individuals of all ages, with or without pain or musculoskeletal problems, as long as the instructions are understood. The total score of KOJI AWARENESS is correlated with that of FMS [2], an established PBOM with interexaminer reliability [3]. This finding supports the validity of KOJI AWARENESS. Moreover, there is emerging evidence supporting its usefulness, such as KOJI AWARENESS scores correlated with the degree of pain during training in athletes, and pain during training was also reduced after exercises that improved the KOJI AWARENESS score [4,5].

However, the reliability of this tool has not been fully investigated yet. KOJI AWARENESS is a self-assessment tool; hence, investigating its intersession reliability is a research priority. Furthermore, interexaminer reliability, i.e., reliability between self-rating and examiner rating, may not be a research priority because a correlation between examiner rating movement ability using FMS and self-assessment using KOJI AWARENESS has already been reported [2]. However, there can be a difference between the examiner rating score and the self-rating score.

This study primarily aimed to investigate the intersession reliability of KOJI AWARENESS. The secondary aim was to investigate whether a fixed bias existed between self-rating and external examiner rating.

Materials and methods

Design

This study used a test–retest design, with participants’ self-rating scores and examiner’s rating scores from observation being blinded to the participants and examiner throughout the experiment. Further, self-scores at the first session were blinded to the participants and examiner at the second session. The second session was conducted within 1 week on a different day. Before data collection, each participant provided their written informed consent. This study was approved by an institutional research ethics committee (Saitama Prefectural University, No. 23018) and complied with the Declaration of Helsinki.

Participants and an external examiner

Fifty students were recruited in a local university (Saitama Prefectural University, Japan) from June 5, 2023 to January 17, 2024. Individuals who were unable or contraindicated to perform the movements such as standing up, sitting down, standing on one leg, kneeling, or bending the trunk were not eligible to participate in this study. We asked the participants not to change their exercise habits during the study and excluded those who changed their exercise habits between the first and second measurement weeks due to aggravation of injury or some other reasons. The number of participants was determined according to an adequate criterion in the consensus-based standards for the selection of health measurement instruments (COSMIN) guidelines [6].

The external examiner (SK) was a third year university student studying physical therapy. Prior to this study, he had participated in a level 1 workshop on SFMA to gain skills on movement observation. The examiner participated in this study after having fully understood KOJI AWARENESS by repeatedly watching the video (S1 File).

In addition to basic demographic information of the participants (i.e., age, sex, and body mass index [kg/m2] calculated with height and weight), considering future data accumulation, we used the EuroQol 5 Dimension (EQ5D) [7], which is free, multidimensional in content, and easy to use, to generally understand participant’s health status. The International Physical Activity Questionnaire short version (IPAQ) [8,9] was also collected to allow comparisons with future reliability studies in different populations as physical activity level may be associated with movement ability. The EQ5D is a reliable and valid patient-reported outcome measure of health status that includes five items of mobility, self-care, usual activities, pain or discomfort, and anxiety or depression [10]; a higher EQ5D score indicates better health status (0–1) [11]. The IPAQ is a reliable and valid self-reporting questionnaire for assessing average activity level in minutes × Mets [9,12,13]; a higher IPAQ score indicates greater activity level.

KOJI AWARENESS

KOJI AWARENESS includes the following 11 domains (Table 1), where details are presented in an S1 File in the previous study (https://doi.org/10.1371/journal.pone.0277167.s002) [2]. The total score which ranges from 0 to 50 and indicates an individual’s whole-body movement ability. The movements, notes, and scoring criteria for each movement test (https://doi.org/10.1371/journal.pone.0277167.s003) [2] were presented in a video, with 11 videos of 2–5 minutes each corresponding to the 11 components (S1 File).

thumbnail
Download:
Table 1. KOJI AWARENESS structure.

https://doi.org/10.1371/journal.pone.0308973.t001

No warm-up was allowed before the measurement. For each test and each measurement session, participants watched the video and moved spontaneously, and subsequently marked whether they met the criteria for that test on a sheet with prescored criteria. If participants did not understand a test movement, they were allowed to stop the video and repeat it as many times as they wished. To determine their score, each test was performed once. No instructions or feedback were provided by the external examiner. The examiner moved around freely to observe the participants’ test movements from the best angle. The participants and the external examiner scored simultaneously. To maintain examiner blinding, the above process was conducted under the supervision of the first author, with the two examiners submitting data separately to the first author.

Analysis

The intersession reliability of the self-rating and examiner-rating scores was examined using probability-adjusted kappa (PABAK) for the 22 tests with binary data (neck mobility; shoulder mobility; scapular mobility; hip mobility; and upper and lower extremity mobility and stability) and probability-adjusted kappa-ordinal scale (PABAK-OS) for the five tests with categorical data (thoracic spine mobility; upper extremity stability and strength; hip and spinal mobility; mid-section stability strength; and lower extremity strength). Moreover, PABAK-OS was also calculated in each domain. For the relative reliability of the total score, intraclass correlation coefficients based on a two-way mixed-effects mode (ICCs[3,1]) were calculated based on the ICC guideline [14]. For absolute reliability, minimum detectable changes (MDCs) were calculated using the standard deviations of the values and the ICC values on the basis of the following formulas: (1) (2)

Interpretations of the ICC and kappa values were as follows: insufficient, < .7; sufficient, ≥ .7 [15].

Fixed and proportional biases were examined for the total score using the Bland–Altman analysis [16]. Fixed bias was identified when the 95% confidence intervals (CIs) of the mean difference between the two different rating scores () did not include 0. The 95% CIs of the value was calculated using the following formula: (3) where n indicates the number of the participants; SDd indicates the standard deviations of the value; and t represents the degree of freedom of n–1. Proportional bias was identified when a statistically significant Pearson’s correlation (r) was noted in the mean difference between the two different methods () and the mean of the two different methods.

IBM SPSS Statistics for Windows version 28.0 was used for all statistical analyses, except PABAK-OS, which was performed online [17]. The significance level was set at 5%.

Results

None of the participants changed their exercise routine between the first and second weeks of measurement, and all were included in the analysis. No participants had any events influencing body movements, including injury or aggravation of pain, during the measurement and between the two assessments. The characteristics of the 50 participants are summarized in Table 2. Nine participants (18%) reported some pain in the pain/discomfort section of the EQ5D, but the degree of pain was slight for all of them, and none of them reported any problems with self-care or their usual activities.

thumbnail
Download:
Table 2. Summary of characteristics of the 50 participants.

https://doi.org/10.1371/journal.pone.0308973.t002

The PABAK values for interexaminer and intersession reliabilities in each test are presented in Table 3. The reliability in most tests (85%: 23/156 tests) was interpreted as sufficient. The relative and absolute reliabilities are summarized in Table 4, wherein the interexaminer and intersession reliabilities of the total score were interpreted as sufficient. The mean (SD) of the KOJI AWARENESS total scores were 38.54 (4.65) for examiner rating, and 41.50 (4.87) for self-rating at the first session; and 39.20 (5.42) for examiner rating, and 42.66 (5.03) for self-rating at the second session, respectively. With respect to the overall score, there were neither ceiling nor floor effects that exceeded 15% of the response frequency [18–20]. Bland–Altman plots are presented in Fig 1, where the 95% CIs of the value did not include 0. Thus, fixed bias was identified in each session, indicating lower total scores in external examiner rating than those in self-rating. However, no proportional bias was identified (r = −.07 [95% CIs: −.34 to .21], P = .639 in the first session; r = .14 [−.14 to .40], P = .329 in the second session).

thumbnail
Download:
Fig 1. Bland-Altman plots for comparison of the external examiner rating than self-rating.

X axis: Mean of the two different rating scores. Y axis: Difference from the external examiner rating scores to self-rating scores. Solid line: Mean difference between the two different rating scores (). Dash lines: Upper and lower 95% confidence intervals of the value. (A) 1st session. (B) 2nd session.

https://doi.org/10.1371/journal.pone.0308973.g001

thumbnail
Download:
Table 3. Probability adjusted kappa values for interexaminer and intersession reliability in each test.

https://doi.org/10.1371/journal.pone.0308973.t003

thumbnail
Download:
Table 4. Results of interexaminer and intersession reliability in total scores.

https://doi.org/10.1371/journal.pone.0308973.t004

Discussion

This study evaluated the test–retest reliability of KOJI AWARENESS between external examiner rating scores and self-rating scores. Murohushi et al. [4] reported that the self-rating total score of the KOJI AWARENESS had acceptable test-retest reliability, but the details of the method had not been clarified. To the best of the authors’ knowledge, this study is the first detained verification of the reliability of KOJI AWARENESS.

For the total KOJI AWARENESS score, the ICC exceeded .7 for both the external examiner rating and self-rating scores. Furthermore, both MDCs were comparable at approximately 10% of the total score. Previous studies interpreted an MDC of < 10% as excellent reliability and an MDC of 10%–30% as acceptable reliability [21,22]. Therefore, the total score of the KOJI AWARENESS is considered reliable enough for both the external examiner rating and self-rating scores.

This study revealed a fixed bias in the total score of the KOJI AWARENESS between the external examiner and self-rating scores. For example, a movement that the participant perceived as not moving might be moving from the perspective of an external examiner, or a movement that the participant perceived as being completed up to a certain angle might not be sufficient from the perspective of an external examiner. Thus, the external examiner scored more strictly than the self-rating scores. However, the 95% CIs of the value were below the MDC of test–retest reliability for both the first and second sessions. Therefore, the overall KOJI AWARENESS score, whether scored by an external examiner or by self-rating, is not expected to affect clinical interpretation. The criterion-related validity of the KOJI AWARENESS is already known as its self-rating total score has been confirmed to correlate with the total score of the FMS [2]. Therefore, the findings of this study reveal the clinical utility of KOJI AWARENESS by allowing the assessment of one’s general overview of his or her whole-body movement ability with a total score of KOJI AWARENESS.

A detailed analysis of the individual tests and domain scores showed that the neck and hip mobility domains had insufficient reliability. Furthermore, the interexaminer reliability was insufficient for both the first and second sessions in the scapula mobility domain, despite sufficient intersession reliability for the external examiner rating and self-rating. Thus, these three tests are considered to present with discrepancies in judgment between external examiner rating and self-rating. These tests may need additional or revised KOJI AWARENESS scoring criteria and/or instructions. In particular, as the sensation that the cervical spine or trunk is immobile or not vertical may be altered by symptoms [23,24], it would be prudent to consider how rigorous this criterion should be.

Limitations

There are several limitations in this study. The first limitation relates to the selection of the participants. The majority of participants in this study were healthy university students in their early twenties. This study identified a trend that the external examiner scored more strictly than the self-rating scores, which may have affected the results of this study if the target population had been the general public, who had less interest in and understanding of movement than the university students who participated in this study. This study did not include in-depth interviews about pain. Although some of the participants in this study experienced pain by chance, the authors believe that it was not a limitation that seriously affected the results of this study because the degree of pain was low and there was no subjective functional limitation. On the other hand, pain with certain intensity can influence movement patterns [25–27]. Certain interventions can also change movement patterns [4,25]. Therefore, the test–retest reliability of the KOJI AWARENESS may be lower in those who have substantial changes in pain status and those who receive certain interventions to correct movement patters. Moreover, the reliability may be lower than that observed in the present study in individuals with suspected cognitive decline. To widely disseminate KOJI AWARENESS to the public in the future, verifying the test–retest reliability for individuals with various characteristics is necessary, and this study is valuable in providing a foundation for future considerations of such a study. The second limitation of the study was that there was only one external examiner and he was an undergraduate student studying physical therapy. Although this student had taken the SFMA Level 1 training and may have been more capable of seeing movement than other students, the results may have been different from this study if several movement assessment experts had served as external examiners. However, the clinically important aspect of this study is the test-retest reliability of the self-ratings, and the authors believe that the limitations of this study do not diminish its value. The third limitation is that we did not collect information on participants’ dominant hand and dominant foot. In this study, test-retest reliability in each test was analyzed separately for left and right sides, but further findings could have been obtained by analyzing the results by dominance sides.

Conclusion

This study revealed that KOJI AWARENESS has sufficient intersession reliability among relatively young and healthy participants. The external examiner rating tended to show lower total scores than self-rating; however, the bias seemed not clinically significant.

References

  1. 1. Hodges PW. Pain and motor control: From the laboratory to rehabilitation. J Electromyogr Kinesiol. 2011;21(2): 220–228. pmid:21306915
  2. 2. Murofushi K, Yamaguchi D, Katagiri H, Hirohata K, Furuya H, et al. Validity of the KOJI AWARENESS self-screening test for body movement and comparison with functional movement screening. PLoS One. 2022;17(12): e0277167. pmid:36584031
  3. 3. Moran RW, Schneiders AG, Major KM, Sullivan SJ. How reliable are Functional Movement Screening scores? A systematic review of rater reliability. Br J Sports Med. 2016;50(9): 527–536. pmid:26316583
  4. 4. Murofushi K, Yamaguchi D, Kaneoka K, Oshikawa T, Katagiri H, et al. The effectiveness of corrective exercises on the KOJI AWARENESS score and activity-related pain intensity. J Med Invest. 2023;70(1.2): 208–212. pmid:37164722
  5. 5. Murofushi K, Yamaguchi D, Katagiri H, Hirohata K, Furuya H, et al. The relationship between movement self-screening scores and pain intensity during daily training. J Med Invest. 2022;69(3.4): 204–216. pmid:36244771
  6. 6. Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, et al. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Qual Life Res. 2018;27(5): 1171–1179. pmid:29260445
  7. 7. van Hout B, Janssen MF, Feng Y-S, Kohlmann T, Busschbach J, et al. Interim Scoring for the EQ-5D-5L: Mapping the EQ-5D-5L to EQ-5D-3L Value Sets. Value Health. 2012;15(5): 708–715. pmid:22867780
  8. 8. Craig CL, Marshall AL, Sjostrom M, Bauman AE, Booth ML, et al. International physical activity questionnaire: 12-country reliability and validity. Med Sci Sports Exerc. 2003;35(8): 1381–1395. pmid:12900694
  9. 9. Murase N, Katsumura T, Ueda C, Inoue S, Shimomitsu T. Validity and reliability of Japanese version of International Physical Activity Questionnaire. Journal of Health and Welfare Statistics [In Japanese]. 2002;49(11): 1–9.
  10. 10. Feng YS, Kohlmann T, Janssen MF, Buchholz I. Psychometric properties of the EQ-5D-5L: a systematic review of the literature. Qual Life Res. 2021;30(3): 647–673. pmid:33284428
  11. 11. Shiroiwa T, Ikeda S, Noto S, Igarashi A, Fukuda T, et al. Comparison of Value Set Based on DCE and/or TTO Data: Scoring for EQ-5D-5L Health States in Japan. Value Health. 2016;19(5): 648–654. pmid:27565282
  12. 12. Kim Y, Park I, Kang M. Convergent validity of the international physical activity questionnaire (IPAQ): meta-analysis. Public Health Nutr. 2013;16(3): 440–452. pmid:22874087
  13. 13. The IPAQ group. Guidelines for Data Processing and Analysis of the International Physical Activity Questionnaire (IPAQ)–Short and Long Forms. 2005; Available from: https://sites.google.com/site/theipaq/home.
  14. 14. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2): 155–163. pmid:27330520
  15. 15. Mokkink LB, Boers M, van der Vleuten CPM, Bouter LM, Alonso J, et al. COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study. BMC Med Res Methodol. 2020;20(1): 293. pmid:33267819
  16. 16. Bland JM, Altman DG. Measuring agreement in method comparison studies. Statistical Methods in Medical Research. 1999;8(2): 135–160. pmid:10501650
  17. 17. Vannest KJ, Parker RI, Gonen O, Adiguzel T (2016) Single Case Research: Web Based Calculators for SCR Analysis (Version2.0). TX, USA [Web-based application]: College Station, Texas A&M University
    • 18. Takasaki H, Kawazoe S, Miki T, Chiba H, Godfrey E. Development and validity assessment of a Japanese version of the Exercise Adherence Rating Scale in participants with musculoskeletal disorders. Health Qual Life Outcomes. 2021;19(1): 169. pmid:34167544
    • 19. Takasaki H, Treleaven J. Construct validity and test-retest reliability of the Fatigue Severity Scale in people with chronic neck pain. Arch Phys Med Rehabil. 2013;94(7): 1328–1334. pmid:23274223
    • 20. Takasaki H, Chien CW, Johnston V, Treleaven J, Jull G. Validity and reliability of the perceived deficit questionnaire to assess cognitive symptoms in people with chronic whiplash-associated disorders. Arch Phys Med Rehabil. 2012;93(10): 1774–1781. pmid:22634231
    • 21. Huang S-L, Hsieh C-L, Wu R-M, Tai C-H, Lin C-H, et al. Minimal Detectable Change of the Timed “Up & Go” Test and the Dynamic Gait Index in People With Parkinson Disease. Phys Ther. 2011;91(1): 114–121. pmid:20947672
    • 22. Chen CH, Lin SF, Yu WH, Lin JH, Chen HL, et al. Comparison of the test-retest reliability of the balance computerized adaptive test and a computerized posturography instrument in patients with stroke. Arch Phys Med Rehabil. 2014;95(8): 1477–1483. pmid:24662809
    • 23. Treleaven J, Takasaki H. High variability of the subjective visual vertical test of vertical perception, in some people with neck pain–Should this be a standard measure of cervical proprioception? Man Ther. 2015;20(1): 183–188. pmid:25241660
    • 24. Bagust J, Docherty S, Abdul Razzak R. Re: “High variability of the subjective visual vertical test of vertical perception, in some people with neck pain–Should this be a standard measure of cervical proprioception?”. Man Ther. 2015;20(6): e18. pmid:26453504
    • 25. Takasaki H, Herbowy S. Immediate improvement in the cranio-cervical flexion test associated with MDT-based interventions: a case report. J Man Manip Ther. 2016;24(5): 285–292. pmid:27956822
    • 26. Saito H, Watanabe Y, Kutsuna T, Futohashi T, Kusumoto Y, et al. Spinal movement variability associated with low back pain: A scoping review. PLoS One. 2021;16(5): e0252141. pmid:34029347
    • 27. Hodges PW, Tucker K. Moving differently in pain: A new theory to explain the adaptation to pain. Pain. 2011;152(3, Supplement): S90–S98. pmid:21087823