Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Feasibility and clinical usefulness of deep learning-accelerated MRI for acute painful fracture patients wearing a splint: A prospective comparative study

  • Seunghyeon Roh ,

    Contributed equally to this work with: Seunghyeon Roh, Jae In Park

    Roles Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

  • Jae In Park ,

    Contributed equally to this work with: Seunghyeon Roh, Jae In Park

    Roles Data curation, Methodology, Visualization, Writing – original draft, Writing – review & editing

    Affiliation Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea

  • Gun Young Kim,

    Roles Data curation, Investigation, Methodology, Writing – review & editing

    Affiliation Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea

  • Hye Jin Yoo,

    Roles Formal analysis, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

  • Dominik Nickel,

    Roles Software, Writing – review & editing

    Affiliation Siemens Healthcare GmbH, Erlangen, Germany

  • Gregor Koerzdoerfer,

    Roles Conceptualization, Software, Writing – review & editing

    Affiliation Siemens Healthcare GmbH, Erlangen, Germany

  • JaeKon Sung,

    Roles Software, Writing – review & editing

    Affiliation Siemens Healthineers Ltd, Seoul, Republic of Korea

  • Jiseon Oh,

    Roles Conceptualization, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

  • Hee Dong Chae,

    Roles Conceptualization, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

  • Sung Hwan Hong,

    Roles Conceptualization, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

  • Ja-Young Choi

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    drchoi01@snu.ac.kr

    Affiliations Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea, Department of Radiology, Seoul National University College of Medicine, Seoul, Republic of Korea

Abstract

Objective

To evaluate the feasibility and clinical usefulness of deep learning (DL)-accelerated turbo spin echo (TSEDL) sequences relative to standard TSE sequences (TSES) for acute radius fracture patients wearing a splint.

Methods

This prospective consecutive study investigated 50 patients’ preoperative wrist MRI scans acquired between July 2021 and January 2022. Examinations were performed at 3 Tesla MRI with body array coils due to the wrist splint. Besides TSES obtained according to the routine protocol, TSEDL sequences for axial T2-, coronal T1-, and coronal PD-weighted TSE sequences were scanned for comparison. For quantitative assessment, the relative signal-to-noise ratio (rSNR), the relative contrast-to-noise ratio (rCNR), and the relative contrast ratio (rCR) were measured. For qualitative assessment, all images were assessed by two independent musculoskeletal radiologists in terms of perceived SNR, image contrast, image sharpness, artifacts disturbing evaluation, overall image quality and diagnostic confidence for injuries using a four- or five-point Likert scale.

Results

The scan time was shortened approximately by a factor of two for TSEDL compared to TSES. TSEDL images showed significantly better rSNR, rCNR, and rCR values for all sequences, and scored significantly better in terms of both image quality and diagnostic confidence for both readers than TSES images (all p < .05). Interrater reliabilities were in almost perfect agreement.

Conclusion

The DL-accelerated technique proved to be very helpful not only to reduce scan time but also to improve image quality for acute painful fracture patients wearing a splint despite using body array coils instead of a wrist-specific coil. Based on our study, the DL-accelerated technique can be very useful for MRI of any part of the extremities in trauma settings just with body array coils.

Introduction

The distal radius is one of the most common sites of fracture, and the incidence of distal radius fracture is increasing among people in all age groups. It accounts for approximately 18% of fractures in patients 65 years and older, and it hinders one’s ability to perform daily activities, such as preparing meals and performing housekeeping duties [13].

Furthermore, associated soft tissue injuries (e.g., intrinsic scapholunate and lunotriquetral ligament, triangular fibrocartilage with or without concomitant distal radioulnar joint instability) are reported with high incidence [3]; MRI evaluation is crucial for the detection of these soft tissue injuries [4]. Additionally, MRI is used to identify potential radiographically occult fractures, such as associated occult scaphoid fractures [5]. More than 60% of patients diagnosed with distal radius fracture undergo surgical correction [6]. In the case of acute wrist fracture patients in need of surgical correction, preoperative MRI scans might have an important role in the evaluation of associated soft tissue injury and surgical planning.

When patients with acute distal radius fracture need to undergo preoperative MRI, they are prone to suboptimal MRI quality for the following reasons. These patients tend to undergo manual reduction and splinting as soon as possible after diagnosis to prevent further injury. However, the splints are not easily detachable, and with the wrist splint on, wrist-specific coils are not suitable for MRI scans, causing impaired image quality and subsequent inaccurate diagnosis for soft tissue injuries. Additionally, patients are in such pain that it is difficult for them to remain still for the long time required for an MRI scan, which makes them susceptible to motion artifacts. Over the years, various attempts have been made to reduce the scan time, such as parallel imaging [79] and compressed sensing [1012]. More recently, deep learning (DL)-based reconstruction and acceleration techniques have been developed. With concurrent denoising, DL-based reconstruction and acceleration techniques have achieved some promising results in reducing scan time and improving image quality simultaneously for various body parts, including the prostate [13], pituitary gland [14], liver [15], pelvis [16], shoulder and hip joints [17]. A preemptive study about the feasibility of implementing DL reconstruction in musculoskeletal turbo spin echo (TSE) imaging has been conducted with positive results [18]. To the best of our knowledge, no previous studies have applied deep learning-accelerated MRI in a painful trauma clinical setting. It was hypothesized in this study that with this DL approach, high image quality wrist MRI scans can be acquired and a reduction in scan time can be achieved simultaneously, when utilizing body array coils instead of a wrist-specific coil.

The purpose of this study was to evaluate the feasibility and clinical usefulness of deep learning-accelerated turbo spin echo (TSEDL) sequences relative to standard TSE (TSES) sequences for acute distal radius fracture patients wearing a splint on the wrist.

Materials and methods

This prospective study was approved by the institutional review board of Seoul National University Hospital, and written informed consent was obtained from all subjects before inclusion in the study (IRB No. H-2105-079-1218).

Study population

Fifty-three consecutive consenting patients with acute distal radius fracture diagnosed based on wrist CT scan findings and who were scheduled for preoperative wrist MRI were prospectively included in this study between July 2021 and January 2022. Three patients were excluded for the following reasons: One had a scaphoid fracture with screw fixation in the affected wrist. Another two were in so much pain that they had to end the scanning process as fast as possible. Finally, 50 patients (41 female and 9 male patients; mean age ± SD, 64.68 ± 10.83 years; age range 31–86 years) were included in this study.

Image acquisition

All MRI examinations were performed using a 3 Tesla MRI scanner (MAGNETOM Skyrafit, Siemens Healthcare) with a pair of 30-channel body array coils (Body 30, Siemens Healthcare) instead of a wrist-specific coil. All study participating patients who underwent sugar-tong splint immobilization for distal radius fracture (anteroposterior compression after partial closure reduction via traction and release to prevent joint dislocation; the distal end of the splint was not to cross the distal ends of the metacarpal bones) were scanned in supine position with the hand above the head (Fig 1). First, TSES sequences were obtained according to the routine MRI protocol for distal radius fractures in our institute, which was as follows: axial T2-weighted TSES sequence with fat suppression (FS), axial T1-weighted TSES sequence, coronal T1-weighted TSES sequences, coronal T2-weighted and PD-weighted TSES sequences with FS, and sagittal T2-weighted TSES sequences. In addition, axial T2-weighted TSEDL sequence with FS, coronal T1-weighted TSEDL sequence, and coronal PD-weighted TSEDL sequence with FS were acquired for comparison. The acquisition parameters for each sequence are summarized in Table 1.

thumbnail
Fig 1. Patient’s scan position with a pair of 30-channel body array coils.

(A) The patient with sugar-tong splint immobilization in the supine position, (B) and a pair of 30-channel body array coils in position.

https://doi.org/10.1371/journal.pone.0287903.g001

The prototypical TSEDL sequence employs a deep-learning reconstruction which is designed for improving the signal-to-noise ratio of acquisitions with higher accelerations. It comprises an unrolled variational network [19] and we refer to references [13, 18] for more details on the technical implementation used in this study.

Quantitative image analysis

For objective comparison of image quality, the relative signal-to-noise ratio (rSNR) for bone and muscle, relative contrast-to-noise ratio (rCNR), and relative contrast ratio (rCR) between bone and muscle were measured on TSEDL and TSES images by a 2nd year radiology resident (S.H.R). For the corresponding TSEDL and TSES images of each sequence, the same sized circular ROIs with a diameter of 5 mm were placed in the same location. For axial images, the thenar muscle and 2nd metacarpal base of the same plane were selected for muscle and bone analysis measurements (Fig 2A). For six patients whose 2nd metacarpal base was not covered in the axial scan, the flexor digitorum muscle and distal radius or ulna of the same plane were selected instead. For coronal images, the 1st interosseous muscle and 2nd metacarpal base of the same image plane were selected (Fig 2B). Circular ROIs were placed in homogeneous areas away from other structures, such as vessels, edema or cysts. Regions affected with partial volume averaging were avoided as well. We measured signal intensity (SI) values and standard deviation (SD) within the ROI for comparison analysis. With those values, rSNR for bone and muscle, rCNR and rCR between bone and muscle were calculated using the following expressions [14, 17]:

thumbnail
Fig 2. ROI placement.

(A) A 5 mm circular ROI was placed at the thenar muscle and the 2nd metacarpal base in axial fat-suppressed image. (B) For coronal images, the 1st interosseous muscle and 2nd metacarpal base of the same image plane were selected.

https://doi.org/10.1371/journal.pone.0287903.g002

Qualitative image analysis

All images were independently evaluated by two independent board-certified musculoskeletal radiologists (H.J.Y. with 17 years of experience; J.Y.C., with 23 years of experience). For each case, two image sets of TSEDL and TSES sequences were anonymized and distributed in random order. The readers were blinded to the clinical information, radiological report, and scan parameters. For each image set, image quality parameters, including the perceived signal-to-noise ratio, image contrast, image sharpness, artifacts (motion, grid) disturbing evaluation, and overall image quality, were evaluated using a 4-point Likert scale (Table 2). Grid artifact was defined as a pattern of coarse lines that cross each other to form squares on the images (Fig 3). In addition, diagnostic confidence levels were assessed in terms of the presence of a distal radius fracture, ulnar styloid fracture/bone contusion, and triangular fibrocartilaginous complex (TFCC) injury. Diagnostic confidence for each abnormality was measured based on a 5-point Likert scale. Detailed information on the scale used for qualitative evaluation of image quality and diagnostic confidence is provided in Table 2.

thumbnail
Fig 3. Grid artifacts.

An 80-year-old female with a distal radius fracture. Coronal PD-weighted TSEDL image with fat suppression showed grid artifact (arrows), which was defined as a pattern of lines that cross each other to form squares on the image.

https://doi.org/10.1371/journal.pone.0287903.g003

thumbnail
Table 2. Scoring scale for various parameters in qualitative image analysis.

https://doi.org/10.1371/journal.pone.0287903.t002

Statistical analysis

For comparison of the rSNR, rCNR, and rCR between TSEDL and TSES sequences, the paired t test was used. For qualitative image analysis and diagnostic confidence, the Wilcoxon signed rank test was utilized. A p value of less than .05 was considered to indicate a significant difference. All statistical analyses were performed by using commercially dedicated software (IBM SPSS Statistics 27 software for Windows; IBM). The levels of interrater agreement were evaluated using Gwet’s agreement coefficient (AC) [20] due to the skewed marginal distribution of qualitative scores [21]. Gwet’s ACs were calculated using STATA/MP 17 for Windows; StataCorp LLC [22]. Interrater reliability was categorized as poor (<0), slight (0–0.2), fair (0.21–0.4), moderate (0.41–0.6), substantial (0.61–0.8), or almost perfect agreement (0.8–1) [23].

Results

The total scan time for axial T2-, coronal T1-, and coronal PD-weighted TSE sequences was 11 minutes and 55 seconds for TSES, and 6 minutes and 6 seconds for TSEDL (Table 1).

Quantitative image quality assessment

TSEDL images showed significantly better (p < .05) bone rSNR and muscle rSNR for all sequences than TSES images; axial T2-, coronal T1-, and coronal PD-weighted TSE (Table 3). TSEDL images showed significantly better (p < .05) rCNR and rCR between bone and muscle for all sequences as well (Table 3). See Figs 46 for example images.

thumbnail
Fig 4. A 57-year-old female with a distal radius fracture.

(A) Axial T2-weighted TSES image with fat suppression and (B) the corresponding TSEDL image. TSEDL images showed superior rSNR, rCNR and rCR compared to TSES images.

https://doi.org/10.1371/journal.pone.0287903.g004

thumbnail
Fig 5. A 57-year-old female with a distal radius fracture.

(A) Coronal T1-weighted TSES image and (B) the corresponding TSEDL image. TSEDL images showed superior rSNR, rCNR and rCR compared to TSES images.

https://doi.org/10.1371/journal.pone.0287903.g005

thumbnail
Fig 6. A 57-year-old female with a distal radius fracture.

(A) PD-weighted TSES image with fat suppression and (B) the corresponding TSEDL image. TSEDL images showed superior rSNR, rCNR and rCR compared to TSES images.

https://doi.org/10.1371/journal.pone.0287903.g006

thumbnail
Table 3. Quantitative assessment of image quality between TSEDL and TSES sequences.

https://doi.org/10.1371/journal.pone.0287903.t003

Qualitative evaluation of image quality and diagnostic confidence

TSEDL sequences demonstrated significantly better image quality than TSES sequences for both readers (p < .05) in terms of perceived SNR, image contrast, image sharpness, motion artifacts and overall image quality (Table 4, Fig 7). Regarding diagnostic confidence, TSEDL sequences showed significantly higher confidence levels for all distal radius fractures, ulnar styloid fractures (Fig 8), and TFCC injuries (Fig 9) (p < .001 in all) for both readers except one reader’s TFCC injury evaluation, which was due to both TSEDL sequences and TSES sequences scoring the highest point for all patients. Interrater reliabilities showed almost perfect agreement for qualitative evaluation of image quality and diagnostic confidence except in the following cases: distal radius fracture and ulnar styloid fracture/bone contusion diagnostic confidence for TSES sequences (moderate) and image sharpness for TSES sequence (fair) (Table 4).

thumbnail
Fig 7. Motion artifacts.

(A) Coronal T1-weighted TSES image of a 70-year-old female patient revealed severe margin blurring of the bone and joint by motion artifact (arrowheads). (B) In contrast, sharp-edged anatomic details were noted without motion artifacts in the corresponding TSEDL image.

https://doi.org/10.1371/journal.pone.0287903.g007

thumbnail
Fig 8. Associated ulnar styloid process fracture.

(A) Coronal T1-weighted TSES image of a 60-year-old female with a distal radius fracture demonstrated suspected fragmentation of the ulnar styloid process with indistinct cortices. (B) However, a small bony fragment with distinct cortices and internal fatty marrow was well delineated on the corresponding TSEDL image. (C) Coronal CT image clearly showed ulnar styloid process fracture with bone fragments.

https://doi.org/10.1371/journal.pone.0287903.g008

thumbnail
Fig 9. Associated triangular fibrocartilage complex injury.

(A) On coronal fat-suppressed PD-weighted TSES image of an 86-year-old female with distal radius and ulnar fractures, wavy radial attachment (arrow) of the disc was seen with blurred margin and internal signal alteration. (B) In contrast, the TSEDL image showed a sharp margin of the disc with localized signal alteration of the radial attachment (arrow). Nearby lunate articular cartilage and ulnar cortical fragments were clearly delineated as well.

https://doi.org/10.1371/journal.pone.0287903.g009

thumbnail
Table 4. Qualitative assessment of image quality between TSEDL and TSES sequences.

https://doi.org/10.1371/journal.pone.0287903.t004

Discussion

In the current study, TSEDL images showed significantly better rSNR, rCNR, and rCR than TSES images for all sequences for patients with acute painful distal radius fracture wearing a splint. For qualitative analysis, TSEDL sequences were significantly better than TSES sequences in terms of both image qualities and diagnostic confidence.

It is inspiring that the TSEDL sequences allow increased spatial resolution without image quality deterioration, while scan time is reduced by approximately half of that required for TSES sequences. It is difficult for patients with acute fracture to tolerate the long scan time due to pain even without motion. Therefore, we think that the longer it takes to acquire images, the more susceptible images are to motion artifacts (Fig 7). These artifacts may degrade image quality and subsequently lead to decreased diagnostic accuracy, thus incurring expensive costs for the patient and the institution due to potential exam failure or repeat exam. Our results showed that motion artifacts were less frequent in TSEDL images than in TSES images. It goes without saying that reduced scan time itself is a considerable advantage for the patients and the institution in terms of convenience and cost-effectiveness.

Furthermore, in the present study, despite two unfavorable conditions for wrist MR scans, 1) using body array coils instead of a wrist-specific coil and 2) keeping the splint on at scan time (the culprit for off-center scanning or foreign body artifacts), TSEDL images demonstrated excellent image quality. Based on our study, the TSEDL technique can be very useful for MRI of any part of the extremities in trauma settings, such as fractures with splints or cast immobilization just with body array coils.

A disadvantage of TSEDL images compared with TSES images was the minor ‘grid’ artifact that was sometimes observed (Fig 3). This minor byproduct hardly altered the diagnostic performance of the images, but it would be a candidate for future improvement.

There were several limitations in this study. First, although both quantitative and qualitative analysis generated congruent results, in favor of TSEDL images over TSES images, there was no direct comparison between TSEDL images and the hypothetical images with splint removal and wrist-specific coil application. However, it was difficult to remove the splint for a while during the MRI scan for comparison in daily practice, which might cause patient discomfort and failure of mechanical reduction. Second, this study investigated a vendor provided preset denoising level for DL-based reconstruction. However, appropriate levels of denoising could be further studied [14, 17], as too much denoising could impair the edge margins of distinct structures and impair image quality [13].

In conclusion, the deep learning-accelerated technique proved to be very helpful not only to reduce scan time but also to improve image quality simultaneously for patients with acute painful distal radius fracture wearing a splint on the wrist despite using body array coils instead of a wrist-specific coil. Based on our study, the deep-learning accelerated technique has potential for being applicable to MRI for any part of the extremities in trauma settings just with body array coils.

References

  1. 1. Goldfarb CA, Yin Y, Gilula LA, Fisher AJ, Boyer MI. Wrist fractures: what the clinician wants to know. Radiology. 2001;219(1):11–28. pmid:11274530.
  2. 2. Porrino JA Jr., Maloney E, Scherer K, Mulcahy H, Ha AS, Allan C. Fracture of the distal radius: epidemiology and premanagement radiographic characterization. AJR Am J Roentgenol. 2014;203(3):551–9. pmid:25148157.
  3. 3. Shapiro LM, Kamal RN, Management of Distal Radius Fractures Work G, Nonvoting Clinical C, Nonvoting Oversight C, Staff of the American Academy of Orthopaedic S, et al. Distal Radius Fracture Clinical Practice Guidelines-Updates and Clinical Implications. J Hand Surg Am. 2021;46(9):807–11. Epub 20210809. pmid:34384642.
  4. 4. Spence LD, Savenor A, Nwachuku I, Tilsley J, Eustace S. MRI of fractures of the distal radius: comparison with conventional radiographs. Skeletal Radiol. 1998;27(5):244–9. pmid:9638833.
  5. 5. De Jonge MC, Assink J, Vanhoenacker FM. Acute Bony Injuries of Hand and Wrist. Semin Musculoskelet Radiol. 2021;25(2):277–93. Epub 20210809. pmid:34374063.
  6. 6. Sander AL, Leiblein M, Sommer K, Marzi I, Schneidmuller D, Frank J. Epidemiology and treatment of distal radius fractures: current concept based on fracture severity and not on age. Eur J Trauma Emerg Surg. 2020;46(3):585–90. Epub 20181001. pmid:30276724.
  7. 7. Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang J, et al. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn Reson Med. 2002;47(6):1202–10. pmid:12111967.
  8. 8. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sensitivity encoding for fast MRI. Magn Reson Med. 1999;42(5):952–62. pmid:10542355.
  9. 9. Sodickson DK, Manning WJ. Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays. Magn Reson Med. 1997;38(4):591–603. pmid:9324327.
  10. 10. Donoho DL. Compressed sensing. IEEE Transactions on Information Theory. 2006;52(4):1289–306.
  11. 11. Feng L, Benkert T, Block KT, Sodickson DK, Otazo R, Chandarana H. Compressed sensing for body MRI. J Magn Reson Imaging. 2017;45(4):966–87. Epub 20161216. pmid:27981664; PubMed Central PMCID: PMC5352490.
  12. 12. Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med. 2007;58(6):1182–95. pmid:17969013.
  13. 13. Gassenmaier S, Afat S, Nickel D, Mostapha M, Herrmann J, Othman AE. Deep learning-accelerated T2-weighted imaging of the prostate: Reduction of acquisition time and improvement of image quality. Eur J Radiol. 2021;137:109600. Epub 20210215. pmid:33610853.
  14. 14. Uetani H, Nakaura T, Kitajima M, Morita K, Haraoka K, Shinojima N, et al. Hybrid deep-learning-based denoising method for compressed sensing in pituitary MRI: comparison with the conventional wavelet-based denoising method. Eur Radiol. 2022. Epub 20220215. pmid:35169896.
  15. 15. Tanabe M, Higashi M, Yonezawa T, Yamaguchi T, Iida E, Furukawa M, et al. Feasibility of high-resolution magnetic resonance imaging of the liver using deep learning reconstruction based on the deep learning denoising technique. Magn Reson Imaging. 2021;80:121–6. Epub 20210507. pmid:33971240.
  16. 16. Ueda T, Ohno Y, Yamamoto K, Iwase A, Fukuba T, Hanamatsu S, et al. Compressed sensing and deep learning reconstruction for women’s pelvic MRI denoising: Utility for improving image quality and examination time in routine clinical practice. Eur J Radiol. 2021;134:109430. Epub 20201121. pmid:33276249.
  17. 17. Koch KM, Sherafati M, Arpinar VE, Bhave S, Ausman R, Nencka AS, et al. Analysis and Evaluation of a Deep Learning Reconstruction Approach with Denoising for Orthopedic MRI. Radiology: Artificial Intelligence. 2021;3(6):e200278. pmid:34870214
  18. 18. Herrmann J, Koerzdoerfer G, Nickel D, Mostapha M, Nadar M, Gassenmaier S, et al. Feasibility and Implementation of a Deep Learning MR Reconstruction for TSE Sequences in Musculoskeletal Imaging. Diagnostics (Basel). 2021;11(8). Epub 20210816. pmid:34441418; PubMed Central PMCID: PMC8394583.
  19. 19. Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, et al. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med. 2018;79(6):3055–71. Epub 20171108. pmid:29115689; PubMed Central PMCID: PMC5902683.
  20. 20. Gwet KL. Computing inter‐rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology. 2008;61(1):29–48. pmid:18482474
  21. 21. Thoenen J, Stevens KJ, Turmezei TD, Chaudhari A, Watkins LE, McWalter EJ, et al. Non-contrast MRI of synovitis in the knee using quantitative DESS. Eur Radiol. 2021;31(12):9369–79. Epub 20210515. pmid:33993332; PubMed Central PMCID: PMC8591148.
  22. 22. Klein D. Implementing a general framework for assessing interrater agreement in Stata. The Stata Journal. 2018;18(4):871–901.
  23. 23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. biometrics. 1977:159–74. pmid:843571