https://orcid.org/0000-0001-5511-3780
Figures
Abstract
Over the last 30 years, there has been a growing trend in clinical trials towards assessing novel interventions not only against the benchmark of statistical significance, but also with respect to whether they lead to clinically meaningful changes for patients. In the context of Disorders of Consciousness (DOC), despite a growing landscape of experimental interventions, there is no agreed standard as to what counts as a minimal clinically important difference (MCID). In part, this issue springs from the fact that, by definition, DOC patients are either unresponsive (i.e., in a Vegetative State; VS) or non-communicative (i.e., in a Minimally Conscious State; MCS), which renders it impossible to assess any subjective perception of benefit, one of the two core aspects of MCIDs. Here, we develop a novel approach that leverages published, international diagnostic guidelines to establish a probability-based minimal clinically important difference (pMCID), and we apply it to the most validated and frequently used scale in DOC: the Coma Recovery Scale–Revised (CRS-R). This novel method is objective (i.e., based on published criteria for patient diagnosis) and easy to recalculate as the field refines its agreed-upon criteria for diagnosis. We believe this new approach can help clinicians determine whether observed changes in patients’ behavior are clinically important, even when patients cannot communicate their experiences, and can align the landscape of clinical trials in DOC with the practices in other medical fields.
Citation: Monti MM, Spivak NM, Edlow BL, Bodien YG (2023) What is a minimal clinically important difference for clinical trials in patients with disorders of consciousness? a novel probabilistic approach. PLoS ONE 18(8): e0290290. https://doi.org/10.1371/journal.pone.0290290
Editor: Soojin Park, Columbia University, UNITED STATES
Received: April 13, 2023; Accepted: August 3, 2023; Published: August 24, 2023
Copyright: © 2023 Monti et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The code for calculating pMCID (pMCID_inDOC.m), including the code necessary to reproduce the results shown in Tables 2, 3, and 4, is freely available from the corresponding author (MMM) and at the following OSF link: https://doi.org/10.17605/OSF.IO/D2FYM.
Funding: MMM Funding: Tiny Blue Dot Foundation, NIH National Institute of General Medical Sciences (5R01GM135420). NMS Funding: NIH National Institute of General Medical Sciences (5T32GM008042). BLE Funding: NIH Director’s Office (DP2HD101400), NIH National Institute of Neurological Disorders and Stroke (R21NS109627, RF1NS115268), James S. McDonnell Foundation, Tiny Blue Dot Foundation, and Chen Institute MGH Research Scholar Award. YB Funding: NIDILRR 90DPTB0011-01-00, Department of Defense (W81XWH-14-2-0176), NIH 1DP2HD101400, Tiny Blue Dot Foundation, James S McDonnell Foundation.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In a landscape of ever-increasing sophistication in medical technology, the design of ethical clinical trials requires balancing the risks and benefits of testing novel interventions [1]. Historically, novel investigational interventions are evaluated against the benchmark of statistical significance. That is, they are typically considered successful when an observed difference, across cohorts (e.g., intervention group versus placebo group) or over time (e.g., prior to intervention versus following intervention), is sufficiently large compared to its variability. While this approach is sound with respect to well-established criteria for making inferences based on random samples [2], it is blind as to whether the observed changes are meaningful for patients, families, clinicians, decision-makers, and other stakeholders [3,4]. Over the last thirty years there has been a growing emphasis on evaluating the clinical meaningfulness of the changes associated with investigational interventions alongside, or in lieu of, statistical significance. Indeed, thresholds establishing the minimal effect magnitude associated with a meaningful change (i.e., Minimal Clinically Important Difference; MCID) [5] have now been established for clinical instruments across a broad range of outcomes, including level of patient disability (e..g, Functional Independence Measure [6]), balance (e.g., Berg Balance Scale [7]), depression (e.g., Montgomery–Asberg Depression Rating Scale [8]), motor deficits (e.g., Disability Rating Scale [9,10]) and quality of life (e.g., Heinrichs–Carpenter quality of life scale [11]), among many others.
For patients with disorders of consciousness (DOC) [12,13], despite a burgeoning literature assessing novel therapeutic interventions [1,14,15], there are no accepted MCIDs for the most highly recommended and frequently used instrument [16,17]: the Coma Recovery Scale–Revised (CRS-R) [18]. To date, only one DOC-specific instrument has a published MCID, the Disorders of Consciousness Scale (DOCS-25; [19]). However, a comprehensive review by the American Congress of Rehabilitation Medicine (ACRM) found the CRS-R to be the preferred scale to assess patients with DOC [20]. Consequently, clinical trials in this patient cohort have relied exclusively on statistical significance when assessing the efficacy of pharmacological [21,22], device-based [23–25], and sensory stimulation interventions [26,27].
Establishing MCIDs in patients with DOC is hampered by two main difficulties. First, one of the two core aspects of MCIDs is not accessible in this patient population. Specifically, as originally defined, an MCID is “[t]he smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management” [emphasis added] [5]. By the very nature of the diagnosis, patients with DOC are either unresponsive or responsive but non-communicative [28]. Ipso facto, subjective perceptions of improvement by the patient are either absent (in unconscious patients) or inaccessible (in non-communicative patients). Furthermore, patients who emerge from the minimally conscious state (MCS) and recover basic functional communication often remain in a confusional state, characterized by disorientation and severe impairments in cogntion [29–31], all known limitations in the development of reliable MCIDs [32]. The second core aspect of an MCID, is “[t]he smallest difference […] which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management,” [5] [emphasis added] and can, in principle, be evaluated in non-responsive, non-communicative, and severely disabled patients. Specifically, given the existence of well-defined behaviors that signal a patient’s DOC diagnosis [28], it is possible to operationalize ‘clinically meaningful changes’ in this context as any change–following an intervention–whereby a patient becomes capable of demonstrating novel behaviors that are associated with attaining a diagnostic status implying lesser impairment (i.e., henceforth, a “threshold behavior”). To exemplify, according to the CRS-R protocol [18], when a patient considered to be in a Vegetative State (VS; also referred to as Unresponsive Wakefulness Syndrome; UWS [33]) regains the ability to demonstrate visual fixation, this signals a change in diagnostic status from VS to Minimally Conscious State minus (MCS-). Similarly, when a patient considered to be in a Minimally Conscious State plus (MCS+) regains the ability to demonstrate accurate functional communication or functional object use, this signals a change in diagnostic status from MCS+ to emergence from a Minimally Conscious State (eMCS) [18,28,34,35]. In this proposal, the emergence of such “threshold behaviors,” which imply a change in diagnosis and thus patient management, can be leveraged to identify changes in CRS-R total scores that are clinically meaningful.
The second main difficulty in developing MCIDs for DOC patients is that clinical instruments for this cohort typically describe patient responsiveness in terms of ordinally ranked behaviors, from highest to lowest. The distance between contiguous items on the scale, however, is not designed to be proportional and/or consistent. This characteristic makes it difficult to develop a simple numeric threshold that can be used to unambiguously define clinically meaningful changes across the entirety of the instrument’s diagnostic range. To illustrate, a 1-point improvement at the upper end of the Oromotor/Verbal Function subscale of the CRS-R, from 2 (i.e., Vocalization/Oral Movement) to 3 (i.e., Intelligible Verbalization), seems very different from a 1-point improvement on the lower end of the same subscale from 0 (i.e., No response) to 1 (i.e., Oral Reflexive Movement). For, while the former change implies the return of language comprehension and a state of awareness, also denoting a change in diagnosis from VS to MCS+, the latter scenario simply represents a change from complete unresponsiveness (in this functional domain) to simple reflexive responsiveness. In fact, while some 1-point increases in CRS-R score score can entail a clinically important change, as in the former scenario, there are many instances in which a 1- and even 2-point increase is not associated with any clinically meaningful change. It is thus difficult to find a single minimal change threshold that is reliably associated with a clinically meaningful change.
Given the above, it is difficult to deploy existing procedures for developing MCIDs [36–38] in DOC patients. The distribution-based and the anchor methods, for example, require input from patients regarding their baseline state and any noticeable improvement they might perceive, which, as discussed above, is either absent or inaccessible in DOC patients. The Delphi method is feasible in this context, but relies on subjective clinical opinion and is susceptible to individual biases and that are not unusual in the context of DOC even among experts [39–42]. In what follows, we thus developed a novel, clinical guideline-driven [12], probabilistic approach to establishing MCIDs (pMCID) in DOC. Specifically, we leverage the 2002 Aspen Neurobehavioral Conference Workgroup criteria [28] to identify unambiguous “threshold behaviors” which, when attained, imply a change in patient diagnosis and thus a change in clinical management (i.e., a clinically important change), and then calculate the probability that observing an increase of 1-, 2-, or 3-point change in the total score of the CRS-R is associated with the patient attaining a novel threshold behavior.
Materials and methods
We use a 2-step procedure to developed a theoretically-driven probabilistic approach to establishing an MCIDs for the CRS-R.
Selection of threshold behaviors
As shown in Table 1, of the 29 behaviors that can be scored on the CRS-R, eight represent threshold behaviors. Namely, attaining Localization to Noxious Stimulation or Visual Fixation, index a progression from VS to MCS-; Reproducible Movement to Command, Object Recognition, Intelligible Verbalization, and Intentional (non-functional) communication, index progression to MCS+; and demonstrating Functional Object Use or Accurate (functional) Communication, indexes emergence from a DOC (i.e., eMCS). Threshold behaviors were selected strictly on the basis of accepted published criteria for behviors diagnostic of each DOC (i.e., VS, MCS-, MCS+) [12,13,28,34,43,44]. It should be noted that there are additional behaviors, in the CRS-R, that are consistent with each of the diagnosis above. For example, with respect to the Motor Function Scale, Object Manipulation and Automatic Motor Response are also consistent with an MCS- diagnosis. However, once a patient is capable of Localization to Noxious Stimulation, also demonstrating either of these two additional behaviors does not affect their diagnostic status. In this respect, we only consider threshold behaviors the lowest behavior on each scale that warrants a change in diagnosis.
Threshold behaviors and associated diagnosis are shown in bold.
Calculation of a probabilistic MCID (pMCID)
After having selected the threshold behaviors on the basis of published guidelines, we use simple combinatorics to calculate the probability that any 1-, 2-, and 3-point change in the CRS-R total score is accompanied by the patient exhibiting at least one novel threshold behavior warranting a change in diagnosis. To achieve this, we start by calculating all the combinations of scores that can exist given the structure of the CRS-R scale (i.e., 29 items subdivided into 6 functional domains). In other words, we start by creating a catalog of all conceivable CRS-R configurations. Then, for each of these configurations, we count the fraction of 1-, 2-, and 3-point changes that are associated with meeting at least one new threshold behavior, as a proportion of all the possible 1-, 2-, and 3-point changes that can be attained form that configuration. To exemplify, given an initial CRS-R total score of 6, obtained by scoring 2, 1, 1, 1, 0, 1 on the auditory, visual, motor, oromotor/verbal, communication, and arousal scales respectively, there are six possible ways in which the total score can increase by 1 point (i.e., one per scale). Of these only three are associated with attaining a new threshold behavior (i.e., a 1-point change in the auditory, visual, and communication scale). Thus, given the above initial CRS-R configuration, a 1-point change is associated 50% of the times with a clinically important change. See also Fig 1 for a pictorial illustration of the calculation as applied to 1-point changes. We repeat this process for each conceivable CRS-R configuration and then calculate the overall probability that a 1-, 2-, and 3-point change is associated, for each diagnostic group separately, with an important clinical change. The calculations embed the following four, conservative, stipulations (see the discussion section for an analysis of the impact of these restrictions on the pMCID). First, we do not place any constraint on how the 1-, 2-, and 3-point changes from each initial CRS-R is achieved (e.g., a 3-point change can be obtained by a 3-point increase on a single scale, a 2-point increase in one scale and a 1-point increase on another scale, or a 1-point increase on 3 different scales). Second, we only calculate the probability that each change is associated with at least one threshold behavior. That is, 2- and 3-point changes associated with attaining multiple novel threshold behaviors are treated equally to changes associated with attaining only one novel threshold behavior. Third, changes associated with attaining novel threshold behaviors for the diagnostic group a patient is already part of, or for a lower diagnostic group, are not considered to be clinically important. That is, if a patient is considerd to be MCS+, changes associated with attaining additional threshold behaviors consistent with this diagnosis, as well as changes associated with attaining threshold behaviors consistent with lower diagnoses (e.g., MCS-), are not considered to be clinically important. Finally, our calculations are based on net changes is CRS-R total score. We thus admit, as often seen in the clinic, for scores on individual scales to vary, by up to 3 points, in either direction. That is, a 2-point net change could result by a 3-point increase across two scales and a 1-point decrease on a third scale.
Sample calculation of the average probabilistic minimal clinically important difference (pMCID) for a 1-point change in CRS-R total score across all diagnoses. In brief, the pMCID for a 1-point change is calculated by taking the ratio of the number of 1-point changes that result in a new threshold behavior (numerator) over the number of all possible 1-point changes (denominator). (Note: While not depicted, the calculation also includes “net” 1-point changes; e.g., where a 1-point decrease is observed in one subscale and a 2-point increase is observed on a different subscale; see methods for full detail).
The purely combinatorial approach described above, however, whereby we start by computing all existing configurations of scores across the 6 CRS-R scales, can give rise to impossible score combinations; for example, a CRS-R total score of 6 obtained by attaining Functional Object Use in the Motor Function Scale and 0 on all other scales [45]. We thus repeat the calculation of the proportion of changes associated with attaining novel threshold behaviors for “all existing combinations” as well as for “possible combinations only” (as defined in Ref. [45]). That means, that while in the “all existing combinations” we calculate the pMCID without regard for whether the combination of CRS-R subscores are clinically possible, in the “possible combinations only”, we omit from the pMCID calculus any clinical combination of CRS-R subscores that is manifestly impossible. In addition, as an example of the flexibility and “future compatibility” of the present probabilistic approach, we show how easy it is to recalculate these proportions should diagnostic criteria for DOC change. Specifically, it has been recently proposed that the item “Consistent Movement to Command,” in the auditory function scale of the CRS-R (cf., Table 1), should be considered to be indicative of emergence from MCS (i.e., a threshold behavior for eMCS) as opposed to MCS+ [46,47]. We thus re-calculate the proportion of changes associated with attaining novel threshold behaviors under this new potential guideline (under the “possible combinations only” regime).
Results
When considering all existing combinations, there are 10,080 possible CRS-R configurations. As shown in Table 2, a 1-point change in total CRS-R score is associated with a clinically important change in 81% and 85% of the times, for VS and MCS- patients respectively, and only in 34% of the times for MCS+ patients. In VS and MCS- patients, 2- and 3-point changes are associated with 89% and over 91% chance of a clinically important change, while, in MCS+ patients, 2- and 3-point changes are only associated with a 40% and 47% chance of a clinically important change, respectively.
Out of the 10,080 existing CRS-R configurations, however, according to published data [45], 4,368 are clinically impossible, thus leaving only 5,712 relevant CRS-R configurations. As shown in Table 3, when only considering possible CRS-R configurations, a 1-point change in total score is associated with a clinically important change in 72% and 75% of the times, for VS and MCS- patients respectively, and in 33% of the times for MCS+ patients. In VS and MCS- patients, 2- and 3-point changes are associated with over 80% and 85% chance of a clinically important change, whereas in MCS+ patients 2- and 3-point changes are only associated with a 39% and 46% chance of a clinically important change.
Finally, as shown in Table 4, should international guidelines recognize “Consistent Movement to Command” as a threshold behavior for eMCS [46,47], it would not affect the probabilities reported above for VS and MCS- patients, but it would increase the likelihood of a 1-, 2-, and 3-point change in CRS-R total score being associated with a clinically important change in MCS+ patients to 69%, 72%, and 75% (from 33%, 49%, and 56%), respectively.
Discussion
Given the difficulty in applying established procedures for developing MCIDs [36–38] in DOC patients, we develop a novel probabilistic MCIDs (pMCID) which can be deployed in non-responsive and non-communicative patients and on an ordinal instrument such as the CRS-R. We do so by leveraging well-established diagnostic criteria distinguishing levels of impairment of consciousness [12] in order to calculate the likelihood that any 1-, 2-, or 3-point increase in the CRS-R total score is associated with a clinically meaningful change. Our results show that, under the current guidelines, it might not be appropriate to use a single MCID threshold across the diagnostic spectrum covered by the CRS-R protocol. For example, while a 2-point change in the CRS-R total score is already associated with an over 80% chance of attaining a new threshold behavior for VS and MCS- patients, it is only associated with a 40% chance in MCS+ patients. It is thus not obvious how, in this circumstance, a single MCID threshold could be adopted. The reason for this is that out of the 29 CRS-R items, currently only two are threshold behaviors for eMCS, as compared to 5 and 6 for MCS+ and MCS-, respectively. Second, the data presented here suggest that at least for VS and MCS- a 2-point change is sufficient to return an 80% chance of attaining a clinically meaningful new threshold behavior, which could be a reasonable pMCID. For MCS+ patients, however, current guidelines do not provide a threshold yielding a comparable likeliness of attaining a clinically meaningful change. Nonetheless, should diagnostic guidelines be revised to reclassify Consistent Movement to Command as a threshold behavior for eMCS, as previouslyadvocated [46,47], a 2-to-3 point change in total CRS-R would be associated with an over 70% chance of a clinically meaningful change. Furthermore, under this latter scenario, a 2-point change in CRS-R total score would also happen to be associated–across all three diagnostic groups–with a 78% chance of a clinically meaningful change, which could provide an acceptable, pragmatically simpler, pMCID (i.e., applicable regardless of a patient’s initial diagnosis).
The flexibility demonstrated by this last example is perhaps the most important feature of the method we are presenting. While it has now been over 20 years from the initial distinction of MCS from VS [28], the nosology of DOC has been refined over time [34] and the relationship between individual items on the CRS-R scale (e.g., visual fixation, localization to noxious stimulation, consistent response to command) and diagnosis is still being discussed (cf., [46,47]) and could well change as more evidence and novel techniques are introduced. Furthermore, while we currently focus exclusively on diagnostic behaviors as markers of clinically important changes, our method could be easily extended to include the views of other stakeholders, such as family members, on which items they would consider particularly valuable.
In evaluating these results, a number of important considerations need to be made with respect to the effect of the stipulations we described in the methods section on the resulting pMCID. Specifically, we highlight that our approach is intrinsically very conservative and thus likely to be an overestimate of the change necessary in CRS-R total score to achieve an MCID in DOC patients. First, our approach recognizes only diagnosis-changing behaviors as meeting the threshold for being “clinically meaningful.” It is possible, however, that attaining some non-diagnosis altering behaviors can have meaningful implications. For example, for a patient considered to be MCS+ on the basis of object recognition, the emergence of Intentional (albeit non-functional) Communication or Intelligible Verbalization, is valued by families [48] and can have implications for access to speech and language therapy. Future work combining our quantitative approach with a large sampling of subjective judgments from all stakeholders (e.g., clinicians, care-givers, family members, recovered patients), may lead to a less stringent threshold for the definition of what ought to count as a clinically meaningful change.
Second, the pMCID calculation does not differentiate cases in which one new threshold-behavior is met as compared to cases where multiple new threshold-behaviors are met by a patient. Yet, intuitively, the emergence of multiple novel threshold behaviors, something which is often observed [49], may be associated with better access to multiple pathways for intervention and better prognosis.
Third, we did not count recovery of a threshold behavior as being clinically important if there was no change in diagnosis. For example, should a patient be diagnosed as MCS- on the basis of their ability to demonstrate Localization to Noxious Stimulation, we did not consider the emergence of Object Manipulation or Automatic Motor Responses, behaviors also consistent with MCS-, as being “clinically meaningful” in our approach. Yet, both are novel behaviors that implicate greater cognitive function and that can be leveraged in the context of rehabilitation. Our conservative choice suggests that the pMCIDs may underestimate, but would not overestimate a clinically meaningful change.
In addition, we also note that while the pMCID developed above relies on the observation of motor-response-dependent (responsive) behaviors. Yet, a growing literature demonstrates that it is possible for DOC patients to be unresponsive in traditional bedside assessments while demonstrating responsiveness through neuroimaging and electrophysiology-based approaches [50] as well as enhanced technology-based monitoring (e.g., video eye-tracking) [51]. In as much as it is possible to map such covert responses to items of the CRS-R (e.g., eye-tracking, response to command), the approach developed above should generalize to patients with covert awareness or cognitive-motor dissociation [52]. The degree to which other neuroimaging or neurophysiology-based biomarkers (e.g., resting state network connectivity) which might carry diagnostic or prognostic value [49], but do not map onto any item of existing neurobehavioral scales, could be incorporated in the above framework remains to be understood.
It should also be considered that, as conceived presently, our approach is valid only inasmuch as the CRS-R is applied appropriately. Specifically, the pMCID has no embedded mechanism to account for confounding factors such as those typical of the acute care setting (e.g., sedation, intracranial pressure, hemodynamics, etc). In this respect, our approach relies on being able to abide by well-recognized best practices for CRS-R assessments [12], which are difficult to meet in acute care, potentially limiting the appropriateness of the pMCID to clinical trials in the subacute and chronic phases of the disease. In partial mitigation of this concern, the CRS-R now contains completion codes aimed at identifying (and removing) potential confound sources [53]. In addition, our approach should also be easy to translate to shortened protocols directly derived from the CRS-R, such as the CRSR-FAST [54] and the SECONDs [55].
Finally, by virtue of calculating the pMCID on the basis of relatively large “all-or-none” diagnostic categories, this approach fails to capture finer changes in level of consciousness and what is likely to be a more continuous spectrum of impairments [56]. However, we highlight that this limitation is due to the choice of using well-established diagnostic markers as the basis for threshold behaviors. As the definition of what should count as a change that “would mandate […] a change in the patient’s management” [5] becomes more sophisticated, the pMCID method can be adapted to reflect such finer gradations.
Conclusion
This is the first paper reporting a CRS-R MCID for evaluating treatment efficacy in patients with DOC. Because patients with DOC are inherently unable to provide reliable responses to questions about meaningful recovery, we derived a pMCID based on the liklihood of patients attaining target behaviors that lead to a change in clinical management. The CRS-R pMCIDs proposed here may be used to define intervention responsiveness in future clinical trials and in investigational drug and device applications to the U.S. Food and Drug Administration. As per Table 3, for example, a 2-point change in total CRS-R score is 82% likely to be associated with a clinically important change in VS patients. Thus, if an intervention gave rise to a 2-point change across a sample of patients, it would have to be considered successful. In addition, the tool could also be modified to allow research teams to create pMCIDs tailored to their research question (i.e., by specifying threshold behaviors that are relevant to their specific research question).
References
- 1. Edlow BL, Sanz LRD, Polizzotto L, Pouratian N, Rolston JD, Snider SB, et al. Therapies to Restore Consciousness in Patients with Severe Brain Injuries: A Gap Analysis and Future Directions. Neurocrit Care. 2021;35(Suppl 1):68–85. pmid:34236624
- 2. Nickerson RS. Null hypothesis significance testing: a review of an old and continuing controversy. Psychol Methods. 2000;5(2):241–301. pmid:10937333
- 3. Armijo-Olivo S. The importance of determining the clinical significance of research results in physical therapy clinical research. Braz J Phys Ther. 2018;22(3):175–6. pmid:29602714
- 4. Ogles BM, Lunnen KM, Bonesteel K. Clinical significance: history, application, and current practice. Clin Psychol Rev. 2001;21(3):421–46. pmid:11288608
- 5. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10(4):407–15. pmid:2691207
- 6. Beninato M, Gill-Body KM, Salles S, Stark PC, Black-Schaffer RM, Stein J. Determination of the minimal clinically important difference in the FIM instrument in patients with stroke. Arch Phys Med Rehabil. 2006;87(1):32–9. pmid:16401435
- 7. Gervasoni E, Jonsdottir J, Montesano A, Cattaneo D. Minimal Clinically Important Difference of Berg Balance Scale in People With Multiple Sclerosis. Arch Phys Med Rehabil. 2017;98(2):337–40 e2. pmid:27789239
- 8. Duru G, Fantino B. The clinical relevance of changes in the Montgomery-Asberg Depression Rating Scale using the minimum clinically important difference approach. Curr Med Res Opin. 2008;24(5):1329–35. pmid:18377706
- 9. Hammond FM, Ketchum JM, Patni VV, Nejadnik B, Bates D, Weintraub AH. Determining the Minimally Clinically Important Difference for the Disability Rating Scale in Persons With Chronic Traumatic Brain Injury. Neurotrauma Rep. 2023;4(1):447–57. pmid:37475977
- 10. McCrea MA, Cramer SC, Okonkwo DO, Mattke S, Paadre S, Bates D, et al. Determining minimally clinically important differences for outcome measures in patients with chronic motor deficits secondary to traumatic brain injury. Expert Rev Neurother. 2021;21(9):1051–8. pmid:34402352
- 11. Falissard B, Sapin C, Loze JY, Landsberg W, Hansen K. Defining the minimal clinically important difference (MCID) of the Heinrichs-carpenter quality of life scale (QLS). Int J Methods Psychiatr Res. 2016;25(2):101–11. pmid:26238598
- 12. Giacino JT, Katz DI, Schiff ND, Whyte J, Ashman EJ, Ashwal S, et al. Practice Guideline Update Recommendations Summary: Disorders of Consciousness: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology; the American Congress of Rehabilitation Medicine; and the National Institute on Disability, Independent Living, and Rehabilitation Research. Arch Phys Med Rehabil. 2018;99(9):1699–709. pmid:30098791
- 13. Kondziella D, Bender A, Diserens K, van Erp W, Estraneo A, Formisano R, et al. European Academy of Neurology guideline on the diagnosis of coma and other disorders of consciousness. Eur J Neurol. 2020;27(5):741–56. pmid:32090418
- 14. Schnakers C, Monti MM. Disorders of consciousness after severe brain injury: therapeutic options. Curr Opin Neurol. 2017;30(6):573–9. pmid:28901969
- 15. Thibaut A, Schiff N, Giacino J, Laureys S, Gosseries O. Therapeutic interventions in patients with prolonged disorders of consciousness. Lancet Neurol. 2019;18(6):600–14. pmid:31003899
- 16. American Congress of Rehabilitation Medicine. Assessment scales for disorders of consciousness: evidence-based recommendations for clinical practice and research. Arch Phys Med Rehabil. 2010;91(12):1795–813. pmid:21112421
- 17. Formisano R, Giustini M, Aloisi M, Contrada M, Schnakers C, Zasler N, et al. An International survey on diagnostic and prognostic protocols in patients with disorder of consciousness. Brain Inj. 2019;33(8):974–84. pmid:31146603
- 18. Giacino JT, Kalmar K, Whyte J. The JFK Coma Recovery Scale-Revised: measurement characteristics and diagnostic utility. Arch Phys Med Rehabil. 2004;85(12):2020–9. pmid:15605342
- 19. Mallinson T, Pape TL, Guernon A. Responsiveness, Minimal Detectable Change, and Minimally Clinically Important Differences for the Disorders of Consciousness Scale. J Head Trauma Rehabil. 2016;31(4):E43–51. pmid:26360003
- 20. American Congress of Rehabilitation Medicine, Brain Injury-Interdisciplinary Special Interest Group, Disorders of Consciousness Task Force, Seel RT, Sherer M, Whyte J, Katz DI, Giacino JT, et al. Assessment scales for disorders of consciousness: evidence-based recommendations for clinical practice and research. Arch Phys Med Rehabil. 2010;91(12):1795–813. pmid:21112421
- 21. Giacino JT, Whyte J, Bagiella E, Kalmar K, Childs N, Khademi A, et al. Placebo-controlled trial of amantadine for severe traumatic brain injury. N Engl J Med. 2012;366(9):819–26. pmid:22375973
- 22. Whyte J, Rajan R, Rosenbaum A, Katz D, Kalmar K, Seel R, et al. Zolpidem and restoration of consciousness. Am J Phys Med Rehabil. 2014;93(2):101–13. pmid:24434886
- 23. Thibaut A, Bruno MA, Ledoux D, Demertzi A, Laureys S. tDCS in patients with disorders of consciousness: sham-controlled randomized double-blind study. Neurology. 2014;82(13):1112–8. pmid:24574549
- 24. Magrassi L, Maggioni G, Pistarini C, Di Perri C, Bastianello S, Zippo AG, et al. Results of a prospective study (CATS) on the effects of thalamic stimulation in minimally conscious and vegetative state patients. J Neurosurg. 2016;125(4):972–81. pmid:26745476
- 25. Cain JA, Spivak NM, Coetzee JP, Crone JS, Johnson MA, Lutkenhoff ES, et al. Ultrasonic thalamic stimulation in chronic disorders of consciousness. Brain Stimul. 2021;14(2):301–3. pmid:33465497
- 26. Pape TL, Rosenow JM, Steiner M, Parrish T, Guernon A, Harton B, et al. Placebo-Controlled Trial of Familiar Auditory Sensory Training for Acute Severe Traumatic Brain Injury: A Preliminary Report. Neurorehabil Neural Repair. 2015;29(6):537–47. pmid:25613986
- 27. Cheng L, Cortese D, Monti MM, Wang F, Riganello F, Arcuri F, et al. Do Sensory Stimulation Programs Have an Impact on Consciousness Recovery? Front Neurol. 2018;9:826. pmid:30333789
- 28. Giacino JT, Ashwal S, Childs N, Cranford R, Jennett B, Katz DI, et al. The minimally conscious state: definition and diagnostic criteria. Neurology. 2002;58(3):349–53. pmid:11839831
- 29. Majerus S, Gill-Thwaites H, Andrews K, Laureys S. Behavioral evaluation of consciousness in severe brain damage. Prog Brain Res. 2005;150:397–413. pmid:16186038
- 30. Sherer M, Katz DI, Bodien YG, Arciniegas DB, Block C, Blum S, et al. Post-traumatic Confusional State: A Case Definition and Diagnostic Criteria. Arch Phys Med Rehabil. 2020;101(11):2041–50. pmid:32738198
- 31. Bodien YG, Martens G, Ostrow J, Sheau K, Giacino JT. Cognitive impairment, clinical symptoms and functional disability in patients emerging from the minimally conscious state. NeuroRehabilitation. 2020;46(1):65–74. pmid:32039868
- 32. Copay AG, Subach BR, Glassman SD, Polly DW Jr., Schuler TC. Understanding the minimum clinically important difference: a review of concepts and methods. Spine J. 2007;7(5):541–6. pmid:17448732
- 33. Laureys S, Celesia GG, Cohadon F, Lavrijsen J, Leon-Carrion J, Sannita WG, et al. Unresponsive wakefulness syndrome: a new name for the vegetative state or apallic syndrome. BMC Med. 2010;8(1):68. pmid:21040571
- 34. Bruno MA, Vanhaudenhuyse A, Thibaut A, Moonen G, Laureys S. From unresponsive wakefulness to minimally conscious PLUS and functional locked-in syndromes: recent advances in our understanding of disorders of consciousness. J Neurol. 2011;258(7):1373–84. pmid:21674197
- 35. Monti MM, Laureys S, Owen AM. The vegetative state. BMJ. 2010;341(aug02 1):c3765.
- 36. Vanier A, Toscano A, Sebille V, Hardouin JB. What are all the proposed methods to estimate the minimal clinically important difference of a patient-reported outcomes measure? A systematic review. Qual Life Res. 2017;26(1):20-.
- 37. Wells G, Beaton D, Shea B, Boers M, Simon L, Strand V, et al. Minimal clinically important differences: review of methods. J Rheumatol. 2001;28(2):406–12. pmid:11246688
- 38. Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner: a closer look at the minimal clinically important difference (MCID). J Man Manip Ther. 2012;20(3):160–6. pmid:23904756
- 39. Machado C, Estevez M, DeFina PA, Leisman G. Response to Lewis A: Reconciling the Case of Jahi Mcmath. Neurocrit Care. 2018;29(3):521–2. pmid:30225822
- 40. Lewis A. Reconciling the Case of Jahi McMath. Neurocrit Care. 2018;29(1):20–2. pmid:29923168
- 41. Lewis A. Response to Machado et al. re: Jahi McMath. Neurocrit Care. 2018;29(3):523–4. pmid:30187282
- 42. Zuckerman DA, Giacino JT, Bodien YG. Traumatic Brain Injury: What Is a Favorable Outcome? J Neurotrauma. 2022;39(13–14):1010–2. pmid:34861770
- 43. Giacino JT. The vegetative and minimally conscious states: consensus-based criteria for establishing diagnosis and prognosis. NeuroRehabilitation. 2004;19(4):293–8. pmid:15671583
- 44. Thibaut A, Bodien YG, Laureys S, Giacino JT. Minimally conscious state "plus": diagnostic criteria and relation to functional recovery. J Neurol. 2020;267(5):1245–54. pmid:31773246
- 45. Chatelle C, Bodien YG, Carlowicz C, Wannez S, Charland-Verville V, Gosseries O, et al. Detection and Interpretation of Impossible and Improbable Coma Recovery Scale-Revised Scores. Arch Phys Med Rehabil. 2016;97(8):1295–300 e4. pmid:26944708
- 46. Golden K, Erler KS, Wong J, Giacino JT, Bodien YG. Should consistent command-following be added to the criteria for emergence from the minimally conscious state? Arch Phys Med Rehabil. 2022.
- 47. Weaver JA, Cogan AM, O’Brien KA, Hansen P, Giacino JT, Whyte J, et al. Determining the Hierarchy of Coma Recovery Scale-Revised Rating Scale Categories and Alignment with Aspen Consensus Criteria for Patients with Brain Injury: A Rasch Analysis. J Neurotrauma. 2022;39(19–20):1417–28. pmid:35570725
- 48. Jox RJ, Kuehlmeyer K, Klein AM, Herzog J, Schaupp M, Nowak DA, et al. Diagnosis and decision making for patients with disorders of consciousness: a survey among family members. Arch Phys Med Rehabil. 2015;96(2):323–30.
- 49. Giacino JT, Sherer M, Christoforou A, Maurer-Karattup P, Hammond FM, Long D, et al. Behavioral Recovery and Early Decision Making in Patients with Prolonged Disturbance in Consciousness after Traumatic Brain Injury. J Neurotrauma. 2020;37(2):357–65. pmid:31502498
- 50. Sanz LRD, Thibaut A, Edlow BL, Laureys S, Gosseries O. Update on neuroimaging in disorders of consciousness. Curr Opin Neurol. 2021;34(4):488–96. pmid:34054109
- 51. Alkhachroum A, Aklepi G, Sarafraz A, Robayo LE, Manolovitz BM, Blandino CF, et al. Covert Tracking to Visual Stimuli in Comatose Patients With Traumatic Brain Injury. Neurology. 2023. pmid:37076304
- 52. Edlow BL. Covert Consciousness: Searching for Volitional Brain Activity in the Unresponsive. Curr Biol. 2018;28(23):R1345–R8. pmid:30513331
- 53. Bodien YG, Chatelle C, Taubert A, Uchani S, Giacino JT, Ehrlich-Jones L. Updated Measurement Characteristics and Clinical Utility of the Coma Recovery Scale-Revised Among Individuals With Acquired Brain Injury. Arch Phys Med Rehabil. 2021;102(1):169–71.
- 54. Bodien YB, Vora I, Barra A, Chiang K, Chatelle C, Goostrey K, et al. Feasibility and validity of the Coma Recovery Scale-Revised For Accelerated Standardized Testing: A practical assessment tool for detecting consciousness in the intensive care unit. Ann Neurol. in press,. pmid:37488068
- 55. Wang A, Sun L, Cheng L, Hu N, Chen Y, Sanz LRD, et al. Validation of the simplified evaluation of consciousness disorders (SECONDs) scale in Mandarin. Ann Phys Rehabil Med. 2023;66(7):101764. pmid:37276835
- 56. Fins JJ, Schiff ND. Shades of gray: new insights into the vegetative state. Hastings Cent Rep. 2006;36(6):8. pmid:17278863