Background

Patient-reported outcome measures (PROMs) quantify health from the patient’s perspective. PROMs relevant to musculoskeletal conditions generally quantify levels of comfort (pain intensity) and capability. PROMs were initially developed as research tools and many are lengthy and burdensome. Short-form PROMs and PROMs utilizing computer adaptive testing (CATs) have decreased the burden of measure completion, making them more appealing for routine use. PROMs are incorporated in the electronic medical records at many institutions, offering a potential point of discussion between patient and clinician [1]. There is hope that quantification of musculoskeletal comfort and capability can help align test and treatment options to what matters most to individual patients. The concept is that competition centered on measures of outcomes that are important to patients has the potential to improve health while reducing the variability and cost of care [2,3].

Rationale

In the research setting, PROMs can help compare the outcomes of two or more test or treatment strategies. In the care of an individual patient, the interpretation of PROM scores may be unclear. PROM scores do not have disease or health benchmarks like a blood pressure or hemoglobin A1C, and a person can have low levels of discomfort and incapability in the presence of severe disease [4–7]. Minimal clinically important difference can be used to evaluate change scores, but are not useful at the initiation of care and have other limitations [8]. There is, to our knowledge, little evidence or consensus among surgeons regarding how measures of the subjective are interpreted and whether surgeons use thresholds to interpret or act on scores in terms of decision-making.

To better understand how surgeons interpret and act on the scores of patient-reported measures in the care of individual patients, we performed an online scenario-based experiment in which scores of various patient-reported measures were randomized into fictional scenarios and asked: 1) What factors are associated with surgeon concern regarding the scores of patient-reported measures in a fictional patient scenario? 2) What factors are associated with surgeon likelihood to offer discretionary surgery? 3) What factors are associated with surgeon motivation to discuss mental health or social circumstances? 4) What factors are associated with surgeon likelihood to refer for mental and/or social health services?

Study design and setting

Upper limb specialists from the Science of Variation Group (SOVG) were invited via email between May 5^th 2023 and June 1^st 2023 to participate in a series of experiments addressing the interpretation of PROMs in the care of individual patients (S1 Appendix). In the first stage, a structured questionnaire was used to assess the use of PROMs quantifying capability and mental health in the daily practice of surgeons and to explore the impact, or potential impact, of these scores on care strategies (S2 Appendix).

In the second stage of this pilot work, surgeons reviewed four fictious patient scenarios with PROMs of capability and mental health with varied PROM score distribution (Gaussian and non-Gaussian) for capability and mental health (S2 Appendix). Surgeons indicated which score values concerned them and explained their reasoning, including patient characteristics and perceived barriers to actions. We found that PROMs were infrequently used in practice (40% collected capability scores, 14% mental health scores), and most surgeons did not rely on threshold scores to guide decisions. When asked to interpret scores without guidance, surgeons tended to use statistical benchmarks (e.g., ≥ 2 SD or top quartile). Some considered high discomfort or incapability a reason to delay surgery and address mental health first, while others saw it as an indication to operate. Factors such as legal disputes also influenced interpretation. Barriers to addressing psychosocial issues included patient unawareness of mental health’s role and reluctance to engage in mental or social health support. We assessed the interrater reliability of surgeon rated threshold scores of concern for the 3 ratings ranging between 0 and 100 with normal (2 cases) and non-normal (1 case) distribution. One rating was on a different scale (3 to 15) and could not be included in the reliability analysis. We calculated a two-way mixed effect intraclass correlation coefficient as a measure of variation between specialists and found poor agreement (intraclass correlation coefficient of 0.21; 95% CI: 0.054 to 0.92; S1 Appendix).

In a third stage, building on the first two experiments, all upper limb specialists from the SOVG were invited again to review 10 fictitious scenarios of patients with osteoarthritis (OA) in the upper extremity with randomized elements. We randomized the following elements: age (45, 61 or 83), gender (woman or man), type of pathophysiology (including location of OA and radiographic severity), and scores of fictional patient-reported measures capturing levels of comfort and capability, mental health, and social health. of the trapeziometacarpal (TMC), wrist, and elbow joints were chosen to allow measurement of disease severity as classified from radiographs among different diagnoses. We categorized the pathophysiologic severity at the TMC joint using the Sodha grade, at the wrist using the Scapholunate advanced collapse (SLAC) stage, and at the elbow using the Broberg-Morrey (B-M) grade [9–11]. We classified Sodha grade 1/ SLAC stage 2/ B-M grade 2 as ‘mild’, Sodha grade 2/ SLAC stage 3/ B-M grade 3 as ‘moderate’, and Sodha grade 3/ SLAC stage 4/ B-M grade 4 as ‘severe’. We also randomized various values of fictious PROMs resembling levels of pain intensity and incapability, unhelpful thoughts and feelings of distress (mental health), and social deprivation index (social health). In each scenario, surgeons were informed that all measures (except pain intensity) were scaled from 0 (best possible score) to 100 (worst possible score), in which 50 represented the general population mean, with 10 points higher or lower denoting a standard deviation. Pain intensity scores were presented on a scale from 0 (no pain) to 10 (worst pain imaginable). We decided to randomize the pain intensity and incapability scores together to avoid discrepancy in the scenarios (e.g., high pain intensity with low incapability). The scores were randomized to represent “low”, “average”, or “high” values, in which average scores were near the mean, and low or high scores were between 1.5 to 2.5 standard deviations from the mean.

Participants rated (on a 0–100 scale) their level of concern for the patient, likelihood to offer discretionary surgery, motivation to discuss mental and social health, and likelihood to refer to related services (e.g., a social worker or behavioral therapist), with higher scores indicating greater concern, likelihood, or motivation. The surveys were presented to surgeons using Survey Monkey (Palo Alto, CA, USA).

Participants

On July 14^th, 2023, an email invitation was sent to all upper limb specialist members of the SOVG, an international collaboration of musculoskeletal surgeons that studies sources of variation in care. Participating members receive group authorship or acknowledgment, and there were no financial incentives for their participation. To limit bias, the study participants were not briefed on the study’s purpose. Completion of the surveys implied consent. The study protocol, including this approach to consent, was reviewed and approved by the institutional ethics board. Following the initial invitation, non-responders received weekly reminders for three weeks. The study was closed on August 8^th, 2023. Among approximately 150 upper limb surgeons that completed at least one survey in the last year, a total of 67 surgeons (45%) participated and completed 89% (598 of 670) of all presented scenarios and 87% of the surgeons [58 of 67] completed all 10 scenarios. Of the remaining surgeons, six completed one scenario, one completed two scenarios and one completed five scenarios. As each surgeon received a random selection of scenarios, the missing data is unlikely to be systematically related to specific case types, minimizing the risk of bias. Most surgeons were men (90%), practice in the United States (61%), supervise trainees (85%), and are specialized in hand and wrist (64%) (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of 67 surgeon participants.

https://doi.org/10.1371/journal.pone.0332221.t001

Statistical analysis

We used descriptive statistics to present surgeon demographics. Categorical variables were presented as number with percentage and continuous variables were presented as mean with standard deviation or median with interquartile range dependent on data distribution.

We sought whether surgeon interpretation of scores derived from PROMs capturing comfort and capability, mental health and social health were associated with 1) surgeon concern, 2) likelihood to offer surgery, 3) motivation to discuss mental and social health, and 4) likelihood to refer for mental and social health services. We used multilevel mixed-effects linear regression models to measure associations between these outcomes and randomized patient characteristics (different scores for various PROMs, age, gender, affected joint, and OA severity), while accounting for nesting by surgeon (non-independence as each surgeon rated multiple cases) [12]. We reported the standardized regression coefficient (RC), 95% confidence interval (95%CI), and corresponding P-values. All P-values below 0.05 were considered statistically significant. Quantitative data was analyzed using StataMP (Stata 14.0, StataCorp, College Station, TX).

An a priori sample size calculation was performed using G*Power (version 3.1.9.7). Considering the complexity of accurately specifying the parameters and nesting structure of a multilevel mixed-effects model (e.g., intraclass correlation, variability between raters), we opted to perform a sample size calculation for a simpler multivariable linear regression model [13]. This approach allowed us to estimate the required sample size based on anticipated effect size when all included variables account for 15% of the variation in the assessed outcome (f² = 0.18), desired statistical power (0.80), and significance level (alpha set at 0.05) without relying on potentially unreliable assumptions about nesting by surgeon. This calculation determined that 89 unique observations were required. Given that the achieved sample size of 598 observations (58 participants completed all 10 scenarios, 2 participants completed 5 scenarios, 1 participant completed 2 scenarios, and 6 participants completed 1 scenario) exceeded the a priori estimate by a large margin, it was considered sufficient.

Surgeon concern regarding the patient

Accounting for potential confounders such as the patient’s age and gender as well as nesting by surgeon, we found that greater surgeon concern regarding an individual based on pathophysiologic severity and scores of fictional PROMs was associated moderately with wrist OA (RC = 4.3, 95%CI = 1.3 to 7.2) and elbow OA (RC = 3.5, 95%CI = 0.58 to 6.4) compared to TMC OA, moderately to strongly with higher scores of distress and unhelpful thoughts (average: RC = 4.5, 95%CI 1.5 to 7.4; and high: RC = 12, 95%CI = 8.9 to 15), moderately with high social deprivation index scores (RC = 5.1, 95%CI = 2.1 to 8.1), and strongly with higher scores of discomfort and incapability (average: RC = 8.4, 95%CI = 5.4 to 12; and high: RC = 17, 95%CI = 14–20) (Tables 2 and 3). Lower surgeon concern regarding an individual was associated moderately with 61 years old age (RC = −3.5, 95%CI = −6.5 to −0.57) compared to 45 years old.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of surgeon responses to patient scenarios.

https://doi.org/10.1371/journal.pone.0332221.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Mixed multi level linear regression analysis of patient factors associated with surgeons decision making.

https://doi.org/10.1371/journal.pone.0332221.t003

Likelihood to offer discretionary surgery

Accounting for potential confounders such as the patient’s age and gender as well as nesting by surgeon, we found that greater likelihood to offer surgery based on pathophysiologic severity and scores of fictional PROMs was associated strongly with higher radiographic OA severity (moderate: RC = 10, 95%CI = 6.7 to 14; and severe: RC = 15, 95%CI = 11–18), moderately with wrist OA (RC = 5.9, 95%CI = 2.2 to 9.6) and elbow OA (RC = 4.9, 95%CI = 1.3 to 8.5) relative to TMC OA, moderately with average social deprivation index scores (RC = 3.8, 95%CI = 0.081 to 7.5) compared to low social deprivation, and moderately to strongly with higher scores of discomfort and incapability (average: RC = 7.1, 95%CI = 3.2 to 11; and high: RC = 13, 95%CI = 9.0 to 16) (Tables 2 and 3). A lower likelihood to offer surgery was moderately associated with high scores of distress and unhelpful thinking (RC = −8.3, 95%CI = −12 to −4.6) compared to low scores.

Motivation to discuss mental health and/or social circumstances

Accounting for potential confounders such as the patient’s age and gender as well as nesting by surgeon, we found that greater motivation to discuss mental health and social circumstances based on pathophysiologic severity and scores of fictional PROMs was associated moderately to strongly with higher scores of distress and unhelpful thoughts (average: RC = 7.1, 95%CI = 3.8 to 10; and high: RC = 17, 95%CI = 14–21), moderately with high social deprivation scores (RC = 6.1, 95%CI = 2.7 to 9.4), and moderately to strongly with higher scores of discomfort and incapability (average: RC = 5.0, 95%CI = 1.5 to 8.4; and high: RC = 13, 95%CI = 9.6 to 16) (Tables 2 and 3). Lower motivation to discuss mental health and social circumstances was associated moderately with elbow OA (RC = −4.3, 95%CI = −7.6 to −1.1) compared to TMC OA.

Likelihood to refer for mental and/or social health services

Accounting for potential confounders such as the patient’s age and gender as well as nesting by surgeon, we found that greater likelihood to refer for mental and/or social health services based on pathophysiologic severity and scores of fictional PROMs was associated moderately to substantially with higher scores of distress and unhelpful thoughts (average: RC = 6.8, 95%CI = 3.4 to 10; and high: RC = 21, 95%CI = 17–24), moderately with high deprivation index scores (RC = 6.9, 95%CI = 3.5 to 10), and strongly with high scores of discomfort and incapability (RC = 12, 95%CI = 9.1 to 16) compared to low scores (Tables 2 and 3). Lower likelihood to refer was associated moderately with elbow AO (RC = −3.4, 95%CI = −6.7 to –0.051) and worse radiographic severity (moderate: RC = −6.3, 95%CI = −9.8 to −2.8; and severe: RC = −6.0 95%CI = −9.4 to −2.5) (Tables 2 and 3).

Surgeon concern regarding the patient

The observation that surgeon interpretation of PROMs was related to notable deviations from expected (mean) scores, in which surgeons rated scores as concerning above 2 standard deviations (2.5% of the population) or in the top quartile (25% of the population) (S1 Appendix), merits concern. Surgeons may not realize to which extent these interpretations lead to a different understanding of scores related to personal aspects of illness, and these findings direct us to educate surgeons about the influence of data distribution on the interpretation of patient-reported scores.

The observation that surgeon concern regarding patients was associated with greater scores of discomfort and incapability, greater scores of distress and unhelpful thinking, and greater social deprivation index scores suggests that surgeons are able to interpret PROMs in a comprehensive way, conceptualizing them within the biopsychosocial paradigm of human illness. The relative strength of association as judged by the regression coefficients suggest that surgeon concern is most strongly associated with levels of discomfort and incapability, followed by levels of distress and unhelpful thinking. Surgeons seem to recognize that relatively high levels of discomfort and incapability on patient-reported measure scores can be a signal to consider mental health and social circumstances. This might reflect a growing awareness of the notable evidence that a large proportion of the variation in levels of discomfort and incapability is accounted for by mindset (thoughts and feelings), relatively independent of pathophysiological severity [6,15–20].

The observation that TMC OA elicited comparatively lower concern among surgeons compared to wrist OA and elbow OA could be attributed to the near universal prevalence of TMC OA with age (expected senescence) [21], which may coincide with participant anticipation of a notable rate of effective accommodation to which patients could be guided. In contrast, wrist OA and elbow OA are less common and often post-traumatic [22,23].

Likelihood to offer discretionary surgery

The finding that surgeons were more likely to offer surgery for patients with greater pain intensity and incapability scores, worse radiographic severity, and those with elbow or wrist OA implies that surgeons interpret these factors as indicative of greater potential for benefit by addressing pathophysiology. Surgeons might sense that surgery is best suited for those with advanced pathophysiology and/or greater scores of pain intensity and incapability when there are low levels of psychosocial concern. However, it is important to keep in mind that these findings are based on fictious patient scenarios in which randomized scores may have presented an unlikely or uncommon scenario such as high levels of discomfort and incapability with low levels of distress, unhelpful thoughts, and social deprivation. Given that surgeons may tend to anchor on surgical treatment options there may be a role for a checklist to remind surgeons to consider prioritizing mental and social health when surgery is discretionary (optional) and not time-sensitive (elective) [24]. It is important that surgeons keep in mind that disproportionately greater discomfort can be a signal of important mental and social health factors [6,15–19], and that there is a relatively limited association between radiographic severity in OA and symptom intensity [25,26]. The observation that lower likelihood to offer surgery is associated with an average deprivation index is difficult to interpret. This may reflect unmeasured socioeconomic factors influencing decision-making or access to care. Give the exploratory nature of this study, we advise caution in interpreting this finding.

Motivation to discuss mental health and/or social circumstances and likelihood to refer to mental and social health services

The findings that surgeons exhibit greater motivation to discuss mental and social health when a patient has relatively greater scores of pain intensity and incapability, distress and unhelpful thinking, and social deprivation reiterates that PROMs serve as cues for specialists to address these aspects of patient well-being. Similarly, the observation that surgeons are more inclined to refer a patient for mental and social health support based on greater scores of pain intensity and incapability, distress and unhelpful thinking, and social deprivation, suggests that PROMs could facilitate comprehensive care strategies. This is intriguing, particularly in comparison to previous SOVG studies [27,28], which found that surgeons were likely to recognize mental health issues but remained neutral regarding referrals due to perceived barriers. This might reflect the potential utility of incorporating PROMs into routine care in order to help nudge surgeon awareness of certain aspects of the patient’s health that might be prioritized. Especially considering previous evidence has shown that priming surgeons to be more attuned for opportunities to address mental health can increase the identification of feelings of distress and unhelpful thoughts [29]. Integration of PROMs to attune surgeons may further enhance identification and addressment of mental health opportunities. Recent studies underscore that surgeons express willingness to address psychological factors, but also report practical barriers including 1) perceived time constraints, 2) discomfort with discussing mental and social health, 3) perceived societal stigma hindering effective referrals for mental and social health, and 4) perceived lack of access to resources to provide mental and social health care [27,30]. The preparatory work for this study identified similar barriers (S1 Appendix).

The observation that worse radiographic OA severity and involvement of the elbow decreased surgeon likelihood to refer again suggests some variations in care tendencies related to pathophysiology. Surgeons may associate excessive pain and distress with severe arthritis, influencing their decision-making regarding referrals. It is also possible that surgeons feel more confident or effective in treating pathophysiology, and perhaps think that it could improve a person’s mental health as well. Yet, addressing mental and social health needs may benefit someone regardless of their OA severity.

Limitations

This study can be considered in light of several limitations. First, the participants in this study were predominantly men working in Europe or the United States and 85% were supervising trainees indicating that most of the surgeons work in academic environments. Therefore, the measured rates of categorical variables and means and medians of continuous variables may not be reproduced in other samples of specialists. However, the statistical relationships between variables may be reproducible in any sample with adequate variation in opinions as can be seen in the wide standard deviations in our response variables. Prior SOVG studies confirm a diversity of opinions among participants [29,31,32]. Second, some of the observed variation may relate to the wording and interpretation of the scenarios and alternative wordings could be explored in future studies. Third, the use of fictional patient scenarios does not capture the intricacies and nuances of actual patient encounters. However, the relative simplicity of the printed scenarios might help isolate potentially important factors and determine if more sophisticated studies observing the opinions and behaviors of clinicians with actual patients are worthwhile. Moreover, because the patient characteristics were randomly generated, some scenarios may have presented unlikely or uncommon combinations, such as high levels of discomfort and incapability alongside low levels of distress, unhelpful thoughts, and social deprivation. Fourth, we focused on a single specialty visit as claims data suggest that most patients see a specialist a single time [7]. This precluded the use of benchmarks such as minimal important differences over time. Fifth, our scenarios only presented cases of OA in the upper extremity. Although not directly tested, we hypothesize that the interpretation of PROMs and the influence on treatment recommendation for other non-emergent musculoskeletal conditions would be comparable. We chose to focus on OA because radiographs can be used as a representation of the severity of the pathophysiology. Some readers might consider this an inadequate measure of pathophysiology, but imaging is the best currently available representation of OA severity. Sixth, a small number of surgeons exited the survey before rating all the scenarios. Since each scenario is randomized and unique, this should not alter the findings appreciably. Seventh, the participants likely gleaned the purpose of the study even though it was not explicitly stated, which could result in a Hawthorne effect [33], although it’s difficult to know the degree to which research participation effects lead to misleading results.