https://orcid.org/0000-0003-4403-909X
Registered Report Protocol
4 Jun 2021: Long C, Beres LK, Wu AW, Giladi AM (2021) Developing a protocol for adapting multimedia patient-reported outcomes measures for low literacy patients. PLOS ONE 16(6): e0252684. https://doi.org/10.1371/journal.pone.0252684 View registered report protocol
Figures
Abstract
Introduction
Almost all patient-reported outcomes measures (PROMs) are text-based, which impedes accurate completion by low and limited literacy patients. Few PROMs are designed or validated to be self-administered, either in clinical or research settings, by patients of all literacy levels. We aimed to adapt the Patient Reported Outcomes Measurement Information System Upper Extremity Short Form (PROMIS-UE) to a multimedia version (mPROMIS-UE) that can be self-administered by hand and upper extremity patients of all literacy levels.
Methods
Our study in which we applied the Multimedia Adaptation Protocol included seven phases completed in a serial, iterative fashion: planning with our community advisory board; direct observation; discovery interviews with patients, caregivers, and clinic staff; ideation; prototyping; member-checking interviews; and feedback. Direct observations were documented in memos that underwent rapid thematic analysis. Interviews were audio-recorded and documented using analytic memos; a rapid, framework-guided thematic analysis with both inductive and deductive themes was performed. Themes were distilled into design challenges to guide ideation and prototyping that involved our multidisciplinary research team. To assess completeness, credibility, and acceptability we completed additional interviews with member-checking of initial findings and consulted our community advisory board.
Results
We conducted 12 hours of observations. We interviewed 17 adult English-speaking participants (12 patients, 3 caregivers, 2 staff) of mixed literacy. Our interviews revealed two distinct user personas and three distinct literacy personas; we developed the mPROMIS-UE with these personas in mind. Themes from interviews were distilled into four broad design challenges surrounding literacy, customizability, convenience, and shame. We identified features (audio, animations, icons, avatars, progress indicator, illustrated response scale) that addressed the design challenges. The last 6 interviews included member-checking; participants felt that the themes, design challenges, and corresponding features resonated with them. These features were synthesized into an mPROMIS-UE prototype that underwent rounds of iterative refinement, the last of which was guided by recommendations from our community advisory board.
Discussion
We successfully adapted the PROMIS-UE to an mPROMIS-UE that addresses the challenges identified by a mixed literacy hand and upper extremity patient cohort. This demonstrates the feasibility of adapting PROMs to multimedia versions. Future research will include back adaptation, usability testing via qualitative evaluation, and psychometric validation of the mPROMIS-UE. A validated mPROMIS-UE will expand clinicians’ and investigators’ ability to capture patient-reported outcomes in mixed literacy populations.
Citation: Azad CL, Beres LK, Wu AW, Fong A, Giladi AM (2024) Developing a multimedia patient-reported outcomes measure for low literacy patients with a human-centered design approach. PLoS ONE 19(6): e0304351. https://doi.org/10.1371/journal.pone.0304351
Editor: Xiaodan Tang, Northwestern University Feinberg School of Medicine, UNITED STATES
Received: August 7, 2023; Accepted: May 10, 2024; Published: June 5, 2024
Copyright: © 2024 Azad et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: CLA and AMG received funding from the American Foundation for Surgery of the Hand (Award 3698) and the Johns Hopkins Physician-Scientist Training Program for this work. Each author certifies that there are no funding or commercial associations (consultancies, stock ownership, equity interest, patent/licensing arrangements, etc.) that might pose a conflict of interest in connection with the submitted article related to the author or any immediate family members. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. https://www.afsh.org/s/ https://www.hopkinsmedicine.org/som/gme/pstp/.
Competing interests: The authors have declared that no competing interests exist.
Introduction
A growing focus on patient-centered and value-based care in the United States (US) has supported the importance of collecting data from the patient perspective [1–3]. The patient perspective can be captured by patient-reported outcomes (PROs), which are assessments of a health condition made directly by the patient without interpretation or estimation by clinicians or anyone else [4]. PROs, collected via patient-reported outcomes measures (PROMs), can be used to guide clinical care, as part of performance measures for care quality, or as an outcome measure in research. Despite the utility, importance, and growing popularity of PROs, essentially all existing PROMs are reliant on text. This represents a substantial methodological and implementation challenge among low and limited literacy patient populations. Up to nearly one-half of US adults have low English literacy [5, 6], and low literacy rates are disproportionately higher in ethnic and racial minority populations [7]. Interviewer-administration is an option, but it is labor intensive and expensive, and introduces potential for interviewer bias [8] and patient embarrassment [9] that are especially important considerations in low literacy populations [10, 11]. It is critical that PROMs are designed for implementation in an inclusive, equitable fashion. Otherwise, reliance on PROs to understand outcomes and care quality risks becoming a systemic means of perpetuating and exacerbating health disparities.
To capture PROs more equitably, we need a self-administered PROM that accommodates a range of literacy levels. This may also allow us to capture PROs more accurately by ensuring that low literacy patients’ responses are accurate reflections of their experiences [12]. Because multimedia PROMs (mPROMs) with audiovisual components are a promising solution, we developed the Multimedia Adaptation Protocol (MAP) [13], which adapts validated, text-based PROMs to a multimedia format that supports self-administration in populations with mixed literacy. The MAP includes four stages: forward adaptation, back adaptation, qualitative evaluation, and validation.
To execute and demonstrate the application of the MAP, we elected to adapt the Patient Reported Outcomes Measurement Information System (PROMIS®) Upper Extremity (PROMIS-UE) to a multimedia version (mPROMIS-UE) in a hand and upper extremity surgery patient population. In the U.S., an estimated 3.5 million patients sustain hand and upper extremity trauma each year [14], with more than 1.1 million upper extremity ambulatory procedures performed annually [15]. In hand and upper extremity care, growing evidence of the divergence between patient and provider perception of health and functional status has driven an increase in the use of PROs as outcome measures in research and to guide patient-centered and value-based clinical care [16–19]. The PROMIS-UE and the hand and upper extremity population are ideal for the first application of the MAP because variation in physical function, such as in upper extremity function, may be easier to convey in multimedia elements than variations in mental or social health.
The goal of this study was to demonstrate the feasibility of adapting PROMs to multimedia versions by executing the first stage of the MAP, forward adaptation, to adapt the PROMIS-UE to the mPROMIS-UE. This would result in the first multimedia PROM that could be self-administered in a mixed literacy patient population.
Methods
Research ethics
This study was performed in accordance with the ethical standards in the 1964 Declaration of Helsinki. This study was carried out in accordance with relevant regulations of the US Health Insurance Portability and Accountability Act (HIPAA). This study was approved by the MedStar Health Research Institute institutional review board (STUDY00002319). The recruitment period started November 23, 2020 and ended March 1, 2021. Participants provided written informed consent [20].
Study setting
The Curtis National Hand Center in Baltimore, Maryland was selected as the setting for this research for several reasons. It is one of the largest hand centers with the highest volume of hand patients in the nation. The clinic has broad experience with PRO data collection and management. PROs are either self-administered or administered by clinic staff via a web-based system, with over 90% of patients completing data entry and questionnaires [21]. These PROs are used for research and for direct clinical care including patient education and progress tracking. The patient population draws largely from Baltimore City, ranked last for health outcomes among Maryland’s 24 counties [22]. Baltimore’s prevalence of illiteracy is 15.9% [22] and prevalence of poverty is 22.4% [23], both above the national average.
Instrument selection
For this first effort to adapt a text-based PROM to its corresponding mPROM, we selected a PROM that measures physical function because questions related to physical status or ability are more conducive to visual representations. We chose the Patient Reported Outcomes Measurement Information System Upper Extremity Short Form 7a (PROMIS-UE) [24], a 7-item questionnaire drawn from the PROMIS-UE Item Bank v2.0. The PROMIS-UE is a generic measure that can be used across all diseases and conditions. It is validated to capture patient perception of upper extremity physical status or ability. It has demonstrated sufficient construct validity with moderate to strong correlation with the legacy instrument Quick Disabilities of Arm, Shoulder, and Hand (QuickDASH) (r = -0.47 to -0.83) [25, 26], good internal consistency (Cronbach alpha coefficient ranging between 0.78–0.93) [27–30], and good responsiveness (AUC 0.81–0.82) [29, 30]. Regarding content validity, PROMIS-UE has not been found to be limited by floor effects, but there are mixed findings for ceiling effects [29, 31–36].
We chose this measure because it offers several unique advantages. PROMIS instruments, unlike other upper extremity PROMs, are included among the “Common Data Elements” the National Institutes of Health encourages for broad use in research [37]. Additionally, because the PROMIS-UE Item Bank includes questions developed to have the same measurement properties (i.e., measure the same dimension and fit the item response theory model on which they were calibrated), any subset of questions from the bank can be used and scored appropriately [38–40].
Study design
We used a community-engaged, human-centered design (HCD) approach. Our process for forward adaptation included seven phases, detailed below and summarized in Table 1. The findings and deliverables from each phase were used to inform and refine subsequent phases. This study adheres to our published protocol [13] and is reported according to the Standards for Reporting Qualitative Research (SRQR) [20] as well as Bazzano et al.’s reporting recommendations for health research involving design [41].
A. Community-engaged study planning.
To obtain community input beyond individuals directly associated with the study, we engaged an eight-member community advisory board (CAB) comprising of racially and ethnically diverse patients and clinical and translational research experts. We presented CAB members with our proposed research plan to obtain feedback with particular attention to optimizing participant recruitment. This feedback and recommended solutions guided the development of the final research plan.
B. Direct observation.
The lead researcher (CLA), a surgeon trained in qualitative methodologies, conducted direct observation [42] of PROM administration and completion in the clinic’s waiting and check-in areas. PROMs were either self-administered or administered by clinic staff, via paper-and-pencil or tablet depending on patient preference. To maximize variation, observations were conducted at different times of the day spanning full clinic operating hours, on different days of the week corresponding with different physicians’ clinics, and from different vantage points. Observations were documented in written memos and moved into a data collection sheet. Data were qualitatively analyzed using design-oriented, rapid thematic analysis [43–45] including iterative reading, insight identification using inductive and abductive thinking, and affinity clustering [46–48]. Resulting clusters and early insights were reviewed by a second team member (LKB) to solidify inductive themes; these themes became the deductive themes used in the subsequent analysis of discovery interviews.
C. Discovery interviews.
We conducted semi-structured interviews with three participant groups (patients, caregivers, and clinic staff) to understand PROM usability, instrument design, perceived completion barriers, and completion strategies employed. We recruited adult patients with any hand and upper extremity condition including but not limited to those that are traumatic, infectious, congenital, oncologic, and rheumatologic in nature because this is the population for which the PROMIS Upper Extremity is validated in. Patients who identified as English-speaking and who had previously completed the PROMIS-UE were eligible for recruitment. Participants were screened for eligibility through standard clinic intake forms. Those who met inclusion criteria were approached for participation using a purposive sampling strategy [49] developed to ensure heterogeneity of literacy levels in our cohort. Specifically, we used education as a proxy for literacy and only recruited patients who reported an education level of high school or less. Caregivers accompanying patients eligible for study enrollment were also recruited. Finally, clinic staff involved in PROM completion processes were recruited. Reasons for non-participation and characteristics of the non-participant cohort were not recorded. All participants underwent a written informed consent process before interviews were conducted and were compensated $50 USD for their participation.
The goal of each step of HCD research is to identify insights that can be used in design decision-making, rather than to reach thematic saturation as is typically the goal for other qualitative research [50]. Because the rigor of HCD is derived from iterative processes rather than from saturation, we planned to recruit approximately ten participants for this phase [51, 52]. Fewer or more participants were recruited depending on how informative and rich interviews were.
Interviews were conducted by CLA, a researcher with no clinical relationship with patients and no employment relationship with clinic staff, either in-person or over the phone depending on participant preference during the COVID-19 pandemic. We used interview guides tailored to each participant group (S1 Dataset). To assess patient and caregiver participants’ literacy level, we administered the Rapid Estimate of Adult Literacy in Medicine Short Form (REALM-SF); a score of 6 or less, corresponding to a sixth grade reading level or less, indicated low literacy [53, 54]. Interviews were audio-recorded and documented using analytic memos written by the interviewer (CLA) within 24 hours after each interview. Subsequent analyses were conducted jointly by two research team members (CLA and LKB) to support reflexivity during analysis. Personas, a HCD technique that aims to summarize each user segment [55], were identified using iterative reading of memos and affinity clustering [47]. We conducted rapid framework-guided thematic analysis [43–45, 56] using the deductive themes identified during observation and identified additional themes inductively via affinity clustering and categorization. These themes were distilled into design challenges that were used to guide ideation. Our rapid thematic analysis aligns with the study’s HCD approach. Microsoft Word and Excel software (Redmond, WA) were used to conduct analyses.
D. Ideation workshop.
Participants in the ideation workshop included five members of our multidisciplinary team including domain experts, clinicians, and designers. Each design challenge identified in the prior interview phase was conceptualized as a “How Might We…?” (HMW) question [57]. The workshop included a review of research goals, presentation of findings, individual and group brainstorming around HMW questions, and final discussion. Although patients did not participate in the ideation workshop, we engaged co-creation and participatory processes [58, 59] by incorporating direct patient quotes and patient suggestions into the discussion. All ideas for potential prototype features were recorded using Miro, a visual collaboration software (San Francisco, California).
E. Prototyping.
Ideas from the workshop were translated into features that were synthesized into low-fidelity prototypes. Animations were illustrated as storyboards before execution. Prototypes and storyboards were produced in Figma (San Francisco, California). Prototypes underwent three rounds of iterative refinement by the research team, with the goal of improving multimedia components’ fidelity to PROMIS-UE’s questions and better meeting the design challenges.
F. Member-checking interviews.
To ensure the completeness, credibility, and validity of our findings, thereby increasing the rigor of our work [60], we conducted additional interviews that included member-checking of findings from prior phases. Only patients and caregivers were recruited, and participant eligibility and recruitment processes were the same as for discovery interviews. The interview guides were updated to include a section presenting and eliciting feedback on findings from prior phases. We aimed to recruit a minimum of five participants, sufficient for one phase of HCD testing [52].
G. Community feedback.
In our final phase of forward adaptation, we presented the near-final prototype to the CAB, elicited feedback on issues surrounding acceptability, and further refined the prototype to address those issues. This round of refinement incorporating the CAB’s feedback was the fourth and final round.
Results
A. Community-engaged study planning
The CAB provided several key recommendations that guided research planning. First, the inclusion criteria surrounding English fluency was relaxed to include participants who identify as English-speaking, even if not their primary language or if they were not perfectly fluent. Second, the CAB recommended including caregivers as an additional stakeholder and participant group. Finally, our initial participant compensation of $25 was deemed inadequate, so this was increased to $50.
B. Direct observation
We conducted 12 hours of observations over a two-week period. We identified eight themes surrounding PROM completion: 1) Survey fatigue: the extent to which it exists, what contributes to it, and how technology amplifies or mitigates it. 2) Patient-centered goal of PROMs: when is this compromised due to caregiver and clinic staff involvement? 3) Timing: how do factors surrounding timing influence how PROMs are completed? 4) Setting: what factors are involved in whether patients complete the survey at home or in the clinic, and which is preferred by patients, caregivers, and staff? 5) Role of the caregiver: in what specific ways do caregivers help or hinder PROM completion? 6) Stakeholder buy-in: in what ways is caregiver, patient, and clinic staff buy-in surrounding PROM completion limited? 7) Role of technology: how do the various technology options facilitate or create barriers to PROM completion, and how can they be leveraged? 8) Help: did patients not ask for help because they did not need help or because they did not want to ask for it?
C. Discovery interviews
We interviewed 11 participants (7 patients, 2 caregivers, 2 staff) during discovery interviews and 6 participants (5 patients, 1 caregiver) during member-checking interviews. Table 2 provides the demographics and characteristics for our entire cohort of 17 participants. Importantly, our cohort was of mixed literacy, with approximately half (46%) scoring low literacy on the REALM-SF.
Personas.
Our interviews revealed two distinct user personas: 1) the ‘get-it-done’ persona, i.e., the patient who wants to complete the PROMs as quickly as possible and just wants to get it over with; and 2) the ‘do-it-right’ persona, i.e., the patient who is thoughtful, wants to make sure he/she completes it accurately, may need more time to complete it, and may want help understanding the questions correctly. Our interviews also revealed three literacy personas: 1) the ‘illiterate’ persona, i.e., low literacy to the extent that they cannot complete the form independently; 2) the ‘slow reader’ persona, i.e., not completely illiterate but sufficiently low literacy that they struggle with PROMs, spending a long time in the waiting room attempting to complete it; and 3) the ‘literate’ persona, i.e., literacy level does not pose a challenge to PROM completion. We aimed to design a mPROMIS-UE that is well-suited for all user and literacy personas.
Location.
Between completing the PROMs at home versus at the clinic, the overwhelming preference by all participant groups was completion at home. The reasons cited included comfort, convenience, ability to take as much time as needed, increased privacy, minimizing time needed in a hospital/clinic, and avoidance of disrupting clinic flow. There were still instances in which patients were asked to complete their PROMs in clinic if they did not complete them before their appointment. We concluded that the mPROMIS-UE should allow for completion in both a home and a clinic setting.
Length and timing.
All participant groups, across user and literacy personas, highlighted the importance of a mPROMIS-UE that does not increase the time needed for completion.
Platforms and technology.
All participant groups preferred digital PROMs over paper PROMs. Smartphones, tablets, computers, or laptops were utilized preferentially by different patients and for different reasons, suggesting that the mPROMIS-UE should be adaptable to various digital platforms.
Customizability.
There was a wide range of reactions to the possibility of including multimedia audiovisual components. Some participants thought these components would be helpful, specifically for low literacy or cognitively impaired individuals. Other participants thought that PROMs were straightforward and would not benefit from multimedia components. Several worried that the addition of multimedia components, especially videos, would make the survey longer to complete. This highlighted the importance of designing an instrument that fits the needs of all user and literacy personas. Additionally, staff participants highlighted that they disliked having to administer more than one mode of PROM completion (e.g., paper and pencil vs. tablet vs. staff-administration), and preferred one uniform, streamlined process. Creating a single mPROMIS-UE that is customizable based on patient need or preference would address both patient and staff concerns.
Anxiety and shame.
Both anxiety and shame were themes associated with PROM completion. The first was anxiety due to unfamiliarity with PROMs. One patient explained, “When I first got the tablet in my hand, a little anxiety came over me because I’m not much into computers.” The second was shame or discomfort surrounding difficulty completing PROMs. One staff participant explained, “For the ones that don’t really read, you have some in their 50s or 60s that don’t really mind telling you that, but those ones in their 70s or 80s with their pride won’t tell you that, they’ll just say ‘I can’t do that.’” These findings reinforced the importance of creating a customizable mPROMIS-UE for all literacy levels that avoids the need to identify low literacy patients and minimizes anxiety, shame, and discomfort.
Caregivers’ roles in PROM completion.
We found that there are several types of caregivers, including family members, friends, nurses or nurse’s aides, transportation staff, and house managers. Caregivers assisted with PROM completion in various capacities, such as with holding the device, reading questions and responses aloud, clarifying the meaning of questions, filling in answers, or troubleshooting navigation-related issues. Caregivers’ ability to assist with PROM completion was variable, sometimes influenced by their own literacy or ability. This suggested that the mPROMIS-UE could be helpful even for patients who have caregivers.
D. Ideation workshop
The themes and insights from interviews were distilled into four design challenges: literacy, customizability, convenience, and avoiding discomfort (Table 3). Each idea for addressing the design challenge was given a specific name to facilitate identification and discussion of the various ideas (final column, Table 3).
E. Prototyping
Through three rounds of iterative review and discussion among the research team, ideas from the workshop were translated into six features that were built into a low-fidelity prototype: avatars, audio, GIFs (i.e., animated cartoon), an illustrated response scale (specifically, the COOP/WONCA overall health response scale that has been widely validated in low literacy populations [61]), icons to deploy audio and GIFs, and a progress indicator. The prototype was developed for a smartphone platform because this best addressed the location aspect of the convenience design challenge. Table 4 details which design challenge(s) each prototype feature was designed to address, how it did so, and examples of how these decisions were informed by findings from prior phases.
F. Member-checking interviews
The last six interviews included member-checking. Participants felt that the themes and design challenges resonated with them and were complete; this validated and confirmed the authenticity and completeness of our findings from prior phases. However, the response to potential prototype features varied. For example, while some participants responded enthusiastically to the idea of GIFs, others disliked this idea for various reasons. These responses and preferences reinforced the importance that mPROMIS-UE meets the customizability design challenge.
G. Community feedback
The CAB suggested that we modify certain features to improve instrument design and the user interface (e.g., make the progress indicator more visual), adjust GIFs so that they were more intuitive or easy to understand, and increase avatar diversity.
Screen captures of the final prototype are shown in Fig 1. This prototype corresponds with “mPROM v1.0” in our published protocol [13]. It provides instructions using text, icons, as well as audio. It asks patients to select among four avatars that are gender neutral but vary in skin tone, in order to represent racial and ethnic diversity. Each question is accompanied by audio and a GIF, and the GIF features the avatar that the patient selected. Each answer response is paired with an illustration, and also has an audio option. The audio, GIFs, and illustrated scale are deployed only if the patient clicks the corresponding icon. A progress bar indicates how many questions have been completed. Unanswered questions are highlighted if the user tries to submit an incomplete form.
Screen Captures of the multimedia Patient Reported Outcomes Measurement Information System Upper Extremity (mPROMIS-UE).
Discussion
By executing forward adaptation, the first stage of the MAP, we successfully adapted the PROMIS-UE to the mPROMIS-UE by addressing the challenges identified by a mixed literacy hand and upper extremity patient cohort. The mPROMIS-UE is customizable and responsive to our five user and literacy personas, convenient in terms of timing, location, and platform, and is designed to avoid potential shame/discomfort. When we shared our initial findings and prototype ideas with patients, caregivers, and our CAB, their responses indicated that our research goals and findings broadly resonated with them. They offered recommendations for improving the mPROMIS-UE in various domains that were integrated into the final mPROMIS-UE prototype. This work demonstrates the feasibility of adapting PROMs to multimedia versions that can be self-administered by mixed literacy populations.
There is limited literature on developing PROMs for mixed literacy populations. The only attempt to address the challenge of PROMs for mixed literacy populations that has been studied and reported on is the “Talking Touchscreen” (TT). TT is a platform for administering PROMs with audio recordings (but no visuals) [62, 63]. This was initially developed for health-related quality of life assessment in Latino cancer patients with low literacy levels [63]. The PROMIS Global Health is currently the only PROM available for administration via TT [64].
The mPROMIS-UE adds to the limited repository of PROMs (i.e., TT) designed for mixed literacy populations. It however is a prototype and will not be ready for implementation until the remaining stages of the MAP (back adaptation, qualitative evaluation, psychometric validation) [13] have been executed. Back adaptation will ensure that we have not altered the content of the PROMIS-UE during forward adaptation. Qualitative evaluation will ensure a well-designed user interface that increases the likelihood of successful implementation and long-term adoption and minimizes the risk of human error and operating inefficiencies. Psychometric validation is needed to confirm whether the mPROMIS-UE measures the construct it is intended to measure, in the intended population. Once these stages are completed, to advance a multimedia approach to capturing PRO data, we recommend future research efforts focus on dissemination, implementation, and expansion. This can include but is not limited to making mPROMIS-UE an official PROMIS instrument that is publicly available for widespread use, deploying it in hand and upper extremity clinics, integrating with the electronic health record, and collaborating with investigators from other specialties to adapt additional PROMs to mPROMs.
Implications
There are several implications to this work. To our best knowledge the mPROMIS-UE is the first PROM that incorporates visual components and was designed through a community-engaged and HCD approach. This allows the mPROMIS-UE to be self-administered by mixed literacy patients, who provided input in its development. mPROMIS-UE and the MAP provide a novel option for collecting PROs in English-speaking patients of mixed literacy, in the United States and other English-speaking countries in which the PROMIS-UE is validated. The mPROMIS-UE was designed to be customizable for all distinct user and literacy personas that we identified, which streamlines future implementation. The mPROMIS-UE was also designed to be self-administered by all user and literacy personas. This reduces the need for staff-administration that is labor intensive and expensive [8], and also perceived as undesirable by our patients, caregivers, and clinic staff. As PROs are adopted increasingly in practice and research, mPROMIS-UE and mPROMs broadly may be a preferred, practical, and cost-effective option for all stakeholders. In addition, this study demonstrated the feasibility of the first stage of the MAP, forward adaptation [13].
Although the primary goal of the mPROMIS-UE was to meet the needs of mixed literacy populations, particularly in support of low literacy patients, our findings suggest that there may be secondary benefits. For example, there is known variability in the comprehension and interpretation of the PROMIS-UE questions [65]. Participants noted that multimedia components can help address ambiguity of certain questions, so multimedia components have the potential to improve the validity and reliability of PROMs for use in for the general population. The audiovisual cues in a well-designed mPROM may also facilitate understanding for patients with cognitive impairments. Additional research is needed to explore these potential secondary benefits.
Limitations
One limitation related to the scope of our study is that we may have missed some relevant perspectives. For example, patients with cognitive disabilities or with poor English proficiency may have perspectives that differ from the broader low print literacy population. However, because cognitive disabilities and poor English proficiency are related but distinct barriers to PROM completion from low print literacy [66], we chose to focus our recruitment strategy on ensuring variation in print literacy levels. Future research should examine how text-based PROMs pose challenges to cognitively impaired or non-English speaking patients, as well as whether multimedia instruments are an appropriate or ideal solution. Additionally, our community advisory board included only eight members, so our community input beyond those directly associated with the study came from a relatively small number of individuals.
Conclusions
Development of the mPROMIS-UE is the first step towards expanding clinicians’ and researchers’ ability to capture PROs in mixed literacy hand and upper extremity patient populations. More broadly, it demonstrates the feasibility of adapting existing text-based PROMs to mPROMs. As such, this study represents an advancement towards a more equitable and inclusive paradigm in PRO measurement whereby the standard instruments are available in user-accessible multimedia versions.
Supporting information
S1 Dataset. Minimal data set including observation data, discovery interview guides, ideation workshop outline, and member checking interview guide.
https://doi.org/10.1371/journal.pone.0304351.s001
(DOCX)
Acknowledgments
We thank Claudia Udler and Hayelin Choi for their invaluable contributions as the social design and graphic design consultants, respectively, of our research team. Our previous protocol was published in PLOS ONE as a Registered Report Protocol entitled, "Developing a protocol for adapting multimedia patient-reported outcomes measures for low literacy patients," and this work is the corresponding Registered Report. The only change to the author list is the addition of Allan Fong in this work.
References
- 1. Stephens RJ, Hopwood P, Girling DJ, Machin D. Randomized trials with quality of life endpoints: are doctors’ ratings of patients’ physical symptoms interchangeable with patients’ self-ratings? Qual Life Res. 1997;6(3):225–36. pmid:9226980
- 2. Basch E. The missing voice of patients in drug-safety reporting. N Engl J Med. 2010;362(10):865–9. pmid:20220181
- 3. Basch E, Jia X, Heller G, Barz A, Sit L, Fruscione M, et al. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J Natl Cancer Inst. 2009;101(23):1624–32. pmid:19920223
- 4. Health USDo Human Services FDACfDE, Research Health USDo, Human Services FDACfBE Research, et al. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4:79. pmid:17034633
- 5.
Kirsch IS, Jungeblut A, Jenkins L, Kolstad A. Adult Literacy in America: A First Look at the Findings of the National Adult Literacy Survey. 3rd ed: National Center for Education Statistics: U.S. Department of Education, Office of Educational Research and Improvement; 2002.
- 6. Adult Literacy in the United States: U.S. Department of Education NCES; 2019 [Available from: https://nces.ed.gov/pubs2019/2019179/index.asp].
- 7.
Kirsch IS. Adult Literacy in America: A First Look at the Results of the National Adult Literacy Survey. 2nd ed: National Center for Education Statistics; 1993.
- 8.
Spilker B. Quality of Life and Pharmacoeconomics in Clinical Trials. 2nd ed: Lippincott Williams & Wilkins; 1995.
- 9. Yost KJ, Webster K, Baker DW, Jacobs EA, Anderson A, Hahn EA. Acceptability of the talking touchscreen for health literacy assessment. J Health Commun. 2010;15 Suppl 2(Suppl 2):80–92. pmid:20845195
- 10. Baker DW, Parker RM, Williams MV, Pitkin K, Parikh NS, Coates W, et al. The health care experience of patients with low literacy. Arch Fam Med. 1996;5(6):329–34. pmid:8640322
- 11. Parikh NS, Parker RM, Nurss JR, Baker DW, Williams MV. Shame and health literacy: the unspoken connection. Patient Educ Couns. 1996;27(1):33–9. pmid:8788747
- 12. Chang EM, Gillespie EF, Shaverdian N. Truthfulness in patient-reported outcomes: factors affecting patients’ responses and impact on data quality. Patient Relat Outcome Meas. 2019;10:171–86. pmid:31354371
- 13. Long C, Beres LK, Wu AW, Giladi AM. Developing a protocol for adapting multimedia patient-reported outcomes measures for low literacy patients. PLoS One. 2021;16(6):e0252684. pmid:34086774
- 14. Ootes D, Lambers KT, Ring DC. The epidemiology of upper extremity injuries presenting to the emergency department in the United States. Hand (New York, NY). 2012;7(1):18–22.
- 15. Jain NB, Higgins LD, Losina E, Collins J, Blazar PE, Katz JN. Epidemiology of musculoskeletal upper extremity ambulatory surgery in the United States. BMC Musculoskelet Disord. 2014;15:4. pmid:24397703
- 16. Giladi AM, Chung KC. Measuring outcomes in hand surgery. Clin Plast Surg. 2013;40(2):313–22. pmid:23506772
- 17. Marks M. Which patient-reported outcomes shall we use in hand surgery? J Hand Surg Eur Vol. 2020;45(1):5–11. pmid:31690166
- 18. Lloyd-Hughes H, Geoghegan L, Rodrigues J, Peters M, Beard D, Price A, et al. Systematic Review of the Use of Patient Reported Outcome Measures in Studies of Electively-Managed Hand Conditions. J Hand Surg Asian Pac Vol. 2019;24(3):329–41. pmid:31438799
- 19. Hewlett SA. Patients and clinicians have different perspectives on outcomes in arthritis. The Journal of rheumatology. 2003;30(4):877–9. pmid:12672220
- 20. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Academic medicine: journal of the Association of American Medical Colleges. 2014;89(9):1245–51. pmid:24979285
- 21. Shipp MM, Thakkar MY, Sanghavi KK, Means KR Jr., Giladi AM. Disparities Limit the Effect and Benefit of a Web-Based Clinic Intake System. Orthopedics. 2021;44(3):e434–e9. pmid:34039210
- 22. Baltimore City MD: County Health Rankings & Roadmaps; [Available from: https://www.countyhealthrankings.org/explore-health-rankings/maryland/baltimore-city?year= 2019].
- 23.
Baltimore, MD: DATAUSA; [Available from: https://datausa.io/profile/geo/baltimore-md/].
- 24. Kaat AJ, Buckenmaier CT 3rd, Cook KF, Rothrock NE, Schalet BD, Gershon RC, et al. The expansion and validation of a new upper extremity item bank for the Patient-Reported Outcomes Measurement Information System(R) (PROMIS). J Patient Rep Outcomes. 2019;3(1):69.
- 25. Jayakumar P, Teunis T, Vranceanu AM, Lamb S, Williams M, Ring D, et al. Construct Validity and Precision of Different Patient-reported Outcome Measures During Recovery After Upper Extremity Fractures. Clin Orthop Relat Res. 2019;477(11):2521–30. pmid:31490347
- 26. Doring AC, Nota SP, Hageman MG, Ring DC. Measurement of upper extremity disability using the Patient-Reported Outcomes Measurement Information System. The Journal of hand surgery. 2014;39(6):1160–5. pmid:24799143
- 27. Speck RM, Ye X, Bernthal NM, Gelhorn HL. Psychometric properties of a custom Patient-Reported Outcomes Measurement Information System (PROMIS) physical function short form and worst stiffness numeric rating scale in tenosynovial giant cell tumors. J Patient Rep Outcomes. 2020;4(1):61. pmid:32676941
- 28. England DL, Miller TA, Stevens PM, Campbell JH, Wurdeman SR. Assessment of a 9-item PROMIS Upper Extremity Instrument among Individuals with Upper Limb Amputation. American journal of physical medicine & rehabilitation / Association of Academic Physiatrists. 2020.
- 29. Yoon AP, Kaur S, Chou CH, Chung KC, Malay S, Shauver M, et al. Reliability and Validity of Upper Extremity Patient-Reported Outcome Measures in Assessing Traumatic Finger Amputation Management. Plast Reconstr Surg. 2020;145(1):94e–105e. pmid:31592952
- 30. Kaat AJ, Rothrock NE, Vrahas MS, O’Toole RV, Buono SK, Zerhusen T, Jr., et al. Longitudinal Validation of the PROMIS Physical Function Item Bank in Upper Extremity Trauma. J Orthop Trauma. 2017;31(10):e321–e6. pmid:28938284
- 31. Gausden EB, Levack AE, Sin DN, Nwachukwu BU, Fabricant PD, Nellestein AM, et al. Validating the Patient Reported Outcomes Measurement Information System (PROMIS) computerized adaptive tests for upper extremity fracture care. J Shoulder Elbow Surg. 2018;27(7):1191–7. pmid:29567038
- 32. Iyer S, Koltsov JCB, Steinhaus M, Ross T, Stein D, Yang J, et al. A Prospective, Psychometric Validation of National Institutes of Health Patient-Reported Outcomes Measurement Information System Physical Function, Pain Interference, and Upper Extremity Computer Adaptive Testing in Cervical Spine Patients: Successes and Key Limitations. Spine (Phila Pa 1976). 2019;44(22):1539–49. pmid:31689249
- 33. Rojas EO, Glass N, Owens J, Anthony CA, Bollier M, Wolf BR, et al. Performance of the PROMIS in Patients Undergoing 3 Common Elbow Procedures. Orthop J Sports Med. 2019;7(6):2325967119852595. pmid:31245430
- 34. Hajewski CJ, Glass NA, Westermann RW, Bollier M, Wolf BR, Hettrich C. Performance of the PROMIS After Operative Interventions for Shoulder Instability. Orthop J Sports Med. 2019;7(5):2325967119846920. pmid:31205969
- 35. Anthony CA, Glass N, Hancock K, Bollier M, Hettrich CM, Wolf BR. Preoperative Performance of the Patient-Reported Outcomes Measurement Information System in Patients With Rotator Cuff Pathology. Arthroscopy. 2017;33(10):1770–4 e1. pmid:28655475
- 36. Anthony CA, Glass NA, Hancock K, Bollier M, Wolf BR, Hettrich CM. Performance of PROMIS Instruments in Patients With Shoulder Instability. Am J Sports Med. 2017;45(2):449–53. pmid:28146398
- 37. Summary Table for NIH CDE Initiatives: National Library of Medicine; 2020 [Available from: https://wayback.archive-it.org/org]-350/20200602041019/https://www.nlm.nih.gov/cde/summary_table_1.html.
- 38.
Item Response Theory: HealthMeasures; [Available from: https://www.healthmeasures.net/resource-center/measurement-science/item-response-theory].
- 39. Hays RD, Lipscomb J. Next steps for use of item response theory in the assessment of health outcomes. Qual Life Res. 2007;16 Suppl 1:195–9. pmid:17351825
- 40. Fries JF, Krishnan E, Rose M, Lingala B, Bruce B. Improved responsiveness and reduced sample size requirements of PROMIS physical function scales with item response theory. Arthritis Res Ther. 2011;13(5):R147. pmid:21914216
- 41. Bazzano AN, Yan SD, Martin J, Mulhern E, Brown E, LaFond A, et al. Improving the reporting of health research involving design: a proposed guideline. BMJ Glob Health. 2020;5(2):e002248. pmid:32133198
- 42. Mack N, Woodsong C, MacQueen K, Guest G, Namey E. Qualitative Research Methods: A Data Collector’s Field Guide: Family Health International; 2005.
- 43. Vindrola-Padros C, Johnson GA. Rapid Techniques in Qualitative Research: A Critical Review of the Literature. Qual Health Res. 2020;30(10):1596–604. pmid:32667277
- 44. Lewinski AA, Crowley MJ, Miller C, Bosworth HB, Jackson GL, Steinhauser K, et al. Applied Rapid Qualitative Analysis to Develop a Contextually Appropriate Intervention and Increase the Likelihood of Uptake. Medical care. 2021;59(Suppl 3):S242–S51. pmid:33976073
- 45. Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, et al. Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implement Sci. 2019;14(1):11. pmid:30709368
- 46.
Tolley B. Traditional Socio-Behavioral Research And Human-Centered Design: Similarities, Unique Contributions and Synergies. FHI 3602017 [Available from: https://theimpt.org/traditional-socio-behavioral-research-and-human-centered-design-similarities-unique-contributions-and-synergies/].
- 47. Dam RF, Siang TY. Affinity Diagrams: How to Cluster Your Ideas and Reveal Insights. Interaction Design Foundation—IxDF 2022 [Available from: https://www.interaction-design.org/literature/article/affinity-diagrams-learn-how-to-cluster-and-bundle-ideas-and-facts].
- 48. Kolko J. Abductive Thinking and Sensemaking: The Drivers of Design Synthesis. Design Issues. 2010;26(1).
- 49. Martinez-Mesa J, Gonzalez-Chica DA, Duquia RP, Bonamigo RR, Bastos JL. Sampling: how to select participants in my research study? An Bras Dermatol. 2016;91(3):326–30. pmid:27438200
- 50. Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52(4):1893–907. pmid:29937585
- 51. Macefield R. How To Specify the Participant Group Size for Usability Studies: A Practitioner’s Guide. J Usability Stud. 2009;5(1):34–45.
- 52. Nielsen J. Why You Only Need to Test with 5 Users: Nielsen Norman Group; 2000 [Available from: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/].
- 53. Arozullah AM, Yarnold PR, Bennett CL, Soltysik RC, Wolf MS, Ferreira RM, et al. Development and validation of a short-form, rapid estimate of adult literacy in medicine. Medical care. 2007;45(11):1026–33. pmid:18049342
- 54. Bass PF 3rd, Wilson JF, Griffith CH. A shortened instrument for literacy screening. J Gen Intern Med. 2003;18(12):1036–8. pmid:14687263
- 55. Salminen J, Nielsen L, Bahloul M, Jorgensen RG, Santos JM, Jung SG, et al. Persona preparedness: a survey instrument for measuring the organizational readiness for deploying personas. Inf Technol Manag. 2022:1–26. pmid:36119410
- 56. Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutrona SL, White BS, et al. Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR). Implement Sci. 2021;16(1):67. pmid:34215286
- 57.
The Field Guide to Human-Centered Design: IDEO.org; [Available from: https://www.designkit.org/resources/1].
- 58. Crosby S, Younie S, Williamson I, Laird K. Evaluating approaches to designing effective Co-Created hand-hygiene interventions for children in India, Sierra Leone and the UK. PLoS One. 2020;15(9):e0239234. pmid:32931509
- 59. Wolstenholme D, Kidd L, Swift A. Co-creation and co-production in health service delivery: what is it and what impact can it have? Evid Based Nurs. 2019;22(4):97–100. pmid:31462428
- 60. Tobin GA, Begley CM. Methodological rigour within a qualitative framework. J Adv Nurs. 2004;48(4):388–96. pmid:15500533
- 61. Van Weel C. Functional status in primary care: COOP/WONCA charts. Disabil Rehabil. 1993;15(2):96–101. pmid:8513163
- 62. Cella D, Hahn EA, Jensen SE, Butt Z, Nowinski C, Rothrock N. Methodological Issues in the Selection, Administration and Use of Patient-Reported Outcomes in Performance Measurement in Health Care Settings. National Quality Forum. 2012.
- 63. Hahn EA, Cellal D, Dobrez DG, Shiomoto G, Taylor SG, Galvez AG, et al. Quality of life assessment for low literacy Latinos: a new multimedia program for self-administration. J Oncol Manag. 2003;12(5):9–12. pmid:14619989
- 64. PROMIS Short Form v1.1—Global Health—Talking Touchscreen: HealthMeasures; 2021 [Available from: https://www.healthmeasures.net/index.php?option=com_instruments&view=measure&id=617&Itemid=992].
- 65. Azad CL, Beres LK, Wu AW, Giladi AM. Patient interpretations vary for questions in the Patient-Reported Outcomes Measurement Information System Upper Extremity. J Hand Surg Eur Vol. 2023:17531934231162738. pmid:37005736
- 66. Long C, Beres LK, Wu AW, Giladi AM. Patient-level barriers and facilitators to completion of patient-reported outcomes measures. Qual Life Res. 2022;31(6):1711–8. pmid:34533759