Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Graphical user interface design to improve understanding of the patient-reported outcome symptom response

  • Mangyeong Lee ,

    Contributed equally to this work with: Mangyeong Lee, Danbee Kang, Yeongrae Joi

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

    Affiliations Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea, Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯
  • Danbee Kang ,

    Contributed equally to this work with: Mangyeong Lee, Danbee Kang, Yeongrae Joi

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

    Affiliations Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea, Department of Clinical Research Design and Evaluation, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯
  • Yeongrae Joi ,

    Contributed equally to this work with: Mangyeong Lee, Danbee Kang, Yeongrae Joi

    Roles Conceptualization, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

    Affiliation Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea

    ⨯
  • Junghee Yoon,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Department of Clinical Research Design and Evaluation, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯
  • Youngha Kim,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea

    ⨯
  • Jinhwang Kim,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea

    ⨯
  • Minwoong Kang,

    Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

    Affiliation Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯
  • Dongryul Oh,

    Roles Conceptualization, Funding acquisition, Methodology, Supervision, Writing – review & editing

    Affiliation Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea

    ⨯
  • Soo-Yong Shin ,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    jcho@skku.edu (JC); sy.shin@skku.edu (SYS)

    Affiliation Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯
  • Juhee Cho

    Roles Conceptualization, Funding acquisition, Methodology, Supervision, Writing – review & editing

    jcho@skku.edu (JC); sy.shin@skku.edu (SYS)

    Affiliations Center for Clinical Epidemiology, Samsung Medical Center, Seoul, Korea, Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Korea, Department of Clinical Research Design and Evaluation, SAIHST, Sungkyunkwan University, Seoul, Korea

    ⨯

Abstract

Background

Symptom monitoring application (SMA) has clinical benefits to cancer patients but patients experience difficulties in using it. Few studies have identified which types of graphical user interface (GUI) are preferred by cancer patients for using the SMA.

Methods

This is a cross-sectional study aimed to identify preferred GUI among cancer patients to use SMA. Total of 199 patients were asked to evaluate 8 types of GUIs combining text, icon, illustration, and colors using mixed-methods. Subgroup analyses were performed according to age and gender.

Results

The mean age of the patients was 57 and 42.5% was male. The most preferred GUI was “Text + Icon + Color” (mean = 4.43), followed by “Text + Icon” (mean = 4.39). Older patients (≥ 60 years) preferred “Text + Icon” than younger patients (p for interaction < 0.01). Simple and intuitive text and icons were the most useful GUI for cancer patients to use the SMA.

Conclusion

Simple and intuitive text and icons were the most useful GUI for cancer patients to use the SMA. Researchers need to be careful when applying realistic face drawings to cancer symptom monitoring applications because they can recall negative images of cancer.

Citation: Lee M, Kang D, Joi Y, Yoon J, Kim Y, Kim J, et al. (2023) Graphical user interface design to improve understanding of the patient-reported outcome symptom response. PLoS ONE 18(1): e0278465. https://doi.org/10.1371/journal.pone.0278465

Editor: Daswin De Silva, La Trobe University - Melbourne Campus: La Trobe University, AUSTRALIA

Received: February 11, 2022; Accepted: October 18, 2022; Published: January 24, 2023

Copyright: © 2023 Lee et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because of ethical restrictions, as participants of this study did not agree for their data to be shared publicly. Data are available from the Ethics Committee of the Samsung Medical Center (contact via FAX +82 2-3410-3617) for researchers who meet the criteria for access to confidential data. The data underlying the results presented in the study are also available from jcho@skku.edu.

Funding: This study was supported by grants from the National Research Foundation of Korea to DO [No. 2017R1A2B4007720] and to JC [2021R1A2C2011083], and by Samsung Medical Center in the form of a grant to JC [No. SMX1210381]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: ICT, Information and Communication Technology; SMA, Symptom Monitoring Applications; Apps, Applications; GUI, Graphic User Interface; Full HD, Full High Definition; Tab, Tablet PC; SD, Standard Deviation; CI, Confidence Intervals

Introduction

Cancer is a leading cause of death worldwide, and an estimated 19.3 million new cancer cases occurred in 2020 [1]. Cancer patients experience a wide range of physical and psychological symptoms owing to anticancer therapies or the disease itself [2, 3]. Acute symptoms, especially those related to anticancer treatment, typically manifest during the interim period, and waiting to capture these during clinic visits might cause peak symptom distress to be missed. However, poorly controlled symptoms can lead to unplanned clinic or emergency department visits and unplanned hospitalizations [4–7]. In addition, a recent clinical trial demonstrated that active symptom management using mobile technology has contributed to improved survival rates [8, 9]. Therefore, interest in monitoring systems that use patient-reported symptoms through mobile devices has been increasing [10].

There have been several mobile symptom monitoring applications (SMA) that can facilitate the timely detection of patients’ symptoms [9, 11–14]; however, patients with low digital health literacy have difficulties in effectively utilizing and interacting with technologies [15–17]. To resolve this problem, the U.S. Food and Drug Administration recommends applying human factors that identify and address end-user requirements, mobile system design, and functionality to users’ capabilities, needs, and expectations in the development process for digital health services [18–21]. Among these human factors, graphical user interfaces (GUIs), such as the layout, font style, interaction element, and task scenario could affect the patients’ usability for SMAs [11, 22]. GUI is a form of user interface that allows users to interact with electronic devices through graphical icons and visual indicators, instead of text-based interfaces [23]. The elements on the screen, such as color, size, and design, affect user satisfaction and understanding [24].

Previous studies have shown that GUIs are strongly preferred over text-based interfaces for novice and low-literacy users [25], and the extensive use of graphics is recommended for low-literacy communities [26]. Several studies related to interface design were conducted regarding questionnaire layout, statement expression, position of image, or color brightness [27–29]. Still, few studies have evaluated different GUI designs of response scales for patients’ understanding and preferences [20, 30–33]. This study aims to identify preferred GUI among cancer patients to use SMA.

Methods

GUI design

“Vomiting” was chosen to evaluate which GUI is the most helpful in understanding symptoms, because it is one of the most common symptoms in cancer patients [34]. The symptoms (vomiting) were described on a 5-point Likert scale (0 = none, 1 = mild, 2 = moderate, 3 = severe, and 4 = very severe).

Based on a previous study that compared five styles of visuals to examine one that makes information increasingly accessible to patients [35], eight types of GUIs were developed by combining texts, icons, illustrations, and colors, as shown in Fig 1: 1) “Text only”; 2) “Text and Icon”; 3) “Text and Illustration”; 4) “Text and Real image”; 5) “Text and Scaled color”; 6) “Text, Icon, and Scaled color”; 7) “Text, Illustration, and Scaled color”; and 8) “Text, Real image, and Scale color”. In the design, we considered ‘redundancy gain’ which can evoke responses more quickly and accurately by providing separate stimuli redundantly such as color and shape [36]. In terms of shape, we applied metaphorical images implying the meaning of each response scale, with different styles of the presentation [37]. The “Real image” consists of original face photos to represent the given condition of cancer patients with facial expressions [38]. “Illustration” is a simple line drawing based on face photos. Icon represents the pictorial symptom scales, because the smiley is well known as an effective visual scale tool, regardless of a patient’s literacy or language skills [39]. In addition, we included color symbolism to help patients perceive symptoms, as in previous studies [40, 41]. The red and yellow colors were chosen. Red is often associated with negative emotions, such as danger, stop [42, 43], or mistakes [44]. Yellow is often associated with positive emotions, such as cheerful [45, 46]. These colors are also commonly used in health apps [47, 48].

thumbnail
Download:
Fig 1. Graphical user interface (GUI) design to help understand the symptoms.

https://doi.org/10.1371/journal.pone.0278465.g001

The eight types of GUIs were designed by a head and neck oncologist, a behavior scientist, two cancer educators, and a visual graphic designer. The GUIs were refined using iterative design testing, modification, and retesting processes. The designed GUIs were tested on a tablet with full HD resolution (1920px × 1200px) (Samsung Galaxy Tab A).

Study design and participants

A cross-sectional mixed-methods study was conducted of cancer patients attending the outpatient clinic at the Samsung Medical Center in Seoul, Korea, from April 2017 to September 2017. Subjects were eligible to participate in the study if they were 18 years of age or older, had a histologically confirmed diagnosis of cancer, had no evidence of physical or psychological problems at the time of the survey, and were able to read Korean.

A total of 199 patients agreed to participate in the study. Among them, people who had no information on age (N = 6) were excluded from the study. The final study group included 193 patients with cancer, as shown in Fig 2. The study was approved by the Institutional Review Board of Samsung Medical Center, and all patients provided written informed consent (No. SMC-2015-12-011).

thumbnail
Download:
Fig 2. Study design and participant flow.

https://doi.org/10.1371/journal.pone.0278465.g002

Measurement

A cross-sectional study involving both quantitative and qualitative methods was conducted by four trained researchers. For the quantitative phase, a tablet PC containing examples of each GUI was provided so that participants could answer questions based on their understanding of each type of GUI. Participants were also asked to choose the most helpful GUI for understanding the symptoms among the eight types of GUIs. Information regarding gender, age, highest completed level of education, type of cancer, income level, time since cancer diagnosis, and current treatment status were collected using a standardized questionnaire.

After the quantitative survey, the patients were asked to participate in the qualitative phase. If the patients agreed to participate, they were interviewed based on the questionnaires submitted. A total of eight patients agreed to participate in the survey, and the interviewer asked the patients the reasons they preferred or did not prefer a type of GUI. The interview took 10–20 min per patient (S1 Table).

Statistical methods

Descriptive statistics were used to report the characteristics of the participants and the mean and standard deviation (SD) score of each GUI. We used mixed-effect models for the repeated measurement of data in the eight types of GUIs. The mean differences and 95% CIs were calculated using a linear mixed model. Interaction analysis was used to evaluate whether the association of the type of GUI with understanding scores differed by age group (< 60 and ≥ 60 years). In South Korea, 55% of 25–64 year-olds have a higher level of educational attainment than older adults whose age is 55–64 years [49]. Therefore, older adults, in their 60s or older, were set as the low-literacy population. In addition, we compared the effects of GUI by gender.

All reported P-values were two-sided, and the significance level was set as 0.05. All analyses were performed using STATA version 14 (StataCorp LP, College Station, TX, USA).

Results

Characteristics of the study population

The mean age (SD) of the participants was 57.0 ± 10.9 years, and 42.5% were male (Table 1). In terms of clinical characteristics, 70% of the patients were under active treatment, and the median time since cancer diagnosis was one year (range of 0–12 years).

thumbnail
Download:
Table 1. Characteristics of the study population.

https://doi.org/10.1371/journal.pone.0278465.t001

Comparison of patient preferences according to graphical user interface (GUI) designs

As shown in Fig 3, most patients preferred “Text + Icon,” regardless of the presence (64.77%) or absence of (67.88%) scale colors.

thumbnail
Download:
Fig 3. Proportion of most helpful to understand the symptoms among the graphical user interface (GUI) designs.

(a) Without Scaled Color, (b) With Scaled Color.

https://doi.org/10.1371/journal.pone.0278465.g003

In terms of scores for help to understand (Table 2), the highest understanding score was for “Text + Icon + Color” (mean = 4.43), followed by “Text + Icon” (mean = 4.39).

thumbnail
Download:
Table 2. Linear mixed linear regression model to compare the score of understanding of terms according to graphical user interface (GUI) design.

https://doi.org/10.1371/journal.pone.0278465.t002

The lowest understanding score was for “Text + Illustration” (mean = 2.14). When we compared to “Text only,” although “Text + Icon” (Coefficient = 1.18; 95% confidence interval (CI) = 0.97, 1.39) and “Text + Icon+ Color” (Coefficient = 1.22; 95% CI = 1.01, 1.44) had significantly higher understanding scores, “Text + Illustration” (Coefficient = -1.07; 95% CI = -1.28, -0.85) and “Text + Real image” (Coefficient = -0.92; 95% CI = -1.13, -0.70) yielded significantly lower understanding scores, even with color (Table 2).

Although some patients reported that “Icon” was frivolous for severe symptoms, many patients preferred it because it may enhance the understanding of the symptom scale and is a familiar GUI. Otherwise, patients felt emotional discomfort when viewing illustrations or real images, because these images reminded them of how sick they were and distressed them (see S1 Table). In addition, patients reported that it was difficult to intuitively recognize scale changes in illustrations and real images. Patients also mentioned that scaled colors helped them recognize the difference between mild and severe conditions more clearly.

Sex and age differences in the preferences

While the preferences were not different between males and females, there was a difference according to the age group. A significantly higher score for “Text + Icon” was consistently observed in all age groups and genders. As shown in Table 3, participants aged ≥ 60 years had a higher score than younger participants (p for interaction < 0.01). Younger patients had a lower score in “Text + Real image” than older patients (p for interaction < 0.01).

thumbnail
Download:
Table 3. Subgroup analysis for multivariable mixed linear regression to compare score of understanding of terms according to graphical user interface (GUI) design.

https://doi.org/10.1371/journal.pone.0278465.t003

Discussion

In this study, “Text + Icon + Color” was the most preferred GUI for improving understanding of their symptoms in cancer patients. Cancer patients also responded that “Text + Icon” was helpful to understand the severity of their symptoms even without scaled color. On the other hand, “Illustration” or “Real image” was associated with a lower understanding score. Especially, older patients preferred “Icon” compared to younger patients.

In this study, “Text + Icon + Color” was the most preferred GUI for improving understanding of their symptoms. As in previous studies [40, 41], colors increased the perception of symptoms experienced when compared with the non-color condition. However, when we compared understanding score of patients according to the color, and the differences were not statistically significant. “Icon” had a much greater impact on the patients’ preferences. Regardless of scaled color, “Icon” was preferred by the patients. A study regarding interface design of multimedia learning presented that shape with neutral color (grayscale) was most effective compared to the shape combined with color in learning process [50]. It might be due to that “Icon” which is based on pictographic presentation delivers more intuitive and concrete information than color symbolism.

On the other hands, illustration or real image was associated with lower understand score. As in a previous study [51], the detailed parts made patients feel redundant and did not provide additional information. In addition, cognitive overload can occur when the visuals contain too much demand [52]. ‘Cognitive Load Theory’ basically assumes that the human cognitive system has a limited working memory capacity and humans may feel difficult to take a task if cognitive efforts for the task exceed their limits [53]. In addition, some patients reported psychological repulsion as it directly reminded them that they had lost hair and how sick they were in the qualitative interview. Patients’ experience had a negative perception for “Illustration” or “Real image,” as it seemed to have cancer stigma [54]. According to a study with the public, strong identification with celebrity cancer death through the mass media was associated with negative emotional reactions, resulting in stigma-related perceptions [55]. Thus, in our findings, cancer patients might experience negative feelings from identification with the displayed images reminiscent of pain and hair loss. This result suggests that while providing GUIs to cancer patients, cancer stigma should be considered.

In subgroup analysis, older patients preferred “Icon” to “Text only” more often than younger patients. Regarding the impact of scaled colors, younger (Coefficient = 1.16; 95% CI = -0.89, 1.43) and female (Coefficient = 1.32; 95% CI = -1.04, -1.59) participants exhibited a higher score for help to understand than older (Coefficient = 1.32; 95% CI = 0.98, 1.67) and male (Coefficient = 1.10; 95% CI = 0.77, 1.42) participants when “Icon” and colors were used together. Previous studies have also shown that older and male individuals are color deficient compared to younger and female individuals [56–58]. The simple shape of the GUI helps to understand health information in low-literacy patients [59–61]. Previous research has shown that low-literate people are more easily distracted by redundant details than well-literate people, which would explain our findings [51, 62]. Based on the recent education level of Koreans [49, 63], age could be considered a potential indicator of low literacy. Although well-designed visuals could have a positive effect on how information is perceived, not all visuals are effective because literate and low-literate people differ in perception preference [64, 65].

This study had several limitations. First, there is no standard tool for measuring the understanding of GUIs in patients with cancer. Thus, the survey used in this study does not prove its validity or reliability. Second, preference was measured using only a single symptom, vomiting. Therefore, our results may not be generalizable to other types of cancer symptoms or patients with different diseases. Third, we did not measure the patients’ literacy level. Fourth, cancer patients were recruited from a hospital located in the Seoul metropolitan area. Accordingly, future research should be conducted using different patient samples.

In conclusion, simple and intuitive text and icon was the most useful GUI for cancer patients to report their symptoms. This study could improve the GUI design guidelines for mobile healthcare applications (apps) for cancer patients. Design recommendations and useful features that can be applied during mobile app design for cancer patients were identified. The results of this study demonstrated how the potential use and acceptance of different types of GUI features could be influenced not only by the usefulness of specific functions but also by the current context of use. Further research is necessary to investigate the effect of GUI design with diverse patients and contexts.

Supporting information

S1 Table. Findings from qualitative interview.

https://doi.org/10.1371/journal.pone.0278465.s001

(DOCX)

References

  1. 1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021;71(3):209–49. pmid:33538338
  2. 2. Wu H-S, Harden JK. Symptom Burden and Quality of Life in Survivorship: A Review of the Literature. Cancer Nurs. 2015;38(1):E29–E54. pmid:24831042
  3. 3. Yamagishi A, Morita T, Miyashita M, Kimura F. Symptom Prevalence and Longitudinal Follow-up in Cancer Outpatients Receiving Chemotherapy. J Pain Symptom Manage. 2009;37(5):823–30. pmid:18804946
  4. 4. Pandya C, Magnuson A, Flannery M, Zittel J, Duberstein P, Loh KP, et al. Association between Symptom Burden and Physical Function in Older Patients with Cancer. J Am Geriatr Soc. 2019;67(5):998–1004. pmid:30848838
  5. 5. Barbera L, Atzema C, Sutradhar R, Seow H, Howell D, Husain A, et al. Do Patient-Reported Symptoms Predict Emergency Department Visits in Cancer Patients? A Population-Based Analysis. Ann Emerg Med. 2013;61(4):427–37. e5. pmid:23290526
  6. 6. Deshields TL, Potter P, Olsen S, Liu J. The Persistence of Symptom Burden: Symptom Experience and Quality of Life of Cancer Patients across One Year. Support Care Cancer. 2014;22(4):1089–96. pmid:24292095
  7. 7. Maio MD, Gallo C, Leighl NB, Piccirillo MC, Daniele G, Nuzzo F, et al. Symptomatic Toxicities Experienced During Anticancer Treatment: Agreement between Patient and Physician Reporting in Three Randomized Trials. J Clin Oncol. 2015;33(8):910–5. pmid:25624439
  8. 8. Basch E, Mody GN, Dueck AC. Electronic Patient-Reported Outcomes as Digital Therapeutics to Improve Cancer Outcomes. JCO Oncology Practice. 2020. pmid:32484724
  9. 9. Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom Monitoring with Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol. 2016;34(6):557. pmid:26644527
  10. 10. Dueck AC, Mendoza TR, Mitchell SA, Reeve BB, Castro KM, Rogak LJ, et al. Validity and Reliability of the US National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncol. 2015;1(8):1051–9. pmid:26270597
  11. 11. Aiyegbusi OL, Kyte D, Cockwell P, Marshall T, Dutton M, Walmsley-Allen N, et al. Development and Usability Testing of an Electronic Patient-Reported Outcome Measure (ePROM) System for Patients with Advanced Chronic Kidney Disease. Comput Biol Med. 2018;101:120–7. pmid:30130638
  12. 12. Bennett AV, Jensen RE, Basch E. Electronic Patient-Reported Outcome Systems in Oncology Clinical Practice. CA Cancer J Clin. 2012;62(5):337–47. Epub 2012/07/20. pmid:22811342.
  13. 13. Maguire R, Fox PA, McCann L, Miaskowski C, Kotronoulas G, Miller M, et al. The eSMART Study Protocol: A Randomised Controlled Trial to Evaluate Electronic Symptom Management Using the Advanced Symptom Management System (ASyMS) Remote Technology for Patients with Cancer. BMJ Open. 2017;7(5):e015016. Epub 2017/06/09. pmid:28592577; PubMed Central PMCID: PMC5734219.
  14. 14. Nasi G, Cucciniello M, Guerrazzi C. The Role of Mobile Technologies in Health Care Processes: The Case of Cancer Supportive Care. J Med Internet Res. 2015;17(2):e26. Epub 2015/02/14. pmid:25679446; PubMed Central PMCID: PMC4342745.
  15. 15. Wagner DA. What Happened to Literacy? Historical and Conceptual Perspectives on Literacy in UNESCO. J International Journal of Educational Development. 2011;31(3):319–23.
  16. 16. Kim H, Goldsmith JV, Sengupta S, Mahmood A, Powell MP, Bhatt J, et al. Mobile Health Application and e-Health Literacy: Opportunities and Concerns for Cancer Patients and Caregivers. J Cancer Educ. 2019;34(1):3–8. Epub 2017/11/16. pmid:29139070.
  17. 17. Kim H, Xie B. Health Literacy in the eHealth Era: A Systematic Review of the Literature. Patient Educ Couns. 2017;100(6):1073–82. Epub 2017/02/09. pmid:28174067.
  18. 18. US Food and Drug Administration. Guidance for Industry and Food and Drug Administration Staff: Applying Human Factors and Usability Engineering to Medical Devices 2016 [Accessed 02 Feb 2021]. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/applying-human-factors-and-usability-engineering-medical-devices.
  19. 19. US Food and Drug Administration. Guidance for Industry and Food and Drug Administration Staff: Software as a Medical Device (SaMD) Clinical Evaluation. 2017 [Accessed 03 Feb 2021]. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/software-medical-device-samd-clinical-evaluation.
  20. 20. Izard J, Hartzler A, Avery DI, Shih C, Dalkin BL, Gore JL. User-Centered Design of Quality of Life Reports for Clinical Care of Patients with Prostate Cancer. Surgery. 2014;155(5):789–96. Epub 2014/05/03. pmid:24787105; PubMed Central PMCID: PMC4237217.
  21. 21. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A User-Centered Model for Designing Consumer Mobile Health (mHealth) Applications (Apps). J Biomed Inform. 2016;60:243–51. Epub 2016/02/24. pmid:26903153; PubMed Central PMCID: PMC4837063.
  22. 22. Aiyegbusi OL. Key Methodological Considerations for Usability Testing of Electronic Patient-Reported Outcome (ePRO) Systems. Qual Life Res. 2020;29(2):325–33. pmid:31691202
  23. 23. Quinde M, Khan N. An Improved Model for GUI Design of mHealth Context-Aware Applications. International Conference of Design, User Experience, and Usability. 2018.
  24. 24. Pereira S, Hassler S, Hamek S, Boog C, Leroy N, Beuscart-Zephir MC, et al. Improving Access to Clinical Practice Guidelines with an Interactive Graphical Interface Using an Iconic Language. BMC Med Inform Decis Mak. 2014;14:77. Epub 2014/08/28. pmid:25158762; PubMed Central PMCID: PMC4153004.
  25. 25. Medhi I, Patnaik S, Brunskill E, Gautama S, Thies W, Toyama K. Designing Mobile Interfaces for Novice and Low-Literacy Users. J ACM Transactions on Computer-Human Interaction. 2011;18(1):2.
  26. 26. McCaffery KJ, Dixon A, Hayen A, Jansen J, Smith S, Simpson JM. The Influence of Graphic Display Format on the Interpretations of Quantitative Risk Information among Adults with Lower Education and Literacy: A Randomized Experimental Study. Med Decis Making. 2012;32(4):532–44. Epub 2011/11/15. pmid:22074912.
  27. 27. Brosnan K, Grün B, Dolnicar S. Cognitive Load Reduction Strategies in Questionnaire Design. International Journal of Market Research. 2021;63(2):125–33.
  28. 28. Chee Kit Y, Choo Seah L, Wong Seok Y, Wan Mohd Nazmee Wan Z, editors. Gui Design Based on Cognitive Psychology: Theoretical, Empirical and Practical Approaches. 2012 8th International Conference on Computing Technology and Information Management (NCM and ICNIT); 2012 24–26 April 2012.
  29. 29. Lenzner T. Effects of Survey Question Comprehensibility on Response Quality. Field Methods. 2012;24(4):409–28.
  30. 30. Mirkovic J, Kaufman DR, Ruland CM. Supporting Cancer Patients in Illness Management: Usability Evaluation of a Mobile App. JMIR Mhealth Uhealth. 2014;2(3):e33. Epub 2014/08/15. pmid:25119490; PubMed Central PMCID: PMC4147703.
  31. 31. Kuijpers W, Giesinger JM, Zabernigg A, Young T, Friend E, Tomaszewska IM, et al. Patients’ and Health Professionals’ Understanding of and Preferences for Graphical Presentation Styles for Individual-Level EORTC QLQ-C30 Scores. Qual Life Res. 2016;25(3):595–604. Epub 2015/09/12. pmid:26353905; PubMed Central PMCID: PMC4759250.
  32. 32. Hartzler AL, Izard JP, Dalkin BL, Mikles SP, Gore JL. Design and Feasibility of Integrating Personalized PRO Dashboards into Prostate Cancer Care. J Am Med Inform Assoc. 2016;23(1):38–47. Epub 2015/08/12. pmid:26260247; PubMed Central PMCID: PMC5009933.
  33. 33. Seo M, Yoon J, Park T. GRACOMICS: Software for Graphical Comparison of Multiple Results with Omics Data. BMC Genomics. 2015;16:256. Epub 2015/04/19. pmid:25887412; PubMed Central PMCID: PMC4387734.
  34. 34. Naeim A, Dy SM, Lorenz KA, Sanati H, Walling A, Asch SM. Evidence-Based Recommendations for Cancer Nausea and Vomiting. J Clin Oncol. 2008;26(23):3903–10. Epub 2008/08/09. pmid:18688059.
  35. 35. Moll JM. Doctor-Patient Communication in Rheumatology: Studies of Visual and Verbal Perception Using Educational Booklets and Other Graphic Material. Ann Rheum Dis. 1986;45(3):198–209. Epub 1986/03/01. pmid:3954469; PubMed Central PMCID: PMC1001852.
  36. 36. Shepherdson P, Miller J. Redundancy Gain in Semantic Categorisation. Acta Psychol (Amst). 2014;148:96–106. pmid:24508611
  37. 37. Marcus A. Metaphor Design in User Interfaces. SIGDOC Asterisk J Comput Doc. 1998;22(2):43–57.
  38. 38. Sirintawat N, Sawang K, Chaiyasamut T, Wongsirichat N. Pain Measurement in Oral and Maxillofacial Surgery. J Dent Anesth Pain Med. 2017;17(4):253–63. Epub 2018/01/20. pmid:29349347; PubMed Central PMCID: PMC5766084.
  39. 39. Garra G, Singer AJ, Taira BR, Chohan J, Cardoz H, Chisena E, et al. Validation of the Wong-Baker FACES Pain Rating Scale in Pediatric Emergency Department Patients. Acad Emerg Med. 2010;17(1):50–4. Epub 2009/12/17. pmid:20003121.
  40. 40. Wiercioch-Kuzianik K, Babel P. Color Hurts. The Effect of Color on Pain Perception. Pain Med. 2019. Epub 2019/01/17. pmid:30649442.
  41. 41. Grady M, Katz LB, Cameron H, Levy BL. Diabetes App-Related Text Messages from Health Care Professionals in Conjunction with a New Wireless Glucose Meter with a Color Range Indicator Improves Glycemic Control in Patients with Type 1 and Type 2 Diabetes: Randomized Controlled Trial. JMIR Diabetes. 2017;2(2):e19. Epub 2017/08/07. pmid:30291092; PubMed Central PMCID: PMC6238868.
  42. 42. Genschow O, Reutner L, Wanke M. The Color Red Reduces Snack Food and Soft Drink Intake. Appetite. 2012;58(2):699–702. Epub 2012/01/17. pmid:22245725.
  43. 43. Elliot AJ, Maier MA, Moller AC, Friedman R, Meinhardt J. Color and Psychological Functioning: The Effect of Red on Performance Attainment. J Exp Psychol Gen. 2007;136(1):154–68. Epub 2007/02/28. pmid:17324089.
  44. 44. Gnambs T, Appel M, Batinic B. Color Red in Web-Based Knowledge Testing. J Computers in Human Behavior. 2010;26(6):1625–31.
  45. 45. NAz K, Epps H. Relationship between Color and Emotion: A Study of College Students. J College Student. 2004;38(3):396.
  46. 46. Han S, Lee D. The Effects of Treatment Room Lighting Color on Time Perception and Emotion. J Phys Ther Sci. 2017;29(7):1247–9. Epub 2017/07/27. pmid:28744057; PubMed Central PMCID: PMC5509601.
  47. 47. Bosl W, Mandel J, Jonikas M, Ramoni RB, Kohane IS, Mandl KD. Scalable Decision Support at the Point of Care: A Substitutable Electronic Health Record App for Monitoring Medication Adherence. Interact J Med Res. 2013;2(2):e13. Epub 2013/07/24. pmid:23876796; PubMed Central PMCID: PMC3815431.
  48. 48. Grady M, Campbell D, MacLeod K, Srinivasan A, technology. Evaluation of a Blood Glucose Monitoring System with Automatic High-and Low-Pattern Recognition Software in Insulin-Using Patients: Pattern Detection and Patient-Reported Insights. J Journal of diabetes science. 2013;7(4):970–8. pmid:23911178
  49. 49. OECD. Education at a Glance 2018. OECD Indicators. 2018. https://doi.org/10.1787/eag-2018-en
    • 50. Plass JL, Heidig S, Hayward EO, Homer BD, Um E. Emotional Design in Multimedia Learning: Effects of Shape and Color on Affect and Learning. Learning and Instruction. 2014;29:128–40.
    • 51. van Beusekom M, Bos M, Wolterbeek R, Guchelaar HJ, van den Broek J. Patients’ Preferences for Visuals: Differences in the Preferred Level of Detail, Type of Background and Type of Frame of Icons Depicting Organs between Literate and Low-Literate People. Patient Educ Couns. 2015;98(2):226–33. Epub 2014/12/04. pmid:25468391.
    • 52. Mayer RE, Moreno R. Nine Ways to Reduce Cognitive Load in Multimedia Learning. J Educational psychologist. 2003;38(1):43–52.
    • 53. Hollender N, Hofmann C, Deneke M, Schmitz B. Integrating Cognitive Load Theory and Concepts of Human–Computer Interaction. Comput Human Behav. 2010;26(6):1278–88.
    • 54. Rosman S. Cancer and Stigma: Experience of Patients with Chemotherapy-Induced Alopecia. Patient Educ Couns. 2004;52(3):333–9. Epub 2004/03/05. pmid:14998604.
    • 55. Myrick JG. Public Perceptions of Celebrity Cancer Deaths: How Identification and Emotions Shape Cancer Stigma and Behavioral Intentions. Health Communication. 2017;32(11):1385–95. pmid:27739882
    • 56. Pecho OE, Ghinea R, Perez MM, Della Bona A. Influence of Gender on Visual Shade Matching in Dentistry. J Esthet Restor Dent. 2017;29(2):E15–e23. Epub 2017/02/12. pmid:28185440.
    • 57. Jaint N, Verma P, Mittal S, Mittal S, Singh AK, Munjal S. Gender Based Alteration in Color Perception. Indian J Physiol Pharmacol. 2010;54(4):366–70. Epub 2011/06/17. pmid:21675035.
    • 58. Bimler D, Bonnardel V. Age and Gender Effects on Perceptual Color Scaling Using Triadic Comparisons. J optical Society of America A. 2018;35(4):B1–b10. Epub 2018/04/01. pmid:29603932.
    • 59. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The Role of Pictures in Improving Health Communication: A Review of Research on Attention, Comprehension, Recall, and Adherence. Patient Educ Couns. 2006;61(2):173–90. Epub 2005/08/27. pmid:16122896.
    • 60. Dowse R, Barford K, Browne SH. Simple, Illustrated Medicines Information Improves ARV Knowledge and Patient Self-Efficacy in Limited Literacy South African HIV Patients. AIDS Care. 2014;26(11):1400–6. Epub 2014/07/01. pmid:24975116; PubMed Central PMCID: PMC4124945.
    • 61. Kheir N, Awaisu A, Radoui A, El Badawi A, Jean L, Dowse R. Development and Evaluation of Pictograms on Medication Labels for Patients with Limited Literacy Skills in a Culturally Diverse Multiethnic Population. Res Social Adm Pharm. 2014;10(5):720–30. Epub 2013/12/21. pmid:24355379.
    • 62. van Beusekom MM, Land-Zandstra AM, Bos MJW, van den Broek JM, Guchelaar HJ. Pharmaceutical Pictograms for Low-Literate Patients: Understanding, Risk of False Confidence, and Evidence-Based Design Strategies. Patient Educ Couns. 2017;100(5):966–73. Epub 2017/01/04. pmid:28043712.
    • 63. Kim SR, Han K, Choi JY, Ersek J, Liu J, Jo SJ, et al. Age- and Sex-Specific Relationships between Household Income, Education, and Diabetes Mellitus in Korean Adults: The Korea National Health and Nutrition Examination Survey, 2008–2010. PLoS One. 2015;10(1):e0117034. Epub 2015/01/27. pmid:25622031; PubMed Central PMCID: PMC4306546.
    • 64. Zargarzadeh AH, Ahmadi S. Comprehensibility of Selected United States Pharmacopeia Pictograms by Illiterate and Literate Farsi Speakers: The First Experience in Iran—Part II. J Res Med Sci. 2017;22:101. Epub 2017/09/14. pmid:28900457; PubMed Central PMCID: PMC5583615.
    • 65. Dowse R, Ramela T, Browne SH. An Illustrated Leaflet Containing Antiretroviral Information Targeted for Low-Literate Readers: Development and Evaluation. Patient Educ Couns. 2011;85(3):508–15. Epub 2011/02/11. pmid:21306856.