https://orcid.org/0000-0003-0328-921X
Figures
Abstract
Informal caregivers are a critical source of support for cancer patients. However, their perspectives are not routinely collected, despite health impacts related to the burden of caregiving. We created the TOGETHERCare smartphone application (app) to collect observer-reported outcomes regarding the cancer patient’s health and caregiver’s perceptions of their own mental and physical health, and to provide tips and resources for self-care and patient care. We enrolled 54 caregivers between October 2020 and March 2021 from Kaiser Permanente Northern California (KPNC), an integrated healthcare system. Fifty caregivers used the app for approximately 28 days. Usability and acceptability were assessed using questions from the Mobile App Rating Scale (MARS), the System Usability Scale (SUS), the Net Promoter Score (NPS), and semi-structured interviews. The caregivers’ mean age was 54.4 years, 38% were female and 36% were non-White. The SUS total mean score was 83.4 (SD = 14.2), for a percentile rank of 90–95 (“excellent”). Median MARS responses to the functionality questions were also high. The NPS score of 30 at the end of the study indicated that most caregivers would recommend the app. Themes from semi-structured interviews were consistent across the study period and indicated that the app was easy to use and helpful. Caregivers indicated a need for feedback from the app, suggested some changes to the wording of questions, the app’s visuals, and timing of notifications. This study demonstrated that caregivers are willing to complete frequent surveys about themselves and their patients. The app is unique because it provides a remote method to collect caregivers’ observations about the patient that may be useful for clinical care. To our knowledge, TOGETHERCare is the first mobile app developed specifically to capture adult cancer patient symptoms from the informal caregiver’s perspective. Future research will examine whether use of this app can help improve patient outcomes.
Author summary
Family members and friends taking care of their loved ones who have been diagnosed with cancer often experience anxiety and depression due to the caregiving burden. Being able to report symptoms the loved one experiences as well as their own symptoms may provide health benefits to both the cancer patient and the caregiver. We created a smartphone app for informal caregivers of cancer patients (TOGETHERCare) to collect this information and to provide resources for self and patient care. Over 28 days, we tested usability of the application with caregivers of patients at Kaiser Permanente Northern California. The application included three surveys for caregivers to rate ease of learning and use. The average score on the three different rating scales was high and considered “excellent” compared with scores from other applications. Comments that caregivers provided during interviews also indicated that they found it easy to use and helpful. This study documents that caregivers are willing to complete frequent surveys about their loved ones and themselves. Based on the findings presented here and in other manuscripts, we hope to do further research to determine whether this app can help improve patient outcomes.
Citation: Oakley-Girvan I, Yunis R, Fonda SJ, Neeman E, Liu R, Aghaee S, et al. (2023) A novel smartphone application for the informal caregivers of cancer patients: Usability study. PLOS Digit Health 2(3): e0000173. https://doi.org/10.1371/journal.pdig.0000173
Editor: Padmanesan Narasimhan, University of New South Wales, AUSTRALIA
Received: June 13, 2022; Accepted: November 29, 2022; Published: March 3, 2023
Copyright: © 2023 Oakley-Girvan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: This work was conducted, in part, with NIH Small Business Innovation Research (SBIR) funding. Due to the sensitive nature of the data captured, the data release requirements of the KPNC subcontract, and the intellectual property associated with this work, all requests for data should be directed to the Chair of the Publication Committee, Dr. Oakley-Girvan (Ingrid@medable.com). Requests for data will be considered by the publication committee under a data use agreement and follow strict guidelines, including HIPAA regulations, regarding data use and the purpose of the analysis.
Funding: Research reported in this publication was partially supported by the National Cancer Institute of the National Institutes of Health under contract number HHSN261201700030C, which provided partial salaries for IOG, RY, SJF, SA, MER, AK, and SWD. The NIH grant also financially supported the participation of a project manager and external subcontractors. IOG, RY, and SWD received salaries as employees of Medable (www.Medable.com). Additional Medable staff assisted with data analysis. The funders (NIH) had no role in study design, decision to publish, or preparation of the manuscript.
Competing interests: IOG, SWD, and RY are employed by Medable Inc., which developed the TOGETHERCare app with funding from the National Institutes of Health. IOG and RY are also Medable stockholders. AI, SJA, MER, and SJF were all paid under subcontracts with Medable, Inc.
Introduction
There is an urgent need to support informal caregivers of cancer patients. The number of individuals living with cancer has increased from 14 million in 2012 to an estimated 18 million in 2022 [1], and the number of informal caregivers has grown to 6.1 million [2]. Informal caregivers attend medical visits with and advocate for the patient, keeping track of physician instructions and medication changes, providing emotional support, and assisting with activities such as medication administration, wound care, transportation, meals, and finances [3]. Over half of informal caregivers have chronic health conditions of their own [4,5], and their health status can be worsened by the stress and demands of caregiving [6–16], which can then adversely affect their patient’s well-being [17–20]. For instance, overburdened informal caregivers seldom seek mental health care to deal with emotional distress [21–23], despite informal caregivers of patients with advanced cancer reporting more depression than the patients themselves [24]. Further, caregivers have little time to rest or engage in self-care behaviors (e.g., health check-ups or preventive screening), and they often fail to seek medical care when sick [25,26]. The negative effects of caregiving, including a higher mortality rate, are found to be greater among informal caregivers of patients with advanced cancer [24,27,28], particularly when patients are receiving treatments such as chemotherapy [29,30]. Despite the tremendous burden that informal caregivers frequently experience [24,27,28], their emotional and physical needs are often not supported. By developing a system where a mobile app can capture caregivers’ physical and mental health status that could be sent directly to their care providers, preventive interventions could be made before caregivers become severely ill. This has the potential to help the caregivers provide greater care for their loved ones, improving the quality of life and, thereby, prognosis of cancer patients.
Studies indicate that using an electronic symptom reporting system decreases caregivers’ emotional distress [31–33]. Given the high utilization of mobile technologies, mobile applications (apps) may provide a meaningful mechanism for symptom management reporting. Almost all (97%) Americans own a cell phone, and 85% own a smartphone [34]. Mobile phone health apps have been shown to improve self-management of health conditions and [35] increase health-promoting behaviors [36].
Previous research has shown that cancer survivor symptom self-reporting through patient-reported outcomes (PROs) increases patient health-related quality-of-life, decreases emergency room visits and hospitalizations, and increases survival [37–41]. Similarly, the caregiver can provide information and early warnings of declining patient health through observer-reported outcomes (ObsROs) [42–48]. The efficacy of ObsROs, however, has not been studied in the context of cancer, although healthcare providers believe they are important to their understanding of patients’ symptoms and responses to cancer treatments [49].
In this article, we introduce TOGETHERCare, a mobile health application (app) designed to monitor caregiver health, streamline ObsROs reporting, and increase access to resources on health and well-being. To the best of our knowledge, this app is the first to collect caregiver ObsROs about adult cancer patients undergoing intravenous chemotherapy or immunotherapy and report on the app’s usability and acceptability from the perspective of the caregivers.
Methods
Setting
This study was conducted with Kaiser Permanente Northern California (KPNC). KPNC is an integrated healthcare delivery system serving over 4.5 million members in Northern California.
Study design
This single-arm, longitudinal, mixed-methods, prospective usability study included informal caregivers of cancer patients. We aimed to collect meaningful data to evaluate multiple parameters of usability including theme saturation in semi-structured interviews. We did not conduct any power calculations as this was not an intervention, so no effect size was anticipated.
To participate, both the cancer patient and their caregiver had to be eligible. Eligible cancer patients were adult (age 18 or older) KPNC members receiving intravenous chemo- or immunotherapy, living with a caregiver who was willing and eligible to participate, spoke English, and owned an iPhone 6 or above. Stage of cancer was not an inclusion or exclusion criterion. Cancer patients with serious mental health concerns or insufficient cognition to consent, as determined by their physician per KPNC policy, were not eligible to participate. Caregivers were identified by eligible patients based on who lived with them and spent the most time providing unpaid care. Eligible caregivers, lived with the patient, were 18 years of age or older, spoke English, and owned an iPhone 6 or above. There was no requirement for the length of time the caregiver lived with the patient. Patients and caregivers who owned an Android phone were ineligible because, at the time of the study, the application was available only on the iPhone. Caregivers were not required to be KPNC members.
After checking with the primary oncologists for contraindications to participation, study staff sent up to two recruitment emails, one week apart, containing brief study details and eligibility requirements to potential KPNC cancer patients. For those who expressed interest, the KPNC research associate screened for eligibility and reviewed study procedures with potential participants. The patient identified an interested caregiver who was then screened for eligibility. Recruitment was conducted between October 2020 and March 2021. We recruited 54 dyads (patient-caregiver pairs) as our objective was to collect complete study data from at least 45 caregivers.
After confirming eligibility, the research associate emailed the informed consent document and scheduled a call to review it. Informed consent was collected verbally and electronically in writing (eConsent) as one of the initial tasks provided through the app. Caregivers downloaded the TOGETHERCare app and were asked to use it at least several times per week for 28 days, during which the mobile app delivered specific surveys and activity requests at regular intervals (Table 1). Caregivers who completed the surveys received a $100 gift certificate. The KPNC Institutional Review Board (IRB) approved this study.
TOGETHERCare functions
The TOGETHERCare app was designed with input from caregivers, cancer patients, and healthcare providers as previously described [33]. The app included the following tabs: Profile for the study name and copy of the signed informed consent document; Tasks for informed consent, surveys related to caregivers’ health, caregiver demographics, and surveys related to patient health including a targeted symptom list and a pre-appointment concerns survey (Table 1); and Resources for information on how to contact the National Cancer Institute (NCI) and links to the Cancer Support Community and American Association for Retired Persons (AARP) webpages for caregivers. At the time of this study, the app did not include feedback or communications from clinicians regarding the users’ completed survey responses.
Data collection
Usability, acceptability, and user satisfaction were collected through mobile app rating scales during the “App Feedback” task (Table 1) and semi-structured interviews.
Mobile app rating questions/scales.
Three mobile app rating scales were collected at baseline, after approximately 14 days of use, and again at the end of the 28-day study period, via the app itself: a subset of questions from the MARS [53], the full SUS [51], and the NPS [54]. From the MARS, we selected five items that did not duplicate questions asked in the SUS. These questions covered functionality and visual appeal, with responses on a Likert scale of 1 (worst) to 5 (best) [53]. The functionality and aesthetics sections of the MARS have reliability coefficients of 0.87 and 0.90, respectively, and they correlate well with other validated instruments, which indicates good concurrent validity [57]. The SUS has 10 Likert-scaled items ranging from 1 (strongly agree) to 5 (strongly disagree). Eight items address usability and two address learnability. Total SUS scores range from 0 to 100, as do the scores for the usability and learnability sub-scales, with higher scores indicating better usability [52,58]. SUS scores are then converted to a percentile rank to show how the app rates relative to others [59]. The SUS has been used widely to study the usability of software and hardware, and a recent study shows that it has a similar distribution and interpretability when used to assess mobile health apps [60].
The NPS is derived by asking respondents to indicate the likelihood that they will recommend a product to a friend or colleague, using a 0 (least likely) to 10 (most likely) scale [54]. Respondents are defined as “promoters” of the product if they score 9–10, “detractors” if they score 0–6, and “passives” if they score 7–8. The percentage of detractors is subtracted from the percentage of promoters, giving the final NPS [61]. Although the usability questions were presented to the caregivers several times over the course of the study (Table 1), their responses were stable; thus, this article reports the summary statistics from the final survey instance (~28 days) only.
Semi-structured interviews.
Two researchers (IOG, RY) conducted semi-structured interviews after 7 days of app use (with 54 caregivers) and after completion of the study (with 50 caregivers) after at least 28 days of use. Researchers interviewed all caregivers in the study, since coding and evaluation of thematic saturation were conducted at a later date and we could not go back to conduct interviews after the study ended. The semi-structured interview guide was developed through early testing with stakeholders [33] and by considering product development insight from our technology team. Caregivers were asked to elaborate their responses, and those responses prompted more questions. Prior to submission of the interview guides to the KPNC IRB, we were unable to find references related to qualitative analyses of caregiver applications designed to collect observer reported outcomes. Interview topics included anything confusing or useful about the app, thoughts about the frequency of the surveys in the app and the length of time it took to complete them, whether the app included information that the caregiver would like the patient’s support team to know, and whether there were other features the caregiver would like to see. In addition to these topics, the final interviews included outcome questions, presentation of planned revisions based upon earlier feedback, the NPS, and questions about possible additional features and their usefulness. The interviews were conducted and recorded for transcription using a secure video platform (Microsoft Teams). The KPNC Institutional Review Board specifically requested that no characteristics of the interviewees be included with the transcripts of the interviews, and to protect study subject confidentiality, that the ID numbers used for the quantitative data collection be altered in such a way that they could not be matched with semi-structured interview subjects.
Participant background characteristics.
Demographics including age, gender, race, ethnicity, education, employment status, relationship to patient, and cancer health literacy were collected through the “About You” task in the app. Cancer health literacy was collected using the Cancer Health Literacy Test-6 (CHLT-6) [62].
Analyses
Descriptive statistics were used to report the caregivers’ background characteristics and responses to the final usability survey questions collected via the app. For the analysis of the semi-structured interviews, we used content analysis to identify a set of major themes from the interviews. Content analysis is a methodology used for analyzing large amounts of textual data in order to discover underlying patterns as well as make inferences via a process of systematic categorizing and coding of the information [63,64]. This method and related similar methods have several precedents in the medical literature [65–67]. In order to glean insights from the data, the data are read in detail and initial codes are constructed by identifying and analyzing units of the data, which are then grouped into meaningful and related categories [68,69]. The unit of analysis may consist of words, sentences, paragraphs, or a section of ideas and thoughts [63]. In this case, we used all of these units, as the interviews were semi-structured in nature. Given the sample size, we did not attempt to quantify the frequency of the different categories and themes, as this may result in loss of context or confer spurious importance to a category, as frequent mentions may be due to willingness or ability to talk about a topic [64].
We followed the three main steps of inductive content analysis (data preparation, organization, and reporting), as recommended when there is limited knowledge about the phenomenon [70]. In data preparation, the researcher becomes deeply immersed in the data. After multiple readings of the textual data, the researcher decides on the unit of analysis, which can be words, sentences, or phrases. The organization phase involves open coding and creating categories, after which the categories are grouped or collapsed into higher order categories, and in the final phase of reporting, the meanings of the categories are described [63,70]. To ensure reliability, after a trained qualitative researcher (VS) completed the coding and theme-formation, a second researcher (YM) independently reviewed it. An additional member of the research team (SWD) reviewed discrepancies between the two researchers, and, with input from the lead researcher of the projects (IOG), achieved consensus.
The interview qualitative data and quotes presented are from the interviews that were coded up to the point we reached thematic saturation. After carefully analyzing and coding all Day 7 interviews and 18 of the final interviews conducted after Day 28, no new themes emerged as theme saturation had been reached, and thus no further coding for the remaining Day 28 interviews was conducted.
Results
Recruitment and retention
Recruitment emails were sent to 2,155 KPNC cancer patients. Among them, a total of 1,908 patients did not respond to the email or could not be reached by phone. Of the 247 patients, 166 were confirmed ineligible, 20 declined to participate before eligibility could be determined, and 7 declined to participate after learning more details about the study. Fifty-four patient-caregiver dyads were enrolled; 50 dyads completed all study requirements for a retention rate of 92.6%.
Background characteristics of respondents
Of the 50 caregivers who completed the study, 62% (n = 31) were male and 38% (n = 19) female (Table 2). The majority (82%, n = 41) were not Hispanic or Latino, and 64% (n = 32) of the participants identified as White. Fifty-four percent (n = 27) had achieved a college degree, and nearly all (96.0%, n = 48) were cancer literate according to the Cancer Health Literacy Test Cancer-6 (CHLT-6) scores [62]. Each CHLT-6 response is either correct or incorrect; the mean number of correct responses for the six questions was 5.7, and the median was 6. Most of the patients (78%) attended by the caregivers were female, primarily due to the cancer type: breast (n = 17, 34%) and gynecological (n = 12, 24%). Gastrointestinal (n = 12), thoracic (n = 6), and other (n = 3) made up the remainder of patients. Over three-fourths of caregivers were the spouse or partner of the patient (n = 39, 78%).
Mobile app rating scales
The Day-28 in-app MARS items and the SUS scale showed that caregivers, overall, rated the TOGETHERCare app positively (Table 3). Both the mean and the median are presented in Table 3 to help describe the shape of the distributions. Because the mean and median are very similar, this demonstrates that the distributions are close to symmetrical. Median MARS ratings for app functionality (questions 1–4) were 4.5–5.0, at the highest end of the scale. The mean for these MARS questions ranged from 4.3 to 4.6. When combined, the MARS Functionality sub-score had a mean of 4.5 (SD 0.6) and median of 4.8. The median score for the MARS question on visual appeal (question 5) was 3.0, and the mean was 3.3 (SD 0.9). The raw total SUS score, usability sub-score and learnability sub-score translated into percentile rankings within the “excellent” range [59]. For the NPS administered via the app, “promoters” outnumbered “detractors” (4.2 percentage point difference). Results were stable for the in-app scores across the study period, so we present the final survey data in Table 3.
Semi-structured interviews
The interview results suggest that over the course of the study period users were generally positive about the app’s ease of use, and many found the navigation intuitive. Caregivers found the app helpful in raising their awareness about the patient, as well as about their own well-being. Caregivers noted that they wanted feedback from the app to better understand the progress of the patients, or if there were symptoms that should cause them concern. While some felt more graphical features would be helpful, caregivers were pleased by its simplicity. Caregivers suggested survey modifications and enhancements that would be helpful in better responding to a few of the questions, and they wanted more notifications linked to awaiting tasks. Table 4 further describes the themes and concepts and provides selected comments.
During the final semi-structured interviews after at least 28 days of use, interviewers presented revisions to the app that addressed the feedback, including themes, that we had received over the usage period. Interviewers then asked caregivers to provide the NPS considering these revisions. Thus, NPS interview scores were tabulated separately from the in-app NPS. The NPS score collected from the Day 7 interviews was 22; 26% were detractors and 48% were promoters. The NPS score collected from the Day 28 interviews was 30; 24% were detractors and 54% were promoters. We did not compare in-app NPS to interviewer NPS because of the difference in mode of administration and presentation of improved app functionality based on caregiver feedback and input.
Discussion
The objective of this study was to test the usability of a novel app designed for informal caregivers of adult cancer patients receiving intravenous chemo- or immunotherapy. The MARS items and the SUS scale showed that caregivers, overall, rated the TOGETHERCare app very positively and that this was stable over the 28-day period. The SUS scores were in the “excellent” range, indicating that people were able to figure out how to use the app, and that there were no serious usability issues. The NPS also indicated that more respondents would promote the app than those who would not recommend the app.
The analysis of the semi-structured interviews indicated that the app was easy to use and helpful. Caregivers provided suggestions for enhancing the survey questions, visuals, and functionality, such as linking notifications to incomplete app tasks and providing feedback, especially about the progress of their patient. Because this study was designed primarily to determine usability from the caregiver’s standpoint, data collected by the app was not sent to the clinical care team or provided back to the user in the form of charts, graphs, or other compilations. However, it is technically feasible to provide bidirectional interaction with EHR systems.
Despite caregiver interest in using online and mobile tools [71], few apps have been developed with the informal caregiver in mind [72,73]. A recent search on PubMed found no articles on applications for caregivers to report adult patient outcomes. Research is also lacking on caregiver health and psychosocial needs, and there has been insufficient testing of technology interventions that focus on caregiver health and psychosocial needs [74,75]. A recent review of online interventions to support cancer caregivers concludes that they are still in the formative stage; more research is needed to assess the impact on caregiver health [76]. Additionally, users are not involved in the development of most mobile health applications, despite evidence that input from direct users results in a more successful mobile tool [77,78]. Indeed, based upon the NPS from the final semi-structured interviews, this study reinforces the importance of gathering and incorporating participant feedback. Our study is one of the first to evaluate the usability of an app-based program that allows cancer caregivers to report their own health status as well as that of their patients’ including specific symptoms.
Future directions
Future work in a larger clinical sample will include providing survey data completed by caregivers to their own providers within the clinical workflow. We expect, when it is medically acceptable, to provide visualizations of the survey responses over time for users to view so that caregivers can be aware of their well-being and be provided with appropriate resources. We plan to further integrate with EHR systems and test the app’s impact on specific health outcomes in a randomized controlled trial.
Limitations
First, we included only participants who spoke English because the app was only offered in English. Offering the app in other languages would require cultural adaptation, and we plan on developing these after completing efficacy studies. Second, only participants who owned an iPhone were eligible to participate. Because our usability testing and other results (to be published separately) were positive, an Android version of TOGETHERCare, has now been developed. Finally, due to the usability testing nature of this study, and per the IRB recommendations, the collected data were not relayed to the participating patients’ or caregivers’ providers. One exception was that the IRB wanted the research associate to be automatically alerted to survey scores for mental health measures from the PROMIS Global Health scale that exceeded a predetermined threshold. If that occurred, the system was set-up to alert the research associate by email so appropriate next steps could be conducted following KPNC policies.
Conclusions
TOGETHERCare is the first mobile app developed specifically to capture adult cancer patient outcomes from the informal caregiver’s perspective. In addition, it tracks the caregiver’s own health and wellbeing, which can impact the patient. Caregivers’ willingness to complete frequent surveys about their patients and themselves is promising, but additional research is needed to determine if the app can be used to improve outcomes. In addition to monitoring the physical and mental health of the caregiver, future versions of this app could potentially provide important clinical information that could inform patient care in meaningful ways, including by reducing emergency room visits or improving care.
Acknowledgments
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. In addition to the above mentioned contract, Medable staff were materially supported as employees of Medable (www.Medable.com) for this effort. Special thanks to Anushka Gupta (Medable staff), who assisted with initial data analysis, Vandana Shah and Yasamin Miller (subcontractors under the above mentioned contract number), who did initial coding of the semi-structured interviews and Elaine M. Kurtovich, KPNC Research Project Manager (supported under the above mentioned contract for her efforts on this project). Selected KPNC staff (AK, EK, MER, SJA) were paid for their time through the contract and not by any other funding at KPNC.
References
- 1. de Moor JS, Mariotto AB, Parry C, Alfano CM, Padgett L, Kent EE, et al. Cancer survivors in the United States: prevalence across the survivorship trajectory and implications for care. Cancer Epidemiol Biomarkers Prev. 2013 Apr;22(4):561–70. pmid:23535024.
- 2. Appelbaum AJ. Care for the cancer caregiver. The ASCO Post. 2018;2020(Sept. 25, 2018).
- 3. Kasper JD, Steinbach U, Andrews J. Caregiver role appraisal and caregiver tasks as factors in ending caregiving. J Aging Health. 1994 Aug;6(3):397–414. pmid:10135717.
- 4. Mellon S, Northouse LL. Family survivorship and quality of life following a cancer diagnosis. Res Nurs Health. 2001 Dec;24(6):446–59. pmid:11746074.
- 5. Thornton AA, Perez MA, Meyerowitz BE. Patient and partner quality of life and psychosocial adjustment following radical prostatectomy. Journal of Clinical Psychology in Medical Settings. 2004;11(1):15–30.
- 6. Barnhart WR, Ellsworth DW, Robinson AC, Myers JV, Andridge RR, Havercamp SM. Caregiving in the shadows: National analysis of health outcomes and intensity and duration of care among those who care for people with mental illness and for people with developmental disabilities. Disabil Health J. 2020 Jan;13(1):100837. pmid:31506219.
- 7. Baudry AS, Vanlemmens L, Anota A, Cortot A, Piessen G, Christophe V. Profiles of caregivers most at risk of having unmet supportive care needs: Recommendations for healthcare professionals in oncology. Eur J Oncol Nurs. 2019 Dec;43:101669. pmid:31610470.
- 8. Geng HM, Chuang DM, Yang F, Yang Y, Liu WM, Liu LH, et al. Prevalence and determinants of depression in caregivers of cancer patients: A systematic review and meta-analysis. Medicine (Baltimore). 2018 Sep;97(39):e11863. pmid:30278483.
- 9. Ochoa CY, Buchanan Lunsford N, Lee Smith J. Impact of informal cancer caregiving across the cancer experience: A systematic literature review of quality of life. Palliat Support Care. 2019 Oct 7:1–21. pmid:31588882.
- 10. Otto AK, Gonzalez BD, Heyman RE, Vadaparampil ST, Ellington L, Reblin M. Dyadic effects of distress on sleep duration in advanced cancer patients and spouse caregivers. Psychooncology. 2019 Dec;28(12):2358–64. pmid:31518026.
- 11. Perpina-Galvan J, Orts-Beneito N, Fernandez-Alcantara M, Garcia-Sanjuan S, Garcia-Caro MP, Cabanero-Martinez MJ. Level of Burden and Health-Related Quality of Life in Caregivers of Palliative Care Patients. Int J Environ Res Public Health. 2019 Nov 29;16(23). pmid:31795461.
- 12. Peterson RK, Chung J, Barrera M. Emotional symptoms and family functioning in caregivers of children with newly diagnosed leukemia/lymphomas and solid tumors: Short-term changes and related demographic factors. Pediatr Blood Cancer. 2019 Nov 13:e28059. pmid:31724323.
- 13. Shin JY, Lim JW, Shin DW, Kim SY, Yang HK, Cho J, et al. Underestimated caregiver burden by cancer patients and its association with quality of life, depression and anxiety among caregivers. Eur J Cancer Care (Engl). 2018 Mar;27(2):e12814. pmid:29333736.
- 14. Tingey JL, Lum J, Morean W, Franklin R, Bentley JA. Healthcare coverage and utilization among caregivers in the United States: Findings from the 2015 Behavioral Risk Factor Surveillance System. Rehabil Psychol. 2020 Feb;65(1):63–71. pmid:31971434.
- 15. Ell K, Nishimoto R, Mantell J, Hamovitch M. Longitudinal analysis of psychological adaptation among family members of patients with cancer. J Psychosom Res. 1988;32(4–5):429–38. pmid:3236270.
- 16. Haley WE, LaMonde LA, Han B, Burton AM, Schonwetter R. Predictors of depression and life satisfaction among spousal caregivers in hospice: application of a stress process model. J Palliat Med. 2003 Apr;6(2):215–24. pmid:12854938.
- 17. Kedia SK, Collins A, Dillon PJ, Akkus C, Ward KD, Jackson BM. Psychosocial interventions for informal caregivers of lung cancer patients: A systematic review. Psychooncology. 2019 Nov 8. pmid:31701588.
- 18. Ketcher D, Otto AK, Vadaparampil ST, Heyman RE, Ellington L, Reblin M. The Psychosocial Impact of Spouse-Caregiver Chronic Health Conditions and Personal History of Cancer on Well-being in Patients With Advanced Cancer and Their Caregivers. J Pain Symptom Manage. 2021 Aug;62(2):303–11. pmid:33348028.
- 19. Litzelman K. Caregiver Well-being and the Quality of Cancer Care. Semin Oncol Nurs. 2019 Aug;35(4):348–53. pmid:31229346.
- 20. Yang Y, Pan W, Allen D, Hendrix CC. Caregiver Burden as a Mediator Between Emotional Distress and Concentration Problems in Patients With Cancer. Oncol Nurs Forum. 2019 Nov 1;46(6):E180–e4. pmid:31626623.
- 21. Given BA, Given CW, Kozachik S. Family support in advanced cancer. CA Cancer J Clin. 2001 Jul-Aug;51(4):213–31. pmid:11577488.
- 22. Bishop MM, Beaumont JL, Hahn EA, Cella D, Andrykowski MA, Brady MJ, et al. Late effects of cancer and hematopoietic stem-cell transplantation on spouses or partners compared with survivors and survivor-matched controls. J Clin Oncol. 2007 Apr 10;25(11):1403–11. pmid:17416860.
- 23. Vanderwerker LC, Laff RE, Kadan-Lottick NS, McColl S, Prigerson HG. Psychiatric disorders and mental health service use among caregivers of advanced cancer patients. J Clin Oncol. 2005 Oct 1;23(28):6899–907. pmid:16129849.
- 24. Braun M, Mikulincer M, Rydall A, Walsh A, Rodin G. Hidden morbidity in cancer: spouse caregivers. J Clin Oncol. 2007 Oct 20;25(30):4829–34. pmid:17947732.
- 25. Burton LC, Newsom JT, Schulz R, Hirsch CH, German PS. Preventive health behaviors among spousal caregivers. Prev Med. 1997 Mar-Apr;26(2):162–9. pmid:9085384.
- 26. Carter PA. Caregivers’ descriptions of sleep changes and depressive symptoms. Oncol Nurs Forum. 2002 Oct;29(9):1277–83. pmid:12370697.
- 27. Ringdal GI, Ringdal K, Jordhoy MS, Ahlner-Elmqvist M, Jannert M, Kaasa S. Health-related quality of life (HRQOL) in family members of cancer victims: results from a longitudinal intervention study in Norway and Sweden. Palliative Medicine. 2004 Mar;18(2):108–20. pmid:15046407.
- 28. Fang CY, Manne SL, Pape SJ. Functional impairment, marital quality, and patient psychological distress as predictors of psychological distress among cancer patients’ spouses. Health Psychol. 2001 Nov;20(6):452–7. pmid:11714188.
- 29. Pitceathly C, Maguire P. The psychological impact of cancer on patients’ partners and other key relatives: a review. Eur J Cancer. 2003 Jul;39(11):1517–24. pmid:12855257.
- 30. Kim Y, Given BA. Quality of life of family caregivers of cancer survivors: across the trajectory of the illness. Cancer. 2008 Jun 1;112(11 Suppl):2556–68. pmid:18428199.
- 31. Chih MY, DuBenske LL, Hawkins RP, Brown RL, Dinauer SK, Cleary JF, et al. Communicating advanced cancer patients’ symptoms via the Internet: a pooled analysis of two randomized trials examining caregiver preparedness, physical burden, and negative mood. Palliat Med. 2013 Jun;27(6):533–43. pmid:22988042.
- 32. Wishart LR, Brown B, Nund RL, Fotinos E, Hutchison AR, Ward EC, et al. A prospective study monitoring carer distress during (chemo)radiotherapy for head and neck cancer via an electronic platform. J Med Radiat Sci. 2020 Nov 10. pmid:33169922.
- 33. Oakley-Girvan I, Davis SW, Kurian A, Rosas LG, Daniels J, Palesh OG, et al. Development of a Mobile Health App (TOGETHERCare) to Reduce Cancer Care Partner Burden: Product Design Study. JMIR Form Res. 2021;5(8):e22608. pmid:34398787
- 34.
Pew Research Center. Mobile Fact Sheet. Demographics of mobile device ownership and adoption in the United States: pewresearch.org; 2021.
- 35. Whitehead L, Seaton P. The Effectiveness of Self-Management Mobile Phone and Tablet Apps in Long-term Condition Management: A Systematic Review. J Med Internet Res. 2016 May 16;18(5):e97. pmid:27185295.
- 36. Mahmood A, Kedia S, Wyant DK, Ahn S, Bhuyan SS. Use of mobile health applications for health-promoting behavior among individuals with chronic medical conditions. Digit Health. 2019 Jan-Dec;5:2055207619882181. pmid:31656632.
- 37. Matsuda A, Yamada Y, Ishizuka N, Matsushima E, Kobayashi K, Ohkubo T, et al. Effectiveness of a Self-Monitoring Quality of Life Intervention For Patients with Cancer Receiving Palliative Care: A Randomized Controlled Clinical Trial. Asian Pac J Cancer Prev. 2019 Sep 1;20(9):2795–802. pmid:31554379.
- 38. Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol. 2016 Feb 20;34(6):557–65. pmid:26644527.
- 39. Basch E, Stover AM, Schrag D, Chung A, Jansen J, Henson S, et al. Clinical Utility and User Perceptions of a Digital System for Electronic Patient-Reported Symptom Monitoring During Routine Cancer Care: Findings From the PRO-TECT Trial. JCO Clin Cancer Inform. 2020 Oct;4:947–57. pmid:33112661.
- 40. Basch EM, Deal AM, Dueck AC, Bennett AV, Atkinson TM, Scher HI, et al. Overall survival results of a randomized trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. American Society of Clinical Oncology; 2017.
- 41. Denis F, Basch E, Septans AL, Bennouna J, Urban T, Dueck AC, et al. Two-Year Survival Comparing Web-Based Symptom Monitoring vs Routine Surveillance Following Treatment for Lung Cancer. JAMA. 2019 Jan 22;321(3):306–7. pmid:30667494.
- 42. Armstrong TS, Wefel JS, Gning I, Acquaye A, Vera-Bolanos E, Gilbert MR, et al. Congruence of primary brain tumor patient and caregiver symptom report. Cancer. 2012 Oct 15;118(20):5026–37. pmid:22415423.
- 43. Drudge-Coates L, Oh WK, Tombal B, Delacruz A, Tomlinson B, Ripley AV, et al. Recognizing Symptom Burden in Advanced Prostate Cancer: A Global Patient and Caregiver Survey. Clin Genitourin Cancer. 2018 Apr;16(2):e411–e9. pmid:29111175.
- 44. Madden K, Magno Charone M, Mills S, Dibaj S, Williams JL, Liu D, et al. Systematic Symptom Reporting by Pediatric Palliative Care Patients with Cancer: A Preliminary Report. J Palliat Med. 2019 Aug;22(8):894–901. pmid:30759031.
- 45. Kurtz ME, Kurtz JC, Given CC, Given B. Concordance of cancer patient and caregiver symptom reports. Cancer Pract. 1996 Jul-Aug;4(4):185–90. pmid:8900759.
- 46. Sikorskii A, Victorson D, O’Connor P, Hankin V, Safikhani A, Crane T, et al. PROMIS and legacy measures compared in a supportive care intervention for breast cancer patients and caregivers: Experience from a randomized trial. Psychooncology. 2018 Sep;27(9):2265–73. pmid:29956396.
- 47. Silveira A, Monteiro E, Sequeira T. Head and Neck Cancer: Improving Patient-Reported Outcome Measures for Clinical Practice. Curr Treat Options Oncol. 2018 Oct 1;19(11):59. pmid:30276480.
- 48. Latulipe C, Quandt SA, Melius KA, Bertoni A, Miller DP Jr, Smith D, et al. Insights Into Older Adult Patient Concerns Around the Caregiver Proxy Portal Use: Qualitative Interview Study. J Med Internet Res. 2018;20(11):e10524. pmid:30389654
- 49. Neeman E, Kubo A, Kurtovich E, Aghaee S, Ramsey M, Yunis R, et al. Mobile app and wearable sensor-based patient and caregiver physical function and reported outcomes: A survey on oncologists’ current practices and preferences on data delivery. American Society of Clinical Oncology; 2021.
- 50. Bevans M, Ross A, Cella D. Patient-Reported Outcomes Measurement Information System (PROMIS): Efficient, standardized tools to measure self-reported health and quality of life. Nurs Outlook. 2014 Sep-Oct;62(5):339–45. pmid:25015409.
- 51. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human–Computer Interaction. 2008 2008/07/29;24(6):574–94.
- 52. Sauro J. Measuring usability with the system usability scale (SUS). Measuring U.2011.
- 53. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015 Mar 11;3(1):e27. pmid:25760773.
- 54. Reichheld FF. The one number you need to grow. Harvard Business Review. 2003;81(12):46–55. pmid:14712543
- 55. Basch E, Reeve BB, Mitchell SA, Clauser SB, Minasian LM, Dueck AC, et al. Development of the National Cancer Institute’s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Natl Cancer Inst. 2014;106(9):dju244. pmid:25265940
- 56. Oken MM, Creech RH, Tormey DC, Horton J, Davis TE, McFadden ET, et al. Toxicity and response criteria of the Eastern Cooperative Oncology Group. Am J Clin Oncol. 1982 Dec;5(6):649–55. pmid:7165009.
- 57. Terhorst Y, Philippi P, Sander LB, Schultchen D, Paganini S, Bardus M, et al. Validation of the Mobile Application Rating Scale (MARS). PLoS One. 2020;15(11):e0241480. pmid:33137123.
- 58. Smyk A. The System Usability Scale and how it’s used in UX. The System Usability Scale and how it’s used in UX. XD.adobe.2020.
- 59. Sauro J. 5 ways to interpret a SUS score. 5 ways to interpret a SUS score. Measuring U.2018.
- 60. Hyzy M, Bond R, Mulvenna M, Bai L, Dix A, Leigh S, et al. System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis. JMIR Mhealth Uhealth. 2022 Aug 18;10(8):e37290. pmid:35980732.
- 61. Hajesmaeel-Gohari S, Khordastan F, Fatehi F, Samzadeh H, Bahaadinbeigy K. The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health. BMC Med Inform Decis Mak. 2022 Jan 27;22(1):22. pmid:35081953.
- 62. Dumenci L, Matsuyama R, Riddle DL, Cartwright LA, Perera RA, Chung H, et al. Measurement of cancer health literacy and identification of patients with limited cancer health literacy. J Health Commun. 2014;19 Suppl 2:205–24. pmid:25315594.
- 63. Downe-Wamboldt B. Content analysis: method, applications, and issues. Health care for women international. 1992;13(3):313–21. pmid:1399871
- 64. Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013 Sep;15(3):398–405. pmid:23480423.
- 65. Baron-Cohen S, Wheelwright S. ’Obsessions’ in children with autism or Asperger syndrome. Content analysis in terms of core domains of cognition. Br J Psychiatry. 1999 Nov;175:484–90. pmid:10789283.
- 66. Owen PR. Portrayals of schizophrenia by entertainment media: a content analysis of contemporary movies. Psychiatr Serv. 2012 Jul;63(7):655–9. pmid:22555313.
- 67. Ullström S, Andreen Sachs M, Hansson J, Ovretveit J, Brommels M. Suffering in silence: a qualitative study of second victims of adverse events. BMJ Qual Saf. 2014 Apr;23(4):325–31. pmid:24239992.
- 68.
Corbin J, Strauss A. Basics of qualitative research (third edition): Techniques and procedures for developing grounded theory. Thousand Oaks: Sage publications Inc.; 2012.
- 69. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005 Nov;15(9):1277–88. pmid:16204405.
- 70. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008 Apr;62(1):107–15. pmid:18352969.
- 71. Wasilewski MB, Stinson JN, Cameron JI. Web-based health interventions for family caregivers of elderly individuals: A Scoping Review. Int J Med Inform. 2017 Jul;103:109–38. pmid:28550996.
- 72. Grossman MR, Zak DK, Zelinski EM. Mobile Apps for Caregivers of Older Adults: Quantitative Content Analysis. JMIR Mhealth Uhealth. 2018 Jul 30;6(7):e162. pmid:30061093.
- 73. Wittenberg E, Xu J, Goldsmith J, Mendoza Y. Caregiver Communication About Cancer: Development of a mhealth resource to support family caregiver communication burden. Psychooncology. 2019 Feb;28(2):365–71. pmid:30471236.
- 74. Kent EE, Rowland JH, Northouse L, Litzelman K, Chou WY, Shelburne N, et al. Caring for caregivers and patients: Research and clinical priorities for informal cancer caregiving. Cancer. 2016 Jul 1;122(13):1987–95. pmid:26991807.
- 75. Sun V, Raz DJ, Kim JY. Caring for the informal cancer caregiver. Curr Opin Support Palliat Care. 2019 Sep;13(3):238–42. pmid:31157656.
- 76. Shin JY, Choi SW. Online interventions geared toward increasing resilience and reducing distress in family caregivers. Curr Opin Support Palliat Care. 2020 Mar;14(1):60–6. pmid:31842019.
- 77. Davis SW, Oakley-Girvan I. Achieving value in mobile health applications for cancer survivors. J Cancer Surviv. 2017 Aug;11(4):498–504. pmid:28342093.
- 78. Harder H, Holroyd P, Burkinshaw L, Watten P, Zammit C, Harris PR, et al. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv. 2017 Dec;11(6):732–42. pmid:28741202.