https://orcid.org/0000-0002-2267-1038
Figures
Abstract
In this paper, we evaluate the feasibility of using ecological momentary assessment (EMA) to understand urinary (UI) and fecal (FI) incontinence in adults with spina bifida (SB). As part of a larger 30-day prospective study to understand the incontinence in adults with SB (N = 89), participants completed end-of-day EMA diaries assessing the frequency and context of UI and FI. We used these data to assess the method feasibility across six dimensions: (a) compliance, or data entry which is consistent with study protocol and substantially complete; (b) reactivity, or behavior change attributed to study participation; (c) participant acceptability, or convenience and ease of method beneficial to compliance; (d) data capture, or the volume of incontinence behaviors collected; (e) the accuracy of incontinence reports; and f) participant-provided feedback for future studies. Participants were highly compliant with diary entry protocol and schedule: submitting 95.7% (2576/2700) of the expected total daily entries. The average completion time was two minutes. Neither the total number of submissions nor the completion time varied by demographic characteristics or health history. A sufficient volume of incontinence and affective outcomes were captured, with small downtrends in reporting of UI and affect over time. Exit survey recall was highly correlated with diary reports. Participants found the methodology to be acceptable, reported their experiences honestly, enjoyed and felt comfortable participating in the study and would engage in similar study in the future. Accurate information about the daily context of UI and FI is a key factor in the success of intervention or education programs relying on this information. Our findings demonstrate that EMA is a feasible way to describe UI and FI in adults with SB.
Citation: Hensel DJ, Young AI, Szymanski KM (2023) The feasibility of using ecological momentary assessment to understand urinary and fecal incontinence experiences in adults with spina bifida: A 30-day study. PLoS ONE 18(11): e0292735. https://doi.org/10.1371/journal.pone.0292735
Editor: Sandip Varkey George, University of Aberdeen, UNITED KINGDOM
Received: April 18, 2023; Accepted: September 27, 2023; Published: November 30, 2023
Copyright: © 2023 Hensel et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All raw data files are available on the Open Science Framework (osf.io/gefqx).
Funding: This project was supported by grant R21 HD053231-01A1 to DJH and KS from the National Institute of Diabetes Digestive and Kidney Disorders (NIDDK: https://www.niddk.nih.gov/). The contents of this paper are solely the responsibility of the authors and do not necessarily reflect the official views of the NIDDK. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Dr. Hensel is a paid research consultant with For Goodness Sake, LLC.
Abbreviations: EMA, ecological momentary assessment; SB, Spina Bifida; UI, urinary incontinence; FI, fecal incontinence
Introduction
Spina bifida (SB) is a congenital anomaly in which the neural tube does not fully close during embryonic development, resulting in permanent damage to the spinal cord [1]. Depending on defect severity, people with SB can experience several different chronic health complications, including urinary (UI) or fecal (FI) incontinence [2]. Three quarters of adults with SB report UI, half experience either FI or many live with both UI and FI [3]. Incontinence is broadly recognized to decrease mental health (e.g. increased depression or anxiety) and social outcomes (e.g. interference with employment, daily activities and relationships) [4–6], and our own research additionally suggests that specific dimensions of UI and FI (e.g. quantity vs. frequency vs. amount) are differentially linked to lower quality of life [3, 7–10]. The high prevalence and impact of UI and FI in the SB population highlight the need for data collection methods that can capture daily incontinence experiences with fidelity to the ways in which people actually live them [11].
Most SB research assesses incontinence by way of retrospective questionnaires that require individuals to self-recall and summarize incontinence information after what is typically a time delay (e.g., “On average, how many times a day did you experience urinary incontinence in the past week?” or “In general, how bothersome was your fecal incontinence over the past month?”) [12, 13]. This approach is common because it is difficulty for a clinician or researcher to directly observe incontinence in a person’s daily life. At the same time, however, these approaches present challenges. Sensitive and/or stigmatized behaviors, like incontinence, can be underreported out of participant concern for privacy or judgement from others [14]. Longer recall periods may decrease the accuracy with which a participant can recall or describe incontinence experiences [15]. Finally, summarizing multiple sources of information into a single data point (for example, aggregating 30 individual days into a single “month” focused question) removes the ability to understand how day-to-day changes in different aspects of incontinence, like frequency, bother or interference, could have different impact on a person’s well-being [16–18]. All three of these issues–underreport, recall accuracy and lack of event-specific detail–can degrade the reliability and the validity of the information collected.
Recent studies have shown that ecological momentary assessment (EMA) is a promising method to maximize the “real-life” generalizability of self-reported health behavior data. EMA refers to a broad class of prospective–or forward looking–data collection approaches intended to capture snapshots of people’s daily lives through brief-but-repeated assessments in their “natural” environments [19]. “Ecological” refers to collecting data as feasibly close in time and condition to when people themselves experienced them [20, 21], while “momentary” refers to people’s repeated reporting of events or experiences at the current moment or very recently in time [22]. Data are typically collected using the functional capabilities of web-enabled electronic devices, and participants provide data on a study-specific time schedules (for example, once/day or following a specific health event; for an excellent review, see Hufford et al., 2007 [23]) usually over weeks or months [24, 25]. Used in the context of incontinence, EMA is ideal for amassing enough data points to be able understand of how a participant’s personal or social context differs when UI or FI does or does not occur, as well as how the relationship between context and UI or FI may change over time [26]. Such data are important for optimizing the chance that interventions based on this knowledge will be effective when employed in people’s daily lives [27].
There are data quality, programming and data security advantages to using EMA (for excellent reviews, see Hensel, 2014 [28] or Hufford and Shiffman, 2002 [26]). From a data quality perspective, EMA typically produces higher reporting rates, lower levels of missing data, more robust internal validity and lower behavior reactivity (e.g. changing what is reported as a result of participating in the study)compared to other collection approaches [20, 22]. From a programming perspective, EMAs can be designed to be delivered across many different types of web-enabled devices (e.g., smartwatches, cell phone, tablets) [29]. There is incredible flexibility in question structure (e.g. fixed choice vs. free text, single choice vs. “check all that apply,” check box or sliding “rating” type scales, etc.), in assessment frequency (e.g. once per day/week vs. multiple times/day) and in assessment pattern (e.g., at the same time, only when an event happens, or at random intervals) [28]. Multiple language versions of surveys can be offered [30]. All interactions a participant has with the electronic system can be time-stamped, allowing real-time description of compliance (e.g., number of diaries started, completed, or submitted), and/or calculation of average completion time for specific questions, or for the entire diary [31]. In some instances, a diary could be paired with other electronic ‘add-ons’ to enhance or supplement existing data (e.g., ‘apps,’ video-sharing, chatting) [28]. From a data security perspective, EMA can strengthen the security of sensitive or stigmatizing information by allowing information to be vacated from the device immediately upon data entry for storage on a remote server, thereby increasing anonymity and confidentiality [16]. Participants can additionally be encouraged to use the security features (e.g., passcode, Face ID, or fingerprint ID) on their device.
Perhaps most importantly, the electronic structure of EMA additionally confers participant-centered advantages for populations of disabled persons who may otherwise be excluded from clinical research [32]. Mobility challenges create research participation barriers for individuals with SB, either because travel to non-clinical recruitment sites is difficult, or the specialty clinic at which they receive care is not a recruitment site [33]. The electronic nature of EMA allows data collection to happen quite literally “anywhere and anytime” [34, 35], removing nearly all the constraints of specific times or physical spaces in which research must be conducted [36]. Web-enabled devices, such as cell phones and tablets, are the most commonly owned device by people with disabilities [37], including those with SB [38, 39]. Adults and young people in this population use their web-enabled technology daily for various purposes, such as health care, work, education and social interaction [40–43]. Navigating these different tasks means that individuals with SB are already “self-trained” on most, if not all, of the physical skills needed for study research participation (e.g., clicking on links, opening documents, entering/changing text, using fingers to swipe pages or mark boxes or submitting forms) [44]. Engaging technology with which individuals are already actively using permits easy introduction of EMA in a manner that is minimally invasive and minimally disruptive to their daily life [45]. Participants are able to choose where and when they complete data entry and submission, allowing them greater flexibility in participating in a study and potentially increasing their comfort in disclosing sensitive information like incontinence [28].
EMA has been utilized to examine a variety of clinical health outcomes in populations both with and without disabilities, including chronic pain [25], chronic illness in association with disability [46], hearing aid use among individuals with hearing impairment [47], neuropathic pain among individuals with spinal cord injury [24], daily functioning in individuals with schizophrenia [48] and daily affect following traumatic brain injury [49]. EMA has also been engaged to examine urinary symptoms among prostate cancer patients [50] and bowel incontinence among individuals with different conditions [51, 52]. EMA has not yet been used in clinical research among individuals with SB.
In this paper, we evaluate using daily EMAs over 30 days as a means of prospectively understanding experiences of UI and FI among adults with SB. Using a feasibility framework established by the first author’s work [16, 17, 53, 54], we assess (a) compliance, or data entry which is consistent with study protocol and substantially complete; (b) reactivity, or behavior change attributed to study participation; (c) participant acceptability, or convenience and ease of method beneficial to compliance. We also assess (d) data capture, or the volume of incontinence behaviors collected and (e) the accuracy of incontinence reports. Finally, we review participant-provided feedback on ways to improve future studies.
Methods
Study design and data collection
Data were collected as part of a larger 30-day study prospectively examining the daily prevalence and context of UI and FI in adults with SB. In this larger study, data collection was organized around three components: 1) a self-administered enrollment module, which allowed participants to provide informed consent and HIPPA authorization, as well as assessed demographics, quality of life, and past behavioral and social experiences with incontinence; 2) once daily EMA diaries monitoring affect, health, frequency, volume and management of UI and/or FI reported, as well as daily activities; 3) a self-administered exit survey which assessed participant experiences in the study, as well as gathered retrospective reports of incontinence variables reported in the EMA diaries. All data collection was automated, meaning that the completion of one data source intuitively triggered the delivery of the next data source. For example, submission of the enrollment module prompted the distribution of the Day 1 EMA, the completion of the Day 1 EMA signaled the delivery of the Day 2 EMA, and so on, and the completion of the last EMA triggered the delivery of the exit survey. The larger project was approved by the Institutional Review Board of Indiana University (#1907916729).
All data collection was conducted electronically via a secure, cloud-based survey tool called Qualtrics [55]. We chose this program for several reasons. The program is accessible from multiple devices (e.g. a phone, tablet or laptop/desktop computer) via any internet connection and any web browser. In addition, Qualtrics offers intuitive programming of different survey types (e.g. cross-sectional vs. longitudinal) and question types (e.g. scales, open-ended, single- and multiple-choice, and electronic signature). The program additionally does not require that participants have additional tools (e.g., stylus) or specialized software (e.g., Adobe Acrobat) [56] to provide data. These two attributes–flexible accessibility and question type–means generally increased ease-of-access for unique populations needs. For example, participants who have multiple devices (e.g. they use a phone and tablet interchangeably) as well as participants who only have one device (e.g. a laptop) are equally able to complete data collection regardless of any personal limitations. Qualtrics also offers a flexible project management interface, allowing researchers to quickly be able to see and organize multiple data arms for the same study. After data collection, researchers are able to engage several different automated download options to house their data in different programs (e.g. Excel or statistical software).
Participants
Due in part to both COVID-19 clinical restrictions during the recruitment period (e.g. no research personnel were allowed in clinical waiting rooms from which participants would have otherwise been recruited) and to the common physical limitations of many individuals with SB (e.g. mobility challenges make in person recruitment less possible), we engaged a combination of electronic convenience and snowball sampling methods as means of reaching potential participants. Short study announcements were placed on the social media platforms (e.g., Twitter, Facebook, Instagram) of local, national and international SB (e.g., the national Spina Bifida Association, Spina Bifida Indiana) organizations contacted directly by the investigators. In addition, study fliers were mailed to local SB patients receiving care at our center who had agreed to receive study-related information. All announcements and fliers contained a QR code that, when scanned, took interested individuals to a study landing page that provided more detailed study information and the contact information of study staff.
Study eligibility requirements included being 18 years or older, having SB, having any UI and/or FI in the past four weeks, having English language literacy and having normal to mildly impaired cognitive development. Specific device ownership was not a requirement, and we built in study funds to buy a web-enabled phone for any participant who either had no way of accessing the internet for data collection, or who preferred not to use their own device. All participants opted to use devices already in their possession. All potential participants made an appointment for a virtual information and screening session with a member of the study staff. During the appointment, staff reviewed the study purpose, data collection requirements, and compensation schedule with individuals, along with language literacy and developmental delay. Those who passed screening were emailed a link to the enrollment module where they completed informed consent, HIPAA authorization and provided background information. Within this module, they also chose a delivery method–short message service (SMS; e.g., a text message) or email–for the remaining data sources. About 80% of participants opted for the SMS option, and no one asked to change their first chosen method.
EMA diaries
During the enrollment appointment, study staff provided an in-depth orientation to the EMA diary entry protocol with all participants. During this time, they also reviewed the policies on missing entries and tips for troubleshooting procedures for equipment or connectivity issues. Beginning on the day after enrollment completion, participants received an SMS or an email at 7pm in their local time zone that linked them to that day’s EMA diary. While some EMA researchers suggest prompting diary completion at random intervals each day [20], other suggests that data collection at consistent times each day is equally valid [57], and may increase the convenience of, and compliance to, the data collection procedure for participants.
Once participants clicked on the diary link, they arrived on the landing page of the Qualtrics survey (Fig 1, Panel A) associated with that specific EMA diary day in their study progression. Individuals completed a specific sequences of questions that assessed their experiences since their last entry, including their affect (example item, Fig 1, Panel B), UI or FI frequency (example items, Fig 1, Panels C and D) and daily activities. If any incontinence was reported, they answered contingency items about amount, management, activities they avoided and feelings about the incontinence. If no incontinence was reported, participants reported on their worry about any incontinence happening and any activities they avoided around that worry. Diaries were formatted to display one question on the screen at a time, requiring participants to select from a drop down list, or check appropriate boxes. Navigation from question to question was facilitated by the use of a “previous” and “next” button on each screen; however, for most questions, participants could not advance until the current question had been answered. For some sensitive questions, a “Prefer Not to Answer” option was available. These features were built into the diary program to reduce data errors and omissions.
Participants could access that day’s EMA diary at any time for a window of four hours, either completing it in one sitting, or completing it in chunks, as data were saved as the participant entered it. No information resided on the participant’s device. Completion reminders were sent via the original delivery method every 15 minutes during the four-hour window until either the EMA diary was complete, or the time window closed. Participants who accessed the diary after the window closed were unable to get into (or back into) that specific entry, and anything not completed was considered missing. Study staff monitored expected completion daily from actively enrolled participants and contacted anyone who missed more than two consecutive entries. Such monitoring was important given that subsequent data collection was attached to the completion of prior entries. In the event that a participant reported that they did not receive an entry (e.g. they were out of cellular tower signal range, or their phone malfunctioned), or they had a reasonable excuse as to why they could not complete the entry within the window (e.g. a family emergency or they were unexpectedly out of town) were permitted to complete their entry either outside the window on that same day, or they were given a replacement entry for the next day. We worked to accommodate participants where at all possible.
Participant compensation
Compensation amounts were structured using both standards of prior work conducted by the first author and acknowledgment of participant’s sharing of personal and potentially sensitive health information. Participants received $15.00/each for the completion of the enrollment and exit modules, and $0.50 for the submission of each EMA diary. To incentivize high completion of the diary arm, we added a $10.00 bonus for any participant who provided 90% or more of all expected diaries. In addition, individuals who completed all study arms received $30.00 to offset the cost of text messaging and web access during the study. In total, it was possible to earn $85.00 in this 30 day study. Participants were paid via gift card of their choosing at the end of the study. All non-completers were paid for whatever portions they completed.
Measures
For the purposes of feasibility examination in different ways, we analyzed different variables collected in the EMA diaries. Incontinence variables included: UI frequency and FI frequency (both: no times, one time, two times, 3–5 times or six or more times). Emotion variables included: daily affect (PANAS short form; positive mood [Cronbach’s alpha: 0.987] and negative mood [Cronbach’s alpha: 0.965], both 3-item using 5-point Likert type items [strongly disagree to strongly agree]; e.g. “I felt happy” or “I felt sad”) and health (single 4-point item: poor to excellent) [58, 59] and on days when UI or FI was reported, incontinece negativity (4-item using 5-point Likert type items [strongly disagree to strongly agree]; e.g. “I felt [anxious, bothered, stressed, frustrated] about having [urinary or fecal] incontinence today” [Cronbach’s alpha: 0.968 [UI] and 0.976 [FI]). We finally examined two daily behaviors (both no/yes): watching TV/movie and eating.
Analyses
We used descriptive statistics, including percentages, to examine participant compliance (e.g. the extent to which participants complete the study and provide data that aligns with study protocols), participant EMA diary completion (e.g., the daily, weekly and study level degree to which diaries are submitted), participant acceptability (e.g., perceptions of method ease and convenience, question length, interaction with study team and willingness to participate in future research: each on a 5-point strongly disagree to strongly agree Likert scale). Open ended, free text responses on the exit survey were used to additionally inform acceptability and to describe suggestions for future studies. We also engaged descriptive statistics to evaluate data capture. Finally, we used Spearman’s rank correlation–a nonparametric substitute for Pearson’s R when count data (e.g. the number of UI or FI events) are used [60]–and descriptive statistics to assess data accuracy. Effect size interpretation guidelines for Spearman’s rank correlation were taken from Schober et al. [61].
We examined reactivity (e.g., the extent to which participant data change as a function of being in the study) in three domains: participation, behavioral and emotional. Participation reactivity was operationalized using two indicators of study engagement, including EMA completion time (minutes) and total number of EMAs completed. We evaluated behavioral reactivity using a measure of daily UI and FI (both: no/yes), as well as a pair of daily activities, including watched TV or a movie and ate a meal (both no/yes). These chosen measures aided in comparing if and/or how reactivity may differ between incontinence and more routine activities. Emotional reactivity was characterized using positive and negative mood, as well as UI or FI incontinence negativity.
In all models, we used random intercept mixed effects conditional linear (for completion time, mood and incontinence negativity), Poisson (for number of EMAs contributed) or binary logistic (for UI, FI and daily activities) with time (in days) as the sole predictor of each outcome. A significant value indicated a change in reporting as a function of time. For any significant coefficient, we converted the estimate to a percent via the first derivative for better interpretation. Random intercepts allowed estimates to be adjusted for the multiple diaries provided by participants [62, 63]. All data analysis was performed in Stata 17.0 (critical p = 0.05) [64]. Data were de-identified prior to analysis [65].
Results
Participants and retention
A total of 90 individuals enrolled in the study (May 2020-November 2020), and 97.8% of these (N = 88) completed all three expected arms of the project. Two individuals completed enrollment, but were dropped from the study after completing, respectively, none of the EMAs and only the first nine EMAs. Both of these participants were contacted multiple times by the study coordinator via text message, email and voice call to troubleshoot any aspects of EMA completion that could have been challenging. Neither individual responded to any of the attempted contacts, and after ten days were removed from the study per study protocol.
Table 1 provides the demographic and background characteristics of the completed participants. Participants were primarily female identified (71.1%), White (86.7%) and had an average age of 35 (range: 19–72). About seven in ten lived independently and had a median highest education level of a college degree. All participants had residence in the United States, Canada, South Africa and the United Kingdom, and half were working part or full time (51.1%). About a third were single and not dating. Nearly all (88.0%) were heterosexual. Two-thirds of participants self-classified as community ambulators. While all participants reported either UI or FI in the past 4 weeks, 93.3% reported UI, 80.0% reported FI and 73.3% reported both. The median length of time between UI events was four hours or longer, and a week or longer between FI events. During incontinence accidents, participants most commonly reported leaking a “medium amount” of urine (53.3%) and a “little bit” of stool (38.8%). Nearly all participants managed clean up on their own (UI: 83.3%; FI: 64.4%).
Compliance
We analyzed compliance analysis at three levels. First, participants were asked to complete one EMA entries per day, resulting in seven expected submissions per week during the first three weeks of the study (e.g., Week 1: Days 1–7; Week 2: Days 8–14; Week 3: Days 15–21) and nine entries in the last week (e.g. Week 4; Days 22–30). Second, each participant was expected to complete a total of 30 entries by the end of the study. Third, at the study level, the 90 completed participants were expected to contribute 2700 (90 x 30) entries by the end of the study [66].
Compliance was excellent at all three levels. At the weekly level, participants submitted an overall mean of 6.64 (SD = 0.79; Median = 7) of the seven (94.6%) expected weekly diaries during the first three weeks of the study, and a mean of 8.53 (SD = 1.01; Median = 9) of the nine expected entries (94.7%) during the fourth week. At the participant level, the majority (84.4%: 76/90) submitted 28 or more of the total expected 30 diaries at the end of the study. At the study level, participants submitted 95.7% (2576/2700) of the total diaries expected.
A preliminary examination of the distribution of completion time at the weekly level (Mean: 7.2; Median: 7.0; Mode: 7.0) and the participant level (Mean: 29.3; Median: 30.0; Mode: 30.0) suggested that the variable was normally distributed. Therefore, values in Tables 2 and 3 are represented by the mean. Table 2 provides a detailed illustration of EMA as a function of a specific day in the study. On each day, a total 90 entries–one per participant–were expected. As shown in the third column “Number of EMAs submitted”) the total number of EMA submissions on each day ranged between 81 (90%) and 90 (100%) (daily median: 86; data not shown). As shown in Table 3, we examined EMA submission at both the participant and weekly levels as a function of demographic and incontinence history characteristics. There were no significant differences observed in the average number of EMAs submitted–at either level–by any of the chosen factors.
Completion time
We operationalized completion time as the number of minutes elapsed from when a participant accessed the first question (Fig 1, Panel A) to when they submitted the diary by providing an answer to the last question. A preliminary examination of the distribution at the and weekly level (Mean: 35.4; Median: 2.0; Mode: 2.0) and participant level (Mean: 36.1; Median: 2.0; Mode: 2.0) suggested that the variable was positively skewed. Therefore, values in Tables 2 and 3 are represented by the median. The overall median time to completion was two minutes and 75% of all responses were submitted within three minutes.
Table 2 provides a detailed illustration of EMA as a function of a specific day in the study. As shown in the second column (“Completion Time”) the median completion time on each day ranged between 1.8 minutes and 3.3 minutes. As shown in Table 3, we examined completion time at both the participant and weekly levels as a function of demographic and incontinence history characteristics. There were no significant differences observed in the completion time–at either level–by any of the chosen factors.
Reactivity
We evaluated reactivity in three domains–participation, behavioral and emotional–by using time (in days) as a predictor of each outcome. The results of these models are shown at the bottom of Table 3 (“Time Effect”). Time did not have a significant effect on the participation measures, meaning that variability in both EMA submission and completion time were not linked to the passage of study time. Of all behavioral measures, time significantly predicted number of reported UI events and number of eating events. UI reports decreased by 2.1% between the first and last day of the study, and eating events increased by 2.9% in that same time frame. Finally, of all emotional measures positive mood was significantly associated with time: the median reported level decreased by 1.4% over the study.
Acceptability
Table 4 displays participant acceptability ratings. Individuals highly rated their overall study experience: 90.9% agreed or strongly agreed (median: 4) that they “enjoyed participating in the study” and 88.5% agreed or strongly agreed that payment for their time was fair. Study team support was also highly an important aspects to participants. A majority strongly agreed that they were both “able to communicate with the study staff when needed” and “felt supported by the study team throughout the study.” Open-ended free text data support these themes; for example, one participant reported that they “liked being able to express [their] feelings regarding bladder and bowel control” and another suggested that it helped them “track [their] bowel issue more consistently.”
We additionally evaluated the technical aspects of the study that were important to supporting this population’s participation. Seven in ten strongly agreed that it was “easy to take the EMAs on [their] phone or tablet” and 80.2% disagreed to some extent that the EMAs were “too long” each day. Over 80% strongly agrees that they gave “truthful information” on their surveys and disagreed to some extent that they were “uncomfortable answering questions about urine or stool leaks.” Finally, three-quarters strongly agreed that they would participate in a study like this again in the future. Open-ended free text data support these themes; for example, one participant noted that they study was “quick and easy” and another liked that the study “did not require extensive involvement.” Other participants suggested that receiving an SMS with a link to the survey was helpful for remaining “compliant.”
Open-ended questions additionally revealed themes of gratitude over the focus on the topic of incontinence among this population. For example, one participant said that “more [studies like this] need to be conducted to understand difficulties of living with spina bifida and [dealing with] with bladder and bowel issues.” Another respondent echoed this sentiment: “Not enough research like this is being done…[it feels like] doctors are ‘throwing Jell-o at a tree’–with the tree being [those with SB]–to see what sticks in terms of helping.” Two additional participants suggested, respectively, that they were grateful “for [the study’s] taking the time to look into one our struggles that is least talked about,” and that “there is not enough interest in adults with spina bifida.” Five other participants emailed similar sentiments to the study manager after they had completed the study.
Data capture
Commensurate with our presentation of the compliance data, we evaluated EMA’s capabilities as a data collection tool by examining the volume of incontinence data at both the person- and EMA-levels.
At the participant level, 92.3% of individuals (84/91) reported any UI during the 30-day period and 79.1% (72/91) reported any FI during that same time frame. Focusing on the distribution of incontinence types, nearly all (97.1%: 89/91) were incontinent at some point during the study. Of these individuals, 19.1% (17/89) reported only UI, 5.6% (65/89) reported only FI, and 75.2% (67/89) reported both UI and FI. At the EMA level, any UI was reported on a total of 1163 days (45.1%: 1163/2578) and any FI was reported on a total of 508 days (19.7%: 508/2578). Focusing on the 1573/2578 days with any incontinence (61.0% of all days), 1065/1573 (67.7%) were UI only, 166/1573 (10.5%) were FI only, and 342/1573 (21.7%) were characterized with both UI and FI. Rates of opt-out for any of these questions were quite small (0.0%-0.8%).
Data accuracy
We observed a high degree of accuracy between UI and FI reported in the EMA diaries and retrospective reports of any past week UI or FI in the exit surveys. The number of EMA days on which UI was reported in the diaries was moderately correlated with reporting any past 4-week UI in the exit survey (ρ = 0.690; p < .001), and the number of FI events reported in the EMA diaries was strongly correlated with exits survey reports of any FI in the past four weeks (ρ = 0.813; p < .001) [61]. A majority (97.5%) of participants who reported any UI in the EMA diaries also reported past 4 week UI in the exit survey. Nearly all participants who noted any FI (98.5%) in the EMA diaries also reported FI in the exit survey.
Suggestions for future work
Participants identified several areas on which the study could be improved in future iterations. These suggestions fell into three different areas: 1) conducting an additional, longer term study; 2) additional questions to contextualize incontinence; and 3) more questions about individual health conditions.
The need for a follow-up study was addressed via-a-vis the altering of “normal life” during COVID. For example, one participant said ‘A lot of my normal activities living in [a major metropolitan city] were altered by COVID. Questions about social activities, including sex, would be answered differently pre-pandemic.” Participants also though a longer time would capture a more representative picture of life with SB: “I have a few good weeks in between my bad weeks. So I’m more prone to accidents when my back and nerve pain flare up.”
Participants provided suggestions for items to contextualize incontinence experiences in several ways. Multiple participants mentioned that the volume of food and/or drinks consumed often impacts both UI and FI. Two participants suggested adding questions about the degree of incontinence urgency as well as the time of day at which UI or FI happen. For example, one person noted “Sometimes I have feelings of urgency when I make it to the bathroom, and feelings of urgency when I don’t make it to the bathroom. These situations impact my anxiety and stress differently [and also] how I choose to do or not dop activities.” Another participant suggested asking about preparations in case of UI or FI: “[You should ask] Did you take the necessary items to change [if you went out]? Being prepared helps lower my stress.”
Many suggestions about querying individual health conditions. One participant noted that their medications “impact [their] incontinence” while another said that aspects of her menstrual cycle, like bloating and pain, “put pressure on [their] bladder [making me] more nervous/stressed/anxious about leaks than [during] other times of the month, even if I didn’t leak that day.” Multiple participants wanted to see questions about daily physical symptoms: “General numbness, neuropathic numbness and pain levels could also be evaluated, as that can have effects on incontinence.” Another participant said: “I know I feel more inclined to an accident when my legs feel like Jell-O.”
Discussion
Existing clinical and research data collection mechanisms typically lack the precision [12–15] to accurately describe the day-to-day experiences of UI and FI among individuals with SB. Because the success of interventions and/or education programs relies upon reliable and valid information, it is vital to engage data collection designs that capture UI and FI experiences with as much fidelity as possible to the ways in which individuals in this population actually live them. This preliminary study demonstrated that it is feasible to recruit an international sample of adults with SB to complete daily EMAs for a month that describe the behavioral and emotional context associated with their incontinence. Our results–scaffolded by suggestions from participants themselves–provide evidence of the likely success of longer term studies using this method.
An important pillar of a longitudinal study’s success is the extent to which participants can be recruited and retained into a study, and the level to which their data completion is compliant with the proposed schedule. We engaged fully online recruitment methods to recruit international sample of participants from across the age spectrum and retained 97.8% of them through all three arms of data collection. In addition, participants were highly compliant with data submission expectations, completing 95% of the expected number of weekly-, participant- and study-levels. These levels exceed published recommendations of completion (80%) as a means of avoiding data bias [67] and are on par with or exceed completion levels in our past EMA studies [16, 17], as well as those estimated in recent meta-analyses of EMA studies (43%-95%) [68, 69]. We attribute several aspects of study design as helping to achieve this high compliance, including providing compensation for individual diary completion and for completion of a total of 90% of diaries. While it is possible that compensation inflated diary submission, it is also important to note that research staff provided participant support, such as in-depth EMA orientation and contacting participants who missed more than one EMA, also contributed to our compliance levels [16, 17].
Behavior reactivity can be a concern in EMA studies because outcomes of interest are repeatedly assessed close to the context in which they occur, potentially changing the types of data participants provide over the course of a study. Sensitive behaviors, like sexual activity, substance use, or in this case, incontinence, may be particularly vulnerable to reactivity [70, 71]. Our data did show small reporting differences (<2.5%) in the number of UI events and the number of daily activities reported as a function of day in the study. While no studies have assessed EMA reactivity in adults with SB, these findings are in line with reactivity volume observed in EMA studies of substance use [70], chronic conditions like tinnitus [72] diabetes [73] or pain [25] and our own work with sexual behavior [16, 17]. It will be important to continue to evaluate possible reactivity effects in this population as the study reporting period is longer.
Perhaps most importantly, participants highly rated their experience in the study. A primary intent of our person-centered study design was to facilitate participation for individuals whom may otherwise face substantial barriers to clinical research [32, 33]. All three study arms emphasized web-enabled technology that is commonly owned among adults with SB [38, 39]. Daily usage of these devices [40–43] meant that participants could participate in various aspects of data entry without much additional training on the data entry tasks associated with the study [44]. Moreover, the entirely electronic data collection structure allowed the cohort to flexibly provide their data “anywhere and anytime” in spaces that were convenient to them, thereby minimizing the potential disruption to their daily lives [36, 44]. We were able to capture an ample volume of incontinence events over the 30 days, and participants on average tended to strongly agree that the surveys were easy to take and not too long. These features–data entry ease and flexibility in how/when data entry happens–have been noted in other studies to support participant comfort in disclosing sensitive information [28], and participants in our study reported they felt comfortable answering questions about urine and/or stool leaks. While we did not assess whether there was any difference in comfort based on a specific study arm (e.g. enrollment survey vs. EMAs vs. exit survey), our data do provide evidence that engaging a user-centered design holds promise for ongoing research in this population.
Finally, it is important to consider how this framework could be applied to understand health experiences in populations with similar illness trajectories, such as individuals with other mild developmental delays (e.g. Down’s syndrome), mobility issues (e.g. spinal cord injuries) or chronic incontinence associated with other health conditions (e.g. childbirth, cancer, Irritable Bowel Syndrome). EMA would be well suited, for example, to examine the severity of or life interference of chronic health conditions (e.g., pain [74]), or to assess the day-to-day mental health impact of a condition [75]. Careful attention would need to be given to population and/or patient specific needs and limitations, such as ensuring the cognitive and physical ability to participate [76] or evaluating any need to strengthen basic digital literacy prior to participation (e.g. using/navigating a smart phone, using the internet, etc.) [77]. It would also be important to ensure–as in any study, but especially in those with long-term and/or chronic illness–that individuals’ data are respected and kept private and secure, particularly when discovery could adversely impact other aspects of daily life [78].
Limitations
As with any study, there are limitations to consider. This study was limited to individuals 18 years and older. Incontinence can be a lifetime challenge for those with SB, and the nature of how it is experienced in adulthood may have links to experiences in earlier life stages. Future studies may benefit from inclusion of younger individuals as a means of situating incontinence within a developmental life stage lens. Moreover, our sample was primarily White, heterosexual, well-educated, and exhibited a high degree of living and mobility independence. There is possibility of self-selection bias within this set of characteristics, meaning that individuals who are more caregiver-dependent, or those with greater degrees of developmental delay, may have been less likely to be able to participate in the study. However, our participants had reasonably similar gender and neurological characteristics to other samples from SB-focused research [7, 79–81]. It will be important in future studies to understand the extent to which an EMA framework would need to be adapted to accommodate individuals with SB from different cultural or socioeconomic backgrounds. For example, participants in this study used their own phone for the duration of the study. Individuals experiencing financial hardship may not own a phone or have internet. One solution could be to build the cost of a temporary phone and monthly wireless service into the study design [82].
There are also limitations associated with the study design. Because our aims were to evaluate method feasibility, we did not assess the extent to which differences in design could impact the data collected. It will be important for future research to evaluate how cadence of EMA completion (for example, once per day vs. more frequently), a longer study length (e.g. 30 days vs. 120 days) or a different data collection schedule (e.g. continuous vs. weeks on/weeks off). Many of our participants noted that a longer study length would better capture the variability of their “daily lives,” but it remains unclear both how many more weeks of collection and how frequently these assessments would need to occur to fully gauge the bandwidth of experiences. We also engaged a combination of several different convenience and snowball sampling strategies to reaching potential participants. In addition to the usual caveats about non-probability sampling approaches, it is possible that some participants recruited from the same networks (e.g., members of the same SB state chapter, or patients in the same clinic) may have shared characteristics that further limited broad generalizability of our findings to individuals with SB who are outside of those networks.
Finally, it is impossible to verify if any participants provided answers with someone else’s help, and/or to what extent such help may have influenced answered given. Questions about both the context of incontinence itself, as well as how people feel about it, can be deeply personal and raise embarrassment. While individuals can be generally asked about incontinence in clinical encounters, we assessed specific dimensions about urine and stool leaks that could have made some people uncomfortable. However, participants were encouraged during the enrollment process to provide data in private settings, and the survey was built to allow individuals to opt out of most questions. Rates of opt-out were quite small (0.0%-0.8%), suggesting that most people who took the survey had little trouble with the content and nature of the questions.
Conclusion
The ability to keep participants engaged in research over an extended period of time is vital to capturing both a sufficient volume of data around outcomes of interest and the quality of data that reliably and validly reflect the daily experiences of the participants themselves. This study provided evidence that it is feasible to use electronic approaches to recruit a cohort of adults with SB to participate in a study to report daily incontinence via web-enabled data collection. Our cohort was highly compliant with study protocols, highly rated their study experience and provided suggestions on how to improve study approaches moving forward. We suggest that these data provide great promise for ongoing research in a population that can face barriers to clinical research participation.
Acknowledgments
We are grateful to the participants of this study for providing their information, and to Pat Brooks for outstanding support of our participants during the study.
References
- 1.
Spina Bifida Association. What is Spins Bifida?; 2022.
- 2.
National Institute of Neurologic Disorder and Stroke. Spina Bifida Fact Sheet. 2022.
- 3. Szymanski KM, Cain MP, Whittam B, Kaefer M, Rink RC, Misseri R. All incontinence is not created equal: impact of urinary and fecal incontinence on quality of life in adults with spina bifida. The Journal of urology. 2017;197(3):885–91.
- 4. Patel SK, Staarmann B, Heilman A, Mains A, Woodward J, Bierbrauer KS. Growing up with spina bifida: bridging the gaps in the transition of care from childhood to adulthood. Neurosurgical Focus. 2019;47(4):E16. pmid:31574470
- 5. Wiener JS, Suson KD, Castillo J, Routh JC, Tanaka S, Liu T, et al. Bowel management and continence in adults with spina bifida: Results from the National Spina Bifida Patient Registry 2009–15. Journal of pediatric rehabilitation medicine. 2017;10(3–4):335–43. pmid:29125526
- 6.
Wiener J, Routh J, Tanaka S, Souson K, Liu T, Ward E, editors. Urinary management and continence outcomes of adults with spina bifida and comparison to children: findings of National Spina Bifida Patient Registry (NSBPR). Third World Congress on Spina Bifda Research and Care San Diego, United States; 2017.
- 7. Szymanski KM, Misseri R, Whittam B, Raposo S-M, King SJ, Kaefer M, et al. QUAlity of Life Assessment in Spina bifida for Adults (QUALAS-A): development and international validation of a novel health-related quality of life instrument. Quality of Life Research. 2015;24(10):2355–64. pmid:25863685
- 8. Szymanski KM, Misseri R, Whittam B, Kaefer M, Rink RC, Cain MP. Quantity, Not Frequency, Predicts Bother with Urinary Incontinence and its Impact on Quality of Life in Adults with Spina Bifida. J Urol. 2016. pmid:26926556
- 9. Szymanski KM, Misseri R, Whittam B, Yang DY, Raposo S-M, King SJ, et al. Quality of life assessment in spina bifida for children (QUALAS-C): development and validation of a novel health-related quality of life instrument. Urology. 2016;87:178–84. pmid:26456744
- 10. Szymański KM, Misseri R, Whittam B, Casey JT, Yang DY, Raposo S-M, et al. Validation of QUALAS-T, a health-related quality of life instrument for teenagers with spina bifida. Central European journal of urology. 2017;70(3):306. pmid:29104796
- 11.
Stone AA, Shiffman SS. Ecological validity for patient reported outcomes. In: Steptoe A, editor. Handbook of Behavioral Medicine: Springer; 2010. p. 99–112.
- 12.
Morris I, Shiffman S, Beckjord E, Ferguson SG. Ecological Momentary Assessment and Technological Advances in Clinical Care. In: Potenza MN, Faust KA, Faust D, editors. The Oxford Handbook of Digital Technologies and Mental Health: Oxford University Press; 2020. p. 0.
- 13. Courvoisier DS, Eid M, Lischetzke T. Compliance to a cell phone-based ecological momentary assessment study: the effect of time and personality characteristics. Psychological assessment. 2012;24(3):713. pmid:22250597
- 14. Iida M, Shrout PE, Laurenceau J-P, Bolger N. Using diary methods in psychological research. 2012.
- 15. Talari K, Goyal M. Retrospective studies–utility and caveats. Journal of the Royal College of Physicians of Edinburgh. 2020;50(4):398–402. pmid:33469615
- 16. Hensel DJ, Fortenberry JD, Harezlak J, Craig D. The Feasibility of Cell Phone Based Electronic Diaries for STI/HIV Research. BMC Medical Research Methodology. 2012;12(75). pmid:22691189
- 17. Roth AM, Hensel DJ, Fortenberry JD, Garfein RS, Gunn JK, Wiehe SE. Feasibility and Acceptability of Cell Phone Diaries to Measure HIV Risk Behavior Among Female Sex Workers. AIDS and Behavior. 2014:1–11.
- 18. Shiffman S, Stone A, Hufford M. Ecolocial Momentary Assessment. Annual Review of Clinical Psychology. 2008;4:1–32.
- 19. Joseph NT, Jiang Y, Zilioli S. Momentary emotions and salivary cortisol: A systematic review and meta-analysis of ecological momentary assessment studies. Neuroscience & Biobehavioral Reviews. 2021;125:365–79. pmid:33662445
- 20. Shiffman S. Ecological momentary assessment. 2016.
- 21. Shiffman S. Ecological momentary assessment (EMA) in studies of substance use. Psychological Assessment. 2009;21(4):486–97. pmid:19947783
- 22. Russell MA, Gajos JM. Annual Research Review: Ecological momentary assessment studies in child psychology and psychiatry. Journal of Child Psychology and Psychiatry. 2020;61(3):376–94. pmid:31997358
- 23.
Hufford M. Special methodological challenges and opportunities in ecological momentary assessment. In: Stone AA, Shiffman S, Atienza A, Nebeling L, editors. The science of real-time data capture: Self-reports in health research Oxford, England: Oxford University Press; 2007. p. 54–75.
- 24. Todd KR, Shaw RB, Kramer JL, Martin Ginis KA. Using ecological momentary assessment to evaluate neuropathic pain experienced by adults with SCI: recommendations and participant perceptions. Disability and Rehabilitation. 2021;43(17):2439–46. pmid:31850813
- 25. May M, Junghaenel DU, Ono M, Stone AA, Schneider S. Ecological Momentary Assessment Methodology in Chronic Pain Research: A Systematic Review. The Journal of Pain. 2018.
- 26. Hufford MR, Shiffman S. Methodological issues affecting the value of patient-reported outcomes data. Expert review of pharmacoeconomics & outcomes research. 2002;2(2):119–28. pmid:19807324
- 27. Robbins ML, Kubiak T, Mostofsky D. Ecological momentary assessment in behavioral medicine: Research and practice. The handbook of behavioral medicine. 2014;1:429–46.
- 28. Hensel DJ. The benefits of electronic diaries in understanding the experience of health. Sexually Transmitted Infections. 2014;90:352–3 pmid:24813780
- 29. Nam S, Griggs S, Ash GI, Dunton GF, Huang S, Batten J, et al. Ecological momentary assessment for health behaviors and contextual factors in persons with diabetes: A systematic review. Diabetes Research and Clinical Practice. 2021;174:108745. pmid:33713720
- 30. Jylkkä J, Soveri A, Laine M, Lehtonen M. Assessing bilingual language switching behavior with Ecological Momentary Assessment. Bilingualism: Language and Cognition. 2020;23(2):309–22.
- 31. Holube I, von Gablenz P, Bitzer J. Ecological Momentary Assessment in Hearing Research: Current State, Challenges, and Future Directions. Ear and Hearing. 2020;41:79S–90S. pmid:33105262
- 32. Rios D, Magasi S, Novak C, Harniss M. Conducting accessible research: including people with disabilities in public health, epidemiological, and outcomes studies. American journal of public health. 2016;106(12):2137–44. pmid:27736212
- 33. Morley CP, Struwe S, Pratte MA, Clayton GH, Wilson PE, Dicianno BE, et al. Survey of US adults with spina bifida. Disability and health journal. 2020;13(2):100833.
- 34. Gressgard LJ, Stensaker I. The mobile service industry: strategic challenges and future business models. International Journal of Mobile Communications. 2006;4(5):509–31.
- 35. Watson RT, Pitt LF, Berthon P, Zinkhan GM. U-commerce: expanding the universe of marketing. Journal of the Academy of Marketing Science. 2002;30(4):333–47.
- 36. Heron KE, Everhart RS, McHale SM, Smyth JM. Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: A systematic review and recommendations. Journal of pediatric psychology. 2017;42(10):1087–107. pmid:28475765
- 37.
Kane SK, Jayant C, Wobbrock JO, Ladner RE, editors. Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility; 2009: ACM.
- 38. Dicianno BE, Gaines A, Collins DM, Lee S. Mobility, assistive technology use, and social integration among adults with spina bifida. American journal of physical medicine & rehabilitation. 2009;88(7):533–41. pmid:19542778
- 39. Dicianno BE, Fairman AD, McCue M, Parmanto B, Yih E, McCoy A, et al. Feasibility of using mobile health to promote self-management in spina bifida. American journal of physical medicine & rehabilitation. 2016;95(6):425–37. pmid:26488144
- 40. Dicianno BE, Swana HS, Cooper RA, Brei TJ. Innovations in Telemedicine Services in Spina Bifida Clinics in the US During the Covid-19 Pandemic. Technology & Innovation. 2022;22(2):157–64.
- 41. Stiles-Shields C, Garcia B, Villota K, Wartman E, Winning AM, Holmbeck GN. Exploring an Existing Weight Management App for Use With Adolescents and Young Adults With Spina Bifida: Usability Study. JMIR Pediatr Parent. 2019;2(2):e15153. pmid:31603432
- 42. Martin AJ, Strnadová I, Loblinzk J, Danker JC, Cumming TM. The role of mobile technology in promoting social inclusion among adults with intellectual disabilities. Journal of Applied Research in Intellectual Disabilities. 2021;34(3):840–51. pmid:33675142
- 43. Stiles-Shields C, Kritikos TK, Ridosh MM, Starnes M, Holmbeck GN. “We Are Anxious Every Day”: COVID-19 Impacts on Youth with Spina Bifida. Journal of Pediatric Psychology. 2021;46(9):1040–50. pmid:34350961
- 44. Kuntsche E, Labhart F. Using Personal Cell Phones for Ecological Momentary Assessment. European Psychologist. 2013;18(1):3–11.
- 45. Elharake JA, Akbar F, Malik AA, Gilliam W, Omer SB. Mental Health Impact of COVID-19 among Children and College Students: A Systematic Review. Child Psychiatry & Human Development. 2022. pmid:35013847
- 46. McKeon A, McCue M, Skidmore E, Schein M, Kulzer J. Ecological momentary assessment for rehabilitation of chronic illness and disability. Disability and rehabilitation. 2018;40(8):974–87. pmid:28637126
- 47. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit in real life evaluated using ecological momentary assessment. Trends in hearing. 2021;25:2331216521990288. pmid:33787404
- 48. Granholm E, Holden JL, Mikhael T, Link PC, Swendsen J, Depp C, et al. What do people with schizophrenia do all day? Ecological momentary assessment of real-world functioning in schizophrenia. Schizophrenia bulletin. 2020;46(2):242–51. pmid:31504955
- 49. Rabinowitz A, Hart T, Wilson J. Ecological momentary assessment of affect in context after traumatic brain injury. Rehabilitation psychology. 2021. pmid:34516166
- 50. Paterson C. An ecological momentary assessment of self-management in prostate cancer survivors. Journal of Cancer Survivorship. 2019;13(3):364–73. pmid:31115766
- 51. Weinland SR, Morris CB, Hu Y, Leserman J, Bangdiwala SI, Drossman DA. Characterization of episodes of irritable bowel syndrome using ecological momentary assessment. Official journal of the American College of Gastroenterology| ACG. 2011;106(10):1813–20.
- 52. Lehmann J, Schreyer I, Riedl D, Tschuggnall M, Giesinger JM, Ninkovic M, et al. Usability evaluation of the Computer-Based Health Evaluation System (CHES) eDiary for patients with faecal incontinence: a pilot study. BMC Medical Informatics and Decision Making. 2022;22(1):81. pmid:35346170
- 53. Hensel DJ, Hummer TA, Acrurio LR, James TW, Dennis FJ. Feasibility of Functional Neuroimaging to Understand Adolescent Women’s Sexual Decision Making. Journal of Adolescent Health. 2015;56(4):389–95. pmid:25595129
- 54. Hensel DJ, Von Hippel CD, Sandidge R, Lapage CC, Zelin NS, Perkins RH. “OMG, Yes!”: Feasibility, Acceptability, and Preliminary Efficacy of an Online Intervention for Female Sexual Pleasure. The Journal of Sex Research. 2022;59(3):269–82. pmid:34176390
- 55.
Qualtrics XM. Qualtrics. Provo, Utah2023.
- 56. Roberts JK, Pavlakis AE, Richards MP. It’s more complicated than it seems: Virtual qualitative research in the COVID-19 era. International Journal of Qualitative Methods. 2021;20:16094069211002959.
- 57. Tennen H, Affleck G, Coyne JC, Larsen RJ, DeLongis A. Paper and plastic in daily diary research: Comment on Green, Rafaeli, Bolger, Shrout, and Reis (2006). Psychological Methods. 2006;11(1):112–8. pmid:16594771
- 58. Juth V, Smyth JM, Santuzzi AM. How do you feel? Self-esteem predicts affect, stress, social interaction, and symptom severity during daily life in patients with chronic illness. Journal of health psychology. 2008;13(7):884–94. pmid:18809639
- 59. Van Voorhees EE, Dennis PA, Elbogen EB, Fuemmeler B, Neal LC, Calhoun PS, et al. Characterizing anger-related affect in individuals with posttraumatic stress disorder using ecological momentary assessment. Psychiatry Research. 2018;261:274–80. pmid:29329048
- 60. Sedgwick P. Spearman’s rank correlation coefficient. Bmj. 2014;349. pmid:25432873
- 61. Schober P, Boer C, Schwarte LA. Correlation Coefficients: Appropriate Use and Interpretation. Anesthesia & Analgesia. 2018;126(5):1763–8. pmid:29481436
- 62.
Brown H, Prescott R. Applied mixed models in medicine: John Wiley & Sons; 2014.
- 63.
Verbeke G, Molenberghs G. Linear mixed models for longitudinal data: Springer Science & Business Media; 2009.
- 64.
StataCorp LP. Stata. 18.0 ed. College Station, TX.: StataCorp LP; 2023.
- 65. Hensel DJ. Ecological momentary assessment to understand the impact of daily urinary and fecal incontinence on well-being among adults with spina bifida: Open Science Framework [distributor]; 2023 [Available from: https://osf.io/gefqx/.
- 66. Hensel DJ, Fortenberry JD. A Multidimensional Model of Sexual Health and Sexual and Prevention Behavior Among Adolescent Women. Journal of Adolescent Health. 2013;52(2):219–27. pmid:23332488
- 67. Stone AA, Shiffman S. Capturing Momentary, Self-Report Data: A Proposal for Reporting Guidelines. Annals of Behavioral Medicine. 2002;24(3):236. pmid:12173681
- 68. de Vries LP, Baselmans BML, Bartels M. Smartphone-Based Ecological Momentary Assessment of Well-Being: A Systematic Review and Recommendations for Future Studies. Journal of Happiness Studies. 2021;22(5):2361–408. pmid:34720691
- 69. Ono M, Schneider S, Junghaenel DU, Stone AA. What Affects the Completion of Ecological Momentary Assessments in Chronic Pain Research? An Individual Patient Data Meta-Analysis. J Med Internet Res. 2019;21(2):e11398. pmid:30720437
- 70. Carr DJ, Adia AC, Wray TB, Celio MA, Pérez AE, Monti PM. Using the Internet to access key populations in ecological momentary assessment research: Comparing adherence, reactivity, and erratic responding across those enrolled remotely versus in-person. Psychol Assess. 2020;32(8):768–79. pmid:32437190
- 71. Hufford MR, Shields AL, Shiffman S, Paty J, Balabanis M. Reactivity to ecological momentary assessment: An example using undergraduate problem drinkers. Psychology of Addictive Behaviors. 2002;16(3):205–11. pmid:12236455
- 72. Henry JA, Galvez G, Turbin MB, Thielman EJ, McMillan GP, Istvan JA. Pilot study to evaluate ecological momentary assessment of tinnitus. Ear Hear. 2012;33(2):179–290. pmid:21960147
- 73. Castañeda H, Melo MA. Health care access for Latino mixed-status families: Barriers, strategies, and implications for reform. American Behavioral Scientist. 2014;58(14):1891–909.
- 74. Overton M, Ward S, Swain N, Falling C, Gwynne-Jones D, Fillingim R, et al. Are Ecological Momentary Assessments of Pain Valid and Reliable? A Systematic Review and Meta-analysis. The Clinical Journal of Pain. 2023;39(1):29–40. pmid:36524770
- 75. Perski O, Keller J, Kale D, Asare BY-A, Schneider V, Powell D, et al. Understanding health behaviours in context: A systematic review and meta-analysis of ecological momentary assessment studies of five key health behaviours. Health psychology review. 2022;16(4):576–601. pmid:35975950
- 76. Kratz A, Kalpakjian C, Hanks R. Are intensive data collection methods in pain research feasible in those with physical disability? A study in persons with chronic pain and spinal cord injury. Quality of Life Research. 2017;26:587–600. pmid:28097459
- 77. Schreurs K, Quan-Haase A, Martin K. Problematizing the digital literacy paradox in the context of older adults’ ICT use: Aging, media discourse, and self-determination. Canadian Journal of Communication. 2017;42(2):359–77.
- 78. Gromatsky M, Sullivan SR, Spears AP, Mitchell E, Walsh S, Kimbrel NA, et al. Ecological momentary assessment (EMA) of mental health outcomes in veterans and servicemembers: A scoping review. Psychiatry research. 2020;292:113359. pmid:32777594
- 79. Lemelle J, Guillemin F, Aubert D, Guys J, Lottmann H, Lortat-Jacob S, et al. Quality of life and continence in patients with spina bifida. Quality of Life Research. 2006;15(9):1481–92. pmid:17033913
- 80. Schechter MS, Liu T, Soe M, Swanson M, Ward E, Thibadeau J. Sociodemographic attributes and spina bifida outcomes. Pediatrics. 2015;135(4):e957–e64. pmid:25780069
- 81. Sawin KJ, Liu T, Ward E, Thibadeau J, Schechter MS, Soe MM, et al. The National Spina Bifida Patient Registry: profile of a large cohort of participants from the first 10 clinics. The Journal of pediatrics. 2015;166(2):444–50. e1. pmid:25444012
- 82. Sevelius JM, Gutierrez-Mock L, Zamudio-Haas S, McCree B, Ngo A, Jackson A, et al. Research with Marginalized Communities: Challenges to Continuity During the COVID-19 Pandemic. AIDS and Behavior. 2020;24(7):2009–12. pmid:32415617