Step 1—Pre-clinical evaluation of content validity.

For step 1, subject experts (n = 10) were recruited to form a subject expert group to evaluate eVIS’s content validity [31, 41]. A strategic recruitment of study participants was conducted to achieve a broad representation of the eVIS target population, combined with documented expertise that was either clinical or research-based within the fields relating to eHealth, physical activity, behavior change, outcome measures etc. The expertise area was verified by the initial questions in the questionnaire used for Assessment 1. Based on literature recommendations, the number of experts was set à priori to a minimum of 10, to avoid the risk of chance agreement [32, 41]. The group consisted of patient representatives (n = 4), experienced clinicians at specialist and primary care IPRP units (n = 4), and researchers (n = 2). Some were both clinicians and researchers. Patients were representative of the target population and had a chronic pain diagnosis, three of them for more than 5 years and one between 2–5 years, and experience of interdisciplinary pain rehabilitation. The clinicians and researchers specialized in IPRP; most of them had over five years of clinical experience from the targeted intervention. Researchers had research experience in the fields of pain and rehabilitation, physical activity, validation studies, implementation research and technical innovation, and eHealth. Five of the experts (both patients and clinicians) had previous experience of using activity trackers for either personal use or research.

Step 2—Evaluation of content validity and feasibility in a clinical context.

In step 2, subject experts (n = 10) provided real-life experience data on their trial of eVIS. The subject experts were specialized physiotherapists (n = 3) from three different clinical settings across Sweden and a convenience sample of patients (n = 7) who were recruited within the IPRP setting (internal allocation was 1:1, 1:3, and 1:3). In the ethical permit there was an à priori decision to include one to three clinics, each including one to three patient participants. Recruitment of subject experts in step 2 was partly hindered because of reduced activity in tertiary care due to the ongoing Covid-19 pandemic. Patients were recruited based on the following criteria: accepted onto an IPRP due to non-malignant, musculoskeletal, or widespread pain for more than three months, aged 18–67, good ability to comprehend information and instructions given in Swedish, and daily access to a web browser using a computer, smart phone, or tablet. Patients in need of walking aids were not included. Two of the patients had previous experience of using activity trackers.

By the end of step 2, further expert interviews were needed regarding two areas emerging as incomplete in previous assessment processes (recruitment logistics, and pharmaceutical report function). Therefore, the physiotherapists were again invited as subject experts in a follow-up interview, of which two were able to participate. Additionally, subject experts (n = 3) with experience in pharmaceutical pain management of the specific population were recruited though our research group to validate specifics of the pharmaceutical report function. They were experienced physicians and researchers representing different regions in Sweden, and they participated in video-interviews.

All experts provided informed consent before taking part in the study and were told that they could withdraw at any time without giving a reason. Confidentiality was protected throughout the project. Data was stored at a project-specific server at Dalarna University, which prevented unauthorized access and was regularly backed up.

Consensus panel.

A consensus panel handled the results from subject expert assessments and detailed the revisions. In step 1, the panel consisted of five health researchers (the author group) and one researcher specializing in visual perception. The competence in the consensus panel included: chronic pain and IPRP evaluation, physical activity and activity measurement, experience in epidemiological and experimental investigations, questionnaire and intervention development, eHealth and audiovisual communication, design, and production. In addition to the researchers, Nordforce Technology AB was present at all panel meetings to consult on technical and user-experience revision aspects. In step 2, a core consensus panel consisting of three of the researchers from author group remained throughout all steps of the study, but could, if deemed necessary, seek external specific expert advice, for consultation before specific revisions.

Hence, subject experts and consensus panel covered a multitude of expertise of relevance for the study’s objective, e.g., experience of living with chronic pain, clinical pain rehabilitation, research in physical activity, research of patient-reported outcomes, image production, and software development.

Step 1—Pre-clinical evaluation of content validity.

To determine important aspects of content validity, a 4-point ordinal rating scale is commonly used [31, 32, 41]. It can be quantified to Content Validity Index (CVI), signifying the expert raters’ proportion of agreement on an item’s validity. The content validity of eVIS was evaluated based on three validity aspects: relevance, simplicity, and safety [31–34, 42, 43]. Relevance here refers to the extent that the proposed intervention and its included elements seem relevant as a means for facilitating physical activity and improved physical health in the specific population. Similarly, simplicity refers to whether the intervention is easy to conduct, the clarity of instruction etc., and safety refers to whether any potential risks can be identified in using the intervention, for instance the risk of adverse events. Within the context of instrument development, the relevance and simplicity of items constitute common validity aspects for rating, but when used in the context of an intervention validation, safety is of equal significance, and has previously been used within our research group [33].

The relevance, simplicity, and safety was evaluated for the three elements of eVIS: Data collection, Visualization and Communication, and for the eVIS intervention in its entirety using a four-point Likert scale ranging from 1 to 4; 1: Not relevant/simple/safe, 2: Needs revision, 3: Relevant/Simple/Safe but needs minor alteration, 4: Very relevant/simple/safe. The Content validity index (CVI) was then used as quantitative indicator of content validity, operationalized as item-level CVI (I-CVI), scale-level CVI average (S-CVI/Ave), and scale-level CVI average/domain (S-CVI/Ave Domain) [31, 32].

Step 2—Evaluation of content validity and feasibility in a clinical context.

The content validity of eVIS in a clinical context, step 2, was evaluated based on the same three validity aspects as used in step 1: relevance, simplicity, and safety. The feasibility of eVIS, step 2, was evaluated based on five focus areas proposed by Bowen [29] to answer the overarching question “Can it be done?” These focus areas were: practicality (i.e., to what extent the intervention can be carried out with intended participants using existing means, resources without outside intervention), acceptability (i.e., to what extent the intervention is judged as suitable, satisfying or attractive to intervention deliverers and recipients), implementation (i.e., to what extent the intervention can be successfully delivered to intended participants in some defined but not fully controlled context), limited efficacy testing (i.e., does the intervention show promise of being successful with the intended population), and demand (i.e., to what extent the intervention is likely to be used?) [29]

Feasibility of the eVIS intervention in its entirety and its domains was assessed using a four-point Likert scale ranging from 1 to 4, (1: Not at all, 2: To some extent, 3: To a rather large extent, 4: To a large extent). The last item in the patient and physiotherapist questionnaires (Item P-F17 and PT-F23 respectively) was rated 1–4 to reflect the extent of agreement with a statement; 1: Strongly disagree, 2: Disagree, 3: Agree, 4: Strongly agree.

Step 1—Pre-clinical evaluation of content validity.

A web-based questionnaire was used for the expert group to independently assess the content validity of the eVIS intervention in three assessment rounds, every second week over a period of six consecutive weeks. Three rounds of assessment were planned a priori to enable repeated input and enough time and resources for an iterative development process, with revisions made between rounds and modifications again evaluated. Prior to the first assessment round, the rationale for the validation study and the validation process were presented to the participating experts through an online video-meeting along with a brief introduction of the eVIS and designated webpage, for the experts to reference during assessment.

In the questionnaire the eVIS intervention in its entirety plus its three elements (hereafter together referred to as domains) were presented in text, photos, and figures followed by 28 questions (in analyses referred to as content validity items), which were rated regarding relevance, simplicity, and safety, amounting to 78 ratings (for seven items the safety dimension was irrelevant and therefore not included). In addition to every quantitative rating there was a commentary field for additional free-text comments. The questionnaire concluded with three open-ended questions asking whether any part of eVIS should be excluded, or any additional part should be included, and whether there were any safety aspects that the experts wanted to highlight.

Between assessment rounds, ratings and free-text comments were examined by a consensus panel and revisions were made before the following assessment. The consensus panel evaluated the results and agreed on a priority list, sorted in a “need to change” or “nice to change”- order. When possible, within economical restrictions and timeframes, as many of the “nice to change” suggestions were also taken into account, as it was evident these would be of value. In the second round, the questionnaire consisted of only 20 items as eight items had been omitted, so only those that had had revisions made were rated. In the third round the full questionnaire was again rated.

Step 2—Evaluation of content validity and feasibility in a clinical context.

To enable real-life clinical evaluation of the feasibility of eVIS within IPRPs, the intervention was implemented in its intended setting and tested for 2–3 weeks. The implementation was supported by a designated webpage with instructions.

Multiple data sources were used for the assessment: First, an analogue diary was used to record physiotherapists’ and patients’ momentary notes on experiences during the start up and trial of eVIS. Any telephone calls and e-mails from participants directed to the support function of eVIS were also used as a source of information at this step. Secondly, after the testing period had ended, two different web-based questionnaires (one for patients and one for physiotherapists) were used to assess the content validity and clinical feasibility of eVIS, as perceived from the participants’ practical experience of testing it. Five items from the content validity questionnaire, previously used in step 1 –evaluation–were included (C11, C14, C19, C24, and C25 –see items in Table 2); these were three overall items for rating the domains Data collection, Visualisation and Communication, and eVIS objective in its entirety, and one item specifically targeted at the pharmaceutical report function in PATRON (C11). These items were identical in both questionnaires and were evaluated by I-CVI. The feasibility questions (feasibility items, F) were diversified according to respondent category (patient or physiotherapist) and covered the domains and overall implementation of eVIS from patient and caregiver perspectives, resulting in 17 and 23 items respectively (due to the latter being complemented with specific items on recruitment and the start up process). Feasibility items addressed the perceived feasibility of eVIS in relation to our chosen focus areas; acceptability, demand, implementation, limited efficacy, and practicality. All items had ratings and commentary fields.

Following finalization, exports of participant data records in PATRON were used to validate the export procedure (to pilot the functionality and accuracy of the eVIS data registration process and subsequent data export for analysis, i.e., testing of the entire data collection step).

Finally, video interviews were used for additional assessments on two specific areas as a need for immersed evaluation was identified during the analysis process. Details of the recruitment and initiation procedures (recruitment logistics) were qualitatively reviewed by physiotherapists, who participated in a group video interview. In addition, three individual video interviews with physicians with expertise in clinical pain management pharmacology were carried out, addressing the comprehensiveness of the pharmaceuticals list provided in PATRON as well as the rating of content validity of the pharmaceutical report function (item C11 from previous questionnaires) by means of physicians’ medical expertise.

Step 1—Pre-clinical evaluation of content validity.

An Item-level CVI (I-CVI) was calculated for every validity aspect of every item, with values ranging from 0 to 1. In accordance with recommendations by Polit et al., an I-CVI ≥0.78 was selected as cut-off for an excellent I-CVI [32]. As a measure of inter-rater agreement on the overall relevance, simplicity, and safety of eVIS, a scale-level CVI average (S-CVI/Ave) was calculated. S-CVI/Ave implies the average I-CVI across all items in the questionnaire, (sum of all I-CVIs divided by number of I-CVIs) and respectively a S-CVI/Ave Domain, signifying the average I-CVI across items in a domain (eVIS data collection, visualization, communication, eVIS as a whole). A S-CVI/Ave of ≥0.9 was considered to be “excellent” content validity. Additionally, the Scale-level CVI universal agreement (S-CVI/UA) was used to evaluate agreement between experts (sum of all I-CVIs equal to 1 divided by number of I-CVIs) and shows the proportion of items that attained a rating of ≥3 from all the experts, with ≥0.8 considered as excellent [31, 32]. Free-text comments adjacent to the item ratings were interpreted at manifest level, close to the content, and supplemented the interpretation of the ratings.

Step 2—Evaluation of content validity and feasibility in a clinical context.

The content validity in a clinical context was analysed based on I-CVIs from five items, calculated in the same way as in step 1, and adjacent narrative comments.

Quantitative data from the feasibility items in the patient and physiotherapist questionnaires were analysed descriptively and reported as the frequency and range of ratings for every item. The feasibility of an item was considered satisfactory if ratings were ≥3 (3: To a rather large extent/Agree or 4: To a large extent/ Strongly agree). To structure the investigated aspects of feasibility, items were categorized based on Bowen’s suggested focus areas: acceptability, demand, implementation, limited efficacy testing, and practicality [29]. The categorization of feasibility items was performed independently and blinded by two of the researchers, and any disagreements on categorization were discussed and resolved through consensus. Free-text comments adjacent to ratings in questionnaires complemented the interpretation. Qualitative data collected from the diaries, e-mails, and support function were examined with a focus on identifying areas in need of attention with regards to feasibility and avoidance of adverse events.

Handling of missing data.

No specific analyses of missing data were performed in this study. Reasons for missing data were noted if known and presented. In step 1, the I-CVI calculations in the final round were based on eight respondents (of ten) due to two of the participants not being able to attend. Also, in items C22, C24, and C25, there were occasional missing data as one of the respondents declared in the text field that it was not possible to rate these items as there was no option to omit a rating, leading to an arbitrary number being chosen. I-CVI calculations were in this case recalculated with a missing value instead and with initial number respondents in the denominator. In the evaluation of feasibility, in step 2, there was partial missing data on questionnaires and diaries. Two of the participants did not respond to the questionnaire due to sickness or an unknown reason. Two of the diaries were not submitted to the research team.

Step 1—Pre-clinical evaluation of content validity.

According to Polit & Beck (2007) as well as Grant (1997), the soundness of the interrater estimate in content validity is strongly influenced by the selection of experts [32, 41]. We therefore followed recommendations to assure relevant subject expertise was included by recruiting subjects from essential fields relating to the rehabilitation of chronic pain and intervention development. In addition, to adjust the risk of inflated estimates, we ascertained that the study incorporated a sufficient number of experts [41, 44]. Despite such efforts, we identified a lack of input on the pharmaceutical report function of eVIS. This was remediated through video interviews with specifically recruited subject experts, resulting in a deepened evaluation and refinement on the pharmaceutical report function.

Following recommendations, [31, 32, 43] we collapsed the four response options into two categories for our CVI calculation. It has been argued that less detail in information is derived from the interrater estimate [44]. However, in this study, expert ratings were to a large degree coherent. Moreover, valuable addition from rich free-text comments provided enough detail to guide the iterative development and evaluation process.

Step 2—Evaluation of content validity and feasibility in a clinical context.

To our knowledge, methodological guidelines on how to conduct feasibility evaluations are scarce, despite strong recommendations to conduct such studies. We evaluated vital parts of the intervention, rated it after an actual test period, and categorized our feasibility items in terms of five relevant focus areas [29]. We used robust methodology to ensure consensus in the categorization of included items in both the patient and physiotherapist questionnaires. Still, these may be open to interpretation, especially since some items are related to different focus areas depending on whose perspective they represent.

We used diverse data sources (questionnaires, diaries, interviews, and support errands) to increase input on eVIS feasibility. The sample providing assessments on feasibility was rather small, but as the evaluation constituted part of a larger evaluation process, the basis was deemed to be sufficient. Only three physiotherapists participated, however, their ratings were based on their accumulated experience of introducing eVIS to multiple patients, which gave them a broader understanding of the intervention’s applicability to a heterogenous population. To minimize the potential loss of information, we synthesized feasibility ratings descriptively.

Content validity of eVIS.

We found consensus already from the first assessment that eVIS to the main part was considered valid, based on item level indicators of relevance, simplicity and safety, and with only a few items below our cut off. Patients, caregivers, and researchers were generally positive towards the features on which eVIS was based.

Regarding the relevance of eVIS, the key uncertainties and focus of attention was on the included self-reported measures. We found that the construct of self-efficacy was not conceptually relevant for daily measuring. Instead, a more tangible question about perceived interference with daily activities, [39] replaced the initial PSEQ-2, which also had the benefit of being validated in Swedish. Another aspect related to the self-reported measures was the daily assessment of pain intensity. It was indicated by some clinicians as possibly counteractive to the IPRP concepts, as frequently focusing on pain assessment might increase patients’ perceived pain intensity. We found no evidence in support of reactive effects from daily pain assessment. Instead, repeated reporting of pain reportedly improves measurement reliability [45]. Moreover, evidence suggests the reflective act of recording data may result in increased understanding of the variability of pain, as well as recognition of time with less pain, and the observed relationship between pain intensity and activity, [46] which in itself can be perceived as a validation of patients’ experience. Moreover, it is reasonable to believe that pain intensity to some degree will co-vary with activity and interference and therefore is of importance to study, in order to increase our understanding of how these factors interact.

Simplicity was the part that needed most attention, based on our results. To increase usability and satisfaction with the system, we carried out extensive revisions based on repeated, formative assessments. A strength in this process was the help we received from relevant experts, not only from the point of view of subject experts but also from the field of audio-visual communication, giving guidance, for instance on avoiding unintentional signalling through our choice of colours. Altogether, and in dialogue with the software developers, the interface improved markedly.

Safety emerged with high ratings in almost all areas, however, during the clinical testing a safety breach occurred which resulted in lower ratings and comments from both patients and physiotherapists. Although we did not anticipate any major safety issues, safety was an important aspect to validate. In conjunction with relevance and simplicity, commonly used for evaluation in questionnaire development, safety aspects play an important role in intervention development. From previous experiences of validating a safety-critical intervention [33] we were certain that the eVIS intervention would also benefit from systematic validation from subject expertise to identify possible overlooked risks and increase safety confidence. Contrary to many other eHealth applications, eVIS and PATRON do not include specific treatment advice or exercise modules, as this is handled within the clinical setting. Hence, the important parts of our validation process were related to patient integrity. Following discussions on the question “who has access to my daily data?” reinforced our intention to only enable clinicians to see data when patients themselves logged in and showed them, instead of clinicians having access to monitor or “survey” from a distance. Moreover, the incidence during step 2 evaluation resulted in important actions being taken to prevent re-occurrence.

Feasibility of eVIS.

Results from our clinical evaluation of eVIS were in line with other recent feasibility evaluations of digital tools designed to support patients with chronic pain and chronic diseases [47–49]. Outcomes such as system use, usability, acceptability, and ease of use are emphasized as important factors in the achievement of high adherence to digital self-management tools [48]. In our study, stakeholders assessed eVIS as in demand, acceptable, practical, and possible to implement within IPRP, which are important factors to maintain a high adherence to the intervention over time [47, 50]. Moreover, Ricciardo & Pandya (2020) concluded that the perceived ease of use and features facilitating knowledge acquisition were key features of an eHealth intervention designed to facilitate patient self-management [49]. In an evaluation of patients with chronic diseases and their ‘needs and requirements when using eHealth and self-management, individual tailoring of the tool are addressed as paramount [48]. In 2018, Karlsson et al. published a report on how patients living with chronic pain experience physical activity and exercise [50]. In the report, it was confirmed that many patients living with chronic pain experience significant barriers to performing physical activity. Factors such as motivation, self-efficacy, action control (i.e., transferring intention into action to accomplish a certain behaviour), interactions, and profuse contact with healthcare providers are emphasized as necessary requirements that need to be addressed in the process of the individualization of physical activity level.

Patients and physiotherapists rated the limited-efficacy test as high, suggesting a clinical demand for a tool such as eVIS. Contrary to many other eHealth interventions, eVIS is not intended for patients to use on their own and it does not provide any individualized advice or exercises, rather, it has been developed for use in the IPRP context. We believe the IPRP setting, with physiotherapists and other experts from the interdisciplinary team, e.g., occupational therapists, psychologists, or physicians, are an essential part of the successful and effective use of eVIS. For patients with chronic and complex pain conditions, to set and adapt an individualised physical activity goal necessitates expertise and collaboration with the patient, taking into account personal resources, intentions, and available time plan. It appears eVIS may be a useful tool in bridging IPRP clinical context with treatment individualisation and self-management.