Search strategy

A comprehensive search strategy was developed, reviewed and refined by the authors, with the assistance of a health librarian, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [29] (S1 Fig). An electronic database search of PubMed, EMBASE, CINAHL and PsycINFO via EBSCOhost was conducted on the 28^th of June 2022. Key search terms were developed using four individual search filters. These filters included:

1. Population: chronic conditions OR chronic disease OR chronic health OR chronic illness OR long term illness OR long term disability OR long term condition
2. Construct: physical function OR physical performance OR functional capacity OR physical capacity
3. Measurement Instrument: assessment OR evaluation OR outcome OR measure OR test
4. Context: telehealth OR telerehabilitation OR telemedicine OR e-health

These individual filters were combined with the COSMIN search filter for measurement properties [30] to create the search strategy outlined in S2 Fig. Hand-searching of the reference lists of the included articles was also performed to identify additional relevant articles.

Eligibility criteria

Studies were included in the review if they met the following criteria:

1. Population: adults (≥18 years) diagnosed with any chronic health condition, as defined by the ICD-10-CM [31] as a condition that lasts greater than 12 months and results in the need for ongoing medical intervention and limits self-care, independent living and social interaction. Studies including a mixed sample of acute and chronic populations were included if at least 80% of the sample had a chronic diagnosis.
2. Construct: the evaluated measure was a performance-based measure of physical function, as defined by the World Health Organisation (WHO) [32] International Classification of Functioning, Disability and Health (ICF) framework as activities which relate to the ability to move around and perform daily activities e.g. strength, balance, etc.
3. Measurement instrument: an established performance-based measure of physical function, commonly used in clinical practice, which was evaluated synchronously by a tester as the activity was being performed by the individual. This usually involved evaluation by timing, counting or distance methods [33].
4. Setting: the evaluated measure was administered by a tester located remotely from the patient using any telehealth platform, as defined by WHO as “the delivery of health care services, where patients and providers are separated by distance. Telehealth uses information and communication technologies (ICT) for the exchange of information for the diagnosis and treatment of diseases and injuries.”.
5. Measurement properties: In our pre-registered protocol, we highlighted studies must have reported one or more of the psychometric measurement properties from the COSMIN taxonomy [30]. For studies examining the validity of the measurement instrument administered via telehealth, the comparator was a face-to-face administration of the same measurement instrument. Since the comparator was always face-to-face administration of the same measure, when extracting data from the selected studies the measurement properties of interest were reliability, measurement error and criterion validity. Therefore, the remaining measurement properties outline in the COSMIN taxonomy including other forms of validity and interpretability were not considered to be outcomes of interest in this review.

Studies were excluded if (1) the evaluated measure was a self-reported measure of physical function, or a laboratory value (e.g., VO₂ max, spirometry, etc.) indirectly used to assess physical function, or a self-administered measure that did not involve administration and evaluation by an independent tester; or (2) the study population consisted of post-operative patients since post-operative pain and disability levels differ in magnitude and stability from chronic conditions.

A sample of 30% of abstracts from the database search were initially screened by two independent reviewers (CBW & RC) to determine potential eligibility. As good agreement (>80%) was achieved, the remaining abstracts were screened by one reviewer (CBW). Thereafter, a sample of 30% of full texts of potentially eligible studies were reviewed to determine eligibility by two independent reviewers (CBW & RC). Any disagreements were resolved through discussion with a third reviewer (KOS). As above, good agreement was achieved, and the remaining studies were reviewed by one reviewer (CBW).

Data extraction

Data were extracted by two independent authors (CBW & KOS) using a table created by the authors following COSMIN guidance [34]. Firstly, the characteristics of the included studies and the performance-based measures evaluated within the studies were extracted. Thereafter, data relating to the evaluation of the methodological quality of the included studies and the evaluation of the measurement properties (i.e. strength of correlations/associations) were also extracted. The included performance-based measures were categorised according to the domain of physical function that they measured. These domains included exercise capacity, functional strength, functional balance and general functional capacity.

Methodological quality of included studies

The methodological quality of each of the included studies was evaluated by two independent authors (CBW & KOS) using the COSMIN risk of bias checklist and scores were determined by consensus [34]. This tool contains separate standards for each measurement property (i.e., reliability, measurement error and criterion validity) that can be used to determine the trustworthiness of the result. Each of the standards were rated and the ‘worst-score-counts’ method was applied to determine the overall quality of each measurement property reported in the included studies [34].

Evaluation of the measurement properties reported in the included studies

The COSMIN methodology was used to evaluate the measurement property results reported in each of the included studies [34]. These results were evaluated according to the criteria for good measurement properties (strength of correlations/associations with the reference standard face-to-face administration of the measure) to give a rating of sufficient (+), indeterminate (?) or insufficient (-) for each measurement property, as described by Prinsen et al. [34] (See S1 Table).

For the reliability domain, inter-rater and intra-rater reliability were evaluated by comparing the scores for the measure when administered via telehealth between different raters and also when administered by the same rater at two different time points. As recommended, a threshold of 0.70 on the intraclass correlation (ICC) or weighted kappa was used to evaluate the reliability of the measure administered via telehealth [34]. If the correlation was ≥ 0.70 the reliability received a sufficient rating. If the ICC or weighted kappa was not reported it received an indeterminate rating for reliability. The reliability of the measure was rated as insufficient if the ICC or weighted kappa score was < 0.70.

For the criterion validity domain, the measure administered via telehealth was compared to the same measure administered in a face-to-face environment. A correlation of 0.70 with the reference standard [34], which for the purpose of this review was the measure administered in a face-to-face environment, was the threshold. The validity of the measure was rated as sufficient if the correlation was ≥ 0.70. The validity was rated as indeterminate if correlations were not reported. The validity was rated as insufficient if the correlation with the reference standard was < 0.70.

Data synthesis and analysis

To synthesise the results, the evidence was summarised per measurement property (e.g., reliability, validity) per outcome measure to come to an overall conclusion regarding the reliability and validity of the measures. If multiple studies examined the same measure, the results of the studies were synthesised to achieve an overall result. In the case of inconsistency in the results between studies (e.g., both sufficient and insufficient results were found), explanations for the inconsistency were explored. When inconsistent results likely existed due to varying study quality as previously described, the results of lower quality studies were omitted and only the higher quality results were used to determine the overall rating and the quality of summarised evidence was downgraded due to inconsistency. If no logical explanation was found which could explain the inconsistency, the results were considered inconsistent. The modified GRADE approach was used by two independent reviewers and disagreements were resolved by consensus to summarise how confident we can be that the summarised evidence is trustworthy [35]. The summarised evidence was graded as high, moderate, low or very low based on the following four criteria: 1. Risk of Bias (quality of the studies); 2. Inconsistency (of the results of the studies); 3. Imprecision (total sample size of the studies) and 4. Indirectness (evidence from different populations than the population of interest). Detailed instructions on the use of the modified GRADE approach to grade the quality of the summarised evidence can be found in S2 Table. The starting point assumed that the summarised result was of high quality and was downgraded by one, two or three levels depending on the risk of bias. The summarised result was further downgraded depending on the inconsistency, imprecision and indirectness associated with the summarised result as appropriate.

When inconsistency existed between the results of the included studies examining the same measurement instrument, the results were summarised as sufficient or insufficient and the quality of the evidence was downgraded for inconsistency with one or two levels depending on the severity of the inconsistency. As the severity of inconsistency between results is context dependent, the level of severity was discussed and decided by the review team in each situation.

For imprecision, the evidence was downgraded one level if the sample size was 50–100 individuals. If the sample size was less than 50 individuals the evidence was downgraded two levels.

As this review included studies in which the >80% of the population had a chronic diagnosis, the risk of bias associate with indirectness did not exist and therefore the evidence was not downgraded for indirectness.

Study characteristics

A summary of the descriptive characteristics of each included study and the included performance-based measures is presented in Table 1. Four of the included studies were conducted in Australia [36–39], with the remaining study conducted in the United States [40]. A total of 77 individuals were included in the review with sample sizes ranging from 10–26 participants. Two studies included patients with Parkinson’s Disease [38, 39] and the remaining three studies included patient cohorts with stroke, cystic fibrosis and heart failure respectively [36, 37, 40].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of included studies.

https://doi.org/10.1371/journal.pone.0274349.t001

Measurement properties of 15 performance-based physical function measures were investigated in the included studies. Inter-rater and intra-rater reliability were reported for 12 of the measures, while the criterion validity of all 15 measures was reported. No studies reported data on measurement error or responsiveness.

Of the 15 performance-based measures, seven assessed balance (Timed Up and Go Test, functional and lateral reach tests, Berg Balance Scale, step test, steps in 360 degree turn, and timed stance test) [36, 39, 40], two assessed exercise capacity (3 minute step test and 6 minute walk test) [36, 37] and two assessed functional strength (grip and pinch strength) [36, 38]. The remaining four measures assessed diverse aspects of functional capacity including the Functional Independence Measure (FIM) [38], of which only the motor components were assessed (bathing, dressing, toileting, walking, stairs, eating, grooming, bladder management, toilet transfers, bowel management bed/chair transfers, tub/shower transfers), the Unified Parkinson’s Disease Rating Scale (UPDRS) [38], of which relevant items were assessed (posture, gait, sensory complaints, falling, freezing when walking, tremor, tremor at rest, salivation, facial expression, bradykinesia, speech, action or postural hand tremor, handwriting), the Nine Hole Peg Test (38) and the European Stroke Scale [40].

Methodological quality of included studies

The COSMIN risk of bias scores for the measurement properties of the measures in each included study are displayed in Table 2. Of the studies that reported the reliability of the included measures, ten measures demonstrated adequate quality while three demonstrated inadequate quality. Of the studies that reported criterion validity of the measures, eight measures demonstrated very good quality and ten demonstrated inadequate quality. Many of the studies reporting the criterion validity of the included measures received an inadequate quality rating as per COSMIN guidance as the correlation with the reference standard was not calculated [41] (e.g. mean differences between measures administered via telehealth compared to face to face administration were reported as opposed to correlations).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Measurement properties of performance-based measures.

https://doi.org/10.1371/journal.pone.0274349.t002

Overall rating and quality of evidence

A summary of the overall rating and quality of evidence per measurement property of the included measures is presented in Table 3. These scores were developed from the information displayed in Table 2 which included the rating and the COSMIN risk of bias score. Twelve measures received ‘sufficient’ overall ratings for reliability, with a ‘very low’ quality of evidence score. Eight measures received ‘sufficient’ overall ratings and seven received ‘indeterminate’ ratings for criterion validity and were all scored as ‘low’ or ‘very low’ quality of evidence. For example, the Six Minute Walk Test (6MWT) demonstrated sufficient reliability (ICC>0.70) with a ‘very low’ quality of evidence score due to the inadequate COSMIN risk of bias rating of the included study and the small sample size (n<50). The 6MWT also demonstrated ‘sufficient’ validity (correlation with face-to-face>0.70) with a ‘low’ quality of evidence score due to the ‘very good’ COSMIN risk of bias rating of the included study and the low sample size (n<50). The 3-minute step test demonstrated ‘indeterminate’ validity as no correlation with the reference standard (face-to-face administration) was reported, which was insufficient information reported to provide a ‘sufficient’ rating according to the COSMIN guidance. The summary scores for the validity of the Timed Up and Go Test (TUGT) and grip strength reflect adjustments that were made to allow for inconsistencies in the results reported in the included studies. For example, the validity of the TUGT was reported in two studies and received an overall ‘sufficient’ validity rating (correlation>0.70) [36], with ‘very low’ quality of evidence due to the inadequate COSMIN risk of bias score of the included studies, the small sample size (<50), and the inconsistency between the validity findings of the included studies [36–39].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of findings.

https://doi.org/10.1371/journal.pone.0274349.t003

Exercise capacity measures

Measures of exercise capacity included in this review were the 6MWT and three minute step test. The 6MWT demonstrated sufficient reliability and criterion validity when administered via telehealth. Evidence for the administration of three minute step test via telehealth is yet to be determined as there was no information available examining its reliability, and the evidence for criterion validity was indeterminate due to non-optimal analysis. Therefore recommendation for the use of this instrument via telehealth cannot be made. However, the mean differences between the telehealth assessment and the face-to-face assessment observed by Cox et al. [37] were very small and suggest that there was no significant difference between the telehealth assessment and face-to-face administration. Therefore these three minute step test results are encouraging.

Functional strength measures

The grip strength test demonstrated sufficient reliability and criterion validity. Evidence for the pinch strength measure administered via telehealth is yet to be determined due to the lack of information available.

Functional capacity measures

Other measures included in the review which measured various aspects of general functional capacity included the European Stroke Scale, Unified Parkinson’s Disease Rating Scale, Functional Independence Measure and Nine Hole Peg Test. The most robust results were reported for the Functional Independence Measure and the Unified Parkinson’s Disease Rating Scale which both demonstrated sufficient reliability and criterion validity. However, as the Unified Parkinson’s Disease Rating Scale is a population-specific measure, the Functional Independence Measure may be more appropriate for various chronic populations. While the Nine Hole Peg Test demonstrated sufficient reliability, the criterion validity was indeterminate due to non-optimal analysis. However, the mean differences of 0.25 seconds (right hand) and 0.14 seconds (left hand) observed between telehealth administration and face-to-face administration of the measure were both within the limits of agreement of -2.02 to 1.52 (right hand) and -1.34 to 1.05 (left hand), which is encouraging. Evidence for the reliability and criterion validity of the European Stroke Scale administered via telehealth is yet to be determined due to the lack of information available and non-optimal analysis.

Strengths and limitations

Strengths of this review include the prospective protocol registration, following PRISMA guidance, as well as using two reviewers for screening, shortlisting and data extraction. A particular strength is using the COSMIN approach, which had not been used in previous psychometric evaluations for performance measures via telehealth.

There were also some limitations to be acknowledged. As previously stated, two independent reviewers screened a sample of 30% of abstracts and relevant full texts to determine eligibility. As good agreement was achieved on the 30% sample, the remaining screening process was not performed in duplicate. Although the quality of the summarised evidence was rated using the modified GRADE approach by two independent reviewers, neither of these reviewers were formally trained in the use of this method. Due to the heterogeneous nature of the included measures and the populations of the included studies, a meta-analysis could not be performed and the results could not be quantitatively summarised. As the majority of the results could not be combined, best evidence synthesis was mostly obtained from a single study. Further evidence may have been identified from studies of post-operative populations, such as individuals post total knee arthroplasty [53]. However, these were excluded as pain and disability levels immediately post-operatively, and how much these change relatively quickly, are quite different from other chronic conditions where physical function may be more stable over time. There was limited information reported regarding the characteristics of the samples in the included studies in relation to aspects such as socioeconomic status, cognitive status and technological literacy. Also, the included studies were all carried out in countries with ‘very high’ Human Development Index scores [54]. These factors could potentially impact the external validity of the findings. Although Cox et al. [37] reported the usability of the three minute step test administered via telehealth and Hwang et al. [36] reported some information regarding the number and nature of technical issues encountered during telehealth administration of the 6MWT, TUGT and grip and pinch strength, there was limited information reported in the included studies regarding the interpretability and feasibility of the included measures when administered via telehealth. As noted in the eligibility criteria, this review was concerned with examining the validity of measures administered via telehealth when compared to face-to-face administration of the same measure. Therefore, other types of validity, such as content and construct validity, were not reported in this review. Due to the eligibility criteria and aims of this review, the outcomes of interest were reliability, measurement error and criterion validity. For this reason, we followed COSMIN recommendations for evaluating reliability, measurement error and criterion validity but a priori did not choose to evaluate other types of validity, internal structure, interpretability and feasibility.

Clinical implications

Encouragingly, several performance-based measures of physical function across different domains (e.g. exercise capacity, strength and balance) may have satisfactory reliability and criterion validity when used in a telehealth environment. Furthermore, the psychometric properties of these measures appears similar to that reported for the same measures when used in a face-face context. This should reassure clinicians that using performance-based measures of physical function via telehealth is possible. However, this evidence is of low-very low quality and there is a significant lack of information regarding the measurement error and responsiveness of these measures. Furthermore, information regarding the interpretability and feasibility of the included measures was very limited [36, 37].

Future research

This systematic review highlights the need for further larger, high quality research, in line with COSMIN guidance, exploring the psychometric properties of performance-based measures of physical function administered via telehealth among people with various chronic conditions. In particular more studies examining the measurement error, responsiveness, interpretability and feasibility of these instruments are required. Although some of the measures included in this review demonstrated sufficient reliability and validity, none of the measures were evaluated with respect to all measurement properties and therefore strong recommendations cannot yet be made. Additionally, the lack of studies exploring the administration of performance-based measures of physical function via telehealth among chronic musculoskeletal populations is acknowledged. Therefore this review highlights the need for future studies to be conducted in this population.