Introduction

The amount and the intensity of physical activity (PA), or the physical activity energy expenditure (PAEE) as its product, are closely related to cardio-metabolic health. PA is an important outcome, and the promotion of PA plays a major role in cardiac disease prevention and therapy. Due to the complex nature of PA, its accurate measurement is challenging [1]. Ideally, in cardiovascular disease prevention, therapy and rehabilitation, an objective monitoring of PA would allow for accurately differentiating a wide range of PA intensities, identify behavioural patterns and their determinants for personal advice, individualize activity dose and training load, as well as support and mediate self-efficacy [2]. Depending on the design and characteristics, the accuracy and usability of the various options available to capture PA in controlled or free-living environments may differ [3].

The doubly labelled water (DLW) method and indirect calorimetry (IC) provide the gold standard to estimate energy expenditure (EE). Wearable activity trackers may be more cost-effective, and offer better usability [4], while being able to also capture low intensity activities, which are particularly important for older adults with reduced fitness. The ActiGraph accelerometer (ActiGraph LLC, Fort Walton Beach, FL, USA) and the Actiheart combined accelerometer and heart rate monitor (CamNtech Ltd, Cambridge, UK) are most commonly used in PA research [1, 5–7]. Among the consumer-level devices, hip-worn pedometers and smartwatches have been used by individuals to increase PA in several studies, for example following cardiac rehabilitation [8].

The psychometric properties of activity trackers have been tested in evaluation studies to varying degrees. The ActiGraph accelerometer has been validated for older adults in several studies against DLW or IC [9, 10], unlike the Actiheart branched equation model to measure PAEE. There is generally a lack of validation studies for older adults, and especially older women [11, 12].

At individual level, activity trackers may produce substantial measurement errors in PAEE measurement [1]. PAEE prediction equations derived from younger study populations may not be valid for older adults, persons with reduced physical fitness, or cardiac patients [3, 6, 11]. Their PA behaviour is inherently different concerning overall PA level, type of activities, movement characteristics, movement patterns, maximal exercise performance, and resulting PAEE among other factors [3]. Because of the lack of specific validation studies, activity trackers should be used with great caution in these groups [13, 14].

Consumer-level activity trackers are considered even less accurate than the devices used in research [15]. Their algorithms are proprietary and raw data cannot be accessed [6].

Due to the wide range of activity trackers available, it is difficult to understand and overview their advantages and disadvantages [6] when attempting to answer a specific research question relating to a specific population. Limited research has been done to directly compare the validity of these devices in older adults or in a population with cardiovascular risk factors. Overall, there is an urgent need for better information on the validity of different types of activity trackers in populations of older women with an increased cardiac risk or cardiac disease.

Methods

Synchronous registrations of PAEE with up to four different measurement devices were performed to determine criterion validity against IC while participants performed a protocol of simulated activities in a laboratory setting.

Participants were recruited between 12 November 2019 and 04 February 2020 at physiotherapy outpatient practices and venues of different community activities, such as choirs, church communities, and activity groups, in the Ruhr area located in the west of Germany, by direct communication or by means of flyers.

Relevant inclusion and exclusion criteria were screened by telephone interview. We included adults over 60 years of age with cardiovascular risk, such as high blood pressure, high blood cholesterol, or high body mass living independently at home. Exclusion criteria were, among other, no medical clearance to participating in the study in case of unclear health status or existing indications for relative contraindication, serious illness, and severe physical immobility (see S1 Table in S1 File for the full inclusion and exclusion criteria). Coronary risk was assessed using the European cardiovascular disease risk assessment model (Systematic Coronary Risk Evaluation, or SCORE) [16]. Initially, we had planned to recruit n = 45 participants at different cardiac disease stages ranging from light or moderate cardiac risk to manifest cardiac disease. However, due to the COVID-19 pandemic situation, the inclusion of participants with cardiac disease had to be cancelled.

To analyse criterion validity during simulated activities, PAEE as simultaneously measured by activity trackers (ActiGraph wGT3X-BT and Actiheart 4) was compared to the PAEE measured by indirect calorimetry (MetaMax^® 3B). In a subsample of only women, PAEE was additionally registered with two commercially available activity trackers (Fitbit Charge 3 and OMRON Walking Style IV Step Counter). Before entering into the protocol, participants were asked to fast for at least 3 h and to refrain from exercise on this particular day.

The activity protocol consisted of simulated activities of daily living (part 1) and a cycle ergometer test (part 2), performed in a laboratory of the University of Applied Sciences Bochum (see S2 Table in S1 File for the full activity protocol). Part 1 consisted of resting (11 minutes), walking (6 minutes), stair climbing (2 minutes), and household activities (7.5 minutes). Part 2 started with 25 W on the cycle ergometer, increasing 25 W every 2 minutes, and was stopped at exhaustion or at Borg 15 (from 20) on the Borg rating of perceived exertion scale for participants with cardiac risk. When at least half of the final stage of the ergometer test was accomplished, data measured during the stage were included in the analysis.

All activity trackers were initialized simultaneously using individually measured characteristics before each trial, including weight, height, age, gender, and stride length if it was required. Following the protocol, EE estimates were obtained and calculated from the devices themselves using the associated software.

Indirect calorimetry (Cortex MetaMax^® 3B System and MetaSoft^® Studio Software, Cortex Biophysics GmbH, Leipzig, Germany) was used as the gold standard. Before each measurement, the MetaMax 3B was calibrated according to the manufacturer’s guidelines (Calibration Manual, CORTEX Biophysik GmbH, Leipzig, Germany). The VO₂ and the VCO₂ data were collected with a soft mask over the nose and mouth (breath by breath), and the heart rate was measured (POLAR belt around the chest) to determine the individual oxygen uptake rate and EE. An evaluation of the validity and reliability is available elsewhere [17].

The mean PAEE in kcal/min was calculated from the calorimetry-measured VO₂ and VCO₂ for each individual for the periods of resting (11 minutes), walking and climbing stairs (8 minutes), simulated activities of daily living (7.5 minutes), and the ergometer test. The resting metabolic rate was predicted using the Harrington equation [18, 19]:

Female population:

Male population:

Weir’s equation [20] was used to calculate total energy expenditure (TEE) as follows:

The PAEE resulted from the subtraction of the REE from the TEE.

Participants wore the three-plane ActiGraph wGT3X-BT (ActiGraph LLC, FL, USA) on a belt on the right hip. Data were saved in 10-second epochs (ActiLife software V6.13.3, ActiGraph LLC, FL, USA). We deliberately chose the Freedson equation [12] to report PAEE in kcal/hour because it has been widely used in research for estimating energy expenditure or categorise physical activity at different intensity levels [10].

The chest-placed Actiheart 4 (CamNtech, Cambridge, GB) uses vertical plane accelerometry (activity as counts per minute, or cpm) and heart rate data from a one-channel electrocardiogram electrode (heart rate above the resting heart rate, or HRaR). Based on 15-second epochs, PAEE was calculated by branched equations [11, 21] included in the Actiheart software (Version 4.0, CamNtech, Cambridge, GB). The following equation for older adults was entered for the group calibrated model [21]:

(sex: female = 0)

Resting heart rate was obtained before starting the protocol in a seated position, as the Actiheart device could only be initialized in this position. Sleeping heart rate was estimated as resting heart rate minus 10 heartbeats. For measurement and data analysis, published standard procedures were used [11].

The Walking Style IV Step Counter (OMRON Healthcare UK LTD, Milton Keynes, UK) is a commercially available lightweight pedometer with a 3D sensor. It measures steps, calories, and the distance walked per day. The pedometer was initially set to weight, height and stride length and then placed in the right front pocket of the trousers, as recommended in the manual. Action mode was started and stopped manually immediately before and after the measurement to report steps and calories (PAEE) for the total activity protocol.

The Fitbit Charge 3 (Fitbit Inc., San Francisco, CA, USA) is a smartphone-based wireless fitness monitor worn on the right wrist. It contains a triaxial accelerometer (micro electro mechanical system), an altimeter measuring altitude differences and an optical heart rate tracker (Fitbit User Manual Version 3.4). Prior to the measurement, a user profile was created for the participants in the app. This profile included age, weight and height. To record activity, an activity log was activated on the device. Due to the indoor measurement, no global positioning system (GPS) data were used. After taking the measurement, it was synchronized to the Fitbit app (Version 3.12, Fitbit Inc., San Francisco, CA, USA). As Fitbit reports TEE, PAEE was calculated by subtracting the REE from the TEE. Data is only available for the total activity protocol, the activities are not analysed individually.

Participants’ health, PA status, and PA-related health competence were assessed with a screening questionnaire (Physical Activity Readiness Questionnaire) [22], the Global Physical Activity Questionnaire (GPAQ) [23] and the Physical Activity-Related Health Competence questionnaire [24].

Informed written consent was obtained from the participants before study initiation. The study complies with the ethical principles for medical research outlined in the World Medical Association’s Declaration of Helsinki and was approved by the ethical committee of the German Association for Physiotherapy (ZVK) e.V., Wremen, Germany (Number 2019–06).

All devices were synchronized to the same date and time before initializing. In order to compare the PAEE of the different devices, time filters were set in the software or later in the calculation in Excel for the whole protocol or individual activities. We did not read Fitbit and Omron results for every activity, only EE for all activities.

Guidelines for missing data (“lost data”) were set in cases where there was missing data for more than one continuous minute. In that case, the time period was excluded for data calculation, and the mean score was calculated from all other periods with valid data. If the data loss was less than one minute, the data were linearly interpolated. A heart rate of <30 beats/min was considered missing data.

Tracker-derived PAEE was compared against PAEE calculated from indirect calorimetry descriptively and by Bland–Altman plots [25]. Comparisons between tracker-derived PAEE and the criterion measure were performed using one-way ANOVA’s. If differences were found, paired samples t-tests were used to identify the location of the differences. The level of significance for these analyses was set at p < 0.05.

Results

In total, 34 older adults (25 women, 9 men) with at least a light cardiac risk were included in the study. Their risk of cardiovascular death (SCORE¹⁶) within the next 10 years was 2.6% ± 1.1%, with familial predisposition being the most prevalent risk factor (see Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Participant characteristics ^a,b.

https://doi.org/10.1371/journal.pone.0309481.t001

The ActiGraph data for n = 2 participants and Actiheart data for n = 1 participant were missing due to human error during the initialization process. In addition, the Actiheart data could not be analysed due to signal loss for n = 1 for the complete protocol and for the ergometer test for n = 1. For n = 6, data loss occurred for more than one minute. This period was set as invalid, as described above, and mean was calculated from the remaining data. Out of n = 34, the data of n = 21 was valid for the full protocol, without any lost data. Heart rate data were lost during walking activities (n = 4), household activities (n = 2), or during the ergometer test (n = 1).

One participant did not complete all the tasks of the activity protocol (ergometer data are missing) but fulfilled all other activities, so his/her data are included (see S1 Fig in S1 File for Participant flow chart).

For the total protocol, both the ActiGraph (-64% ± 25%) and the Actiheart (-43% ± 26%) underestimated PAEE in the full sample (n = 32). In a subsample of only women (n = 17), the Fitbit Charge 3 (+78% ± 78%) overestimated and the OMRON Pedometer (-55% ± 47%) underestimated PAEE.

The ActiGraph showed the best estimates for walking activities at different speeds and the biggest difference to IC measured PAEE for household activities and the ergometer protocol. The Actiheart PAEE estimates were closest to IC PAEE for resting activities (-25% ± 14% difference) and walking stairs (-25% ± 37% difference) (see Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Physical activity-induced energy expenditure (PAEE) as predicted by branched equation models [11] (group calibration, Actiheart), Freedson algorithms [12] (ActiGraph) compared to PAEE measured by indirect calorimetry during an activity protocol (resting, walking, household activities and cycle ergometer protocol) in older adults with increased cardiac risk (n = 34; n = 25 women).

For resting activities, total energy expenditure (TEE) was reported instead of PAEE.

https://doi.org/10.1371/journal.pone.0309481.t002

A one-way ANOVA was performed on n = 15 (participants who provided data for all four devices) and showed significant differences between the PAEE mean values (F (4, 60) = 44.7, p < .05). In addition, the estimated PAEE of all four activity trackers was significantly different from the criterion measure (p < .05). The PAEE estimates of all devices were significantly different from each other (p < .05), with the exception of the PAEE estimated by the Fitbit compared to the PAEE estimated by the ActiGraph (p = .79) and the Actiheart (p = .12).

For the total study population, Bland–Altman plots showed limits of agreement spanning from +0.05 to +2.82 kcal/min for PAEE estimates from the ActiGraph and -0.31 to +2.32 kcal/min for the Actiheart (see S3 File). In a subsample of only women, Bland–Altman plots showed limits of agreement for PAEE estimates of –3.72 to +0.96 kcal/min and -0.52 to +2.88 kcal/min for the commercially available activity trackers Fitbit Charge 3 and OMRON, respectively (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Bland-Altman plots of PAEE estimated by the four activity trackers and by indirect calorimetry.

Bland–Altman plots assessing agreement between estimated physical activity induced energy expenditure (PAEE) measured with research and commercially available accelerometers and measured by indirect calorimetry (IC) during an activity protocol (resting, walking, household activities and ergometer protocol) of n = 23 older women with increased cardiac risk: (a) ActiGraph and IC PAEE (kcal/min), b) Actiheart and IC PAEE (kcal/min) and n = 17 older women with increased cardiac risk: c) Fitbit and IC PAEE (kcal/min), and d) OMRON and IC PAEE (kcal/min). Dots represent data from participants where PAEE was measured with all four activity trackers (n = 15), rhombus represent data from participants where PAEE was estimated with two or three activity trackers. Numbers in the Bland–Altman plots are mean difference and upper and lower limits of agreement (95% CI).

https://doi.org/10.1371/journal.pone.0309481.g001

Discussion

Precisely measuring PAEE or EE is essential to assess PA behaviour and to understand the dose–response relationship between PA and CVD. The current study is the first to validate different activity trackers (trackers used in research or commercially available activity trackers) against indirect calorimetry in adults over 60 years, with a special focus on older women with cardiovascular risk performing various activities.

The comparison of EE estimates in the current study demonstrated that the research-grade activity trackers appear to have advantages over the consumer-level activity trackers, even though the estimation of PAEE was not consistently good across the devices, as various activities were measured with varying validity.

Both the research-based activity trackers underestimated PAEE for the total activity protocol ranging from sedentary to moderately vigorous activities, in line with previous research [26]. Our Actiheart estimates of PAEE for the total study population indicate that the simultaneous use of HR and accelerometry using a previously established branched equation model (group-calibrated model) results in a smaller mean prediction error when compared to accelerometer only. However, in the female part of the population, the random error is larger than in the ActiGraph estimates. Thus, the Actiheart does not necessarily provide a more accurate estimate of EE than accelerometry only. Previous studies have reported that the ActiGraph and the Actiheart generally allow a valid estimate of PAEE in several populations [9, 26, 27], while they both tend to underestimate the PAEE induced by most activities.

Considering the activities individually, PAEE of walking activities was generally estimated most accurately, with an especially small mean prediction error by the ActiGraph in women. For PAEE of climbing stairs, there was no difference between ActiGraph and Actiheart estimation accuracy. The ActiGraph seriously underestimated PAEE for household activities and cycling on a stationary bike; the Actiheart performed better for these activities.

Our results indicate that when compared to low-intensity PA, vigorous PA can be captured more accurately by the combined analysis of HR and acceleration data (Actiheart), while the ActiGraph is less accurate at high intensity. This has frequently been reported previously (e.g. Crouter et al. [28]). Previous results from Aguilar-Farias et al. [10] indicate that when estimated by the ActiGraph using the Freedson equation, PAEE tended to be underestimated by about 1.5 MET during higher-intensity activities in older adults. However, generally, most of the activities which older adults undertake are of low intensity.

According to the Bland–Altman plots, too, the scattering of the disagreement between the ActiGraph or the Actiheart and IC seems to be related to PA intensity. For the estimation of PAEE during household activities, heterogeneous results are reported in the literature. While Aguilar-Farias et al. [10] stated that the ActiGraph tended to overestimate EE for lower intensity activities by about 1 MET in older adults, Imboden et al. [14] observed a large underestimation of PAEE during household activities (-26% bias to IC). In line with our results, it can be assumed that for individuals spending more time in sedentary behaviour, light physical activity or walking activities during the day, the ActiGraph will likely yield better measurement accuracy than for individuals spending more time in activities such as cycling or household activities [14].

Despite the Actiheart device showing a more accurate PAEE estimate than the ActiGraph, the signal loss of the heart rate is a serious problem that severely affects the reliability of the PAEE estimate. Thus, adding heart rate may indeed reduce the potential error associated with using a single equation for the acceleration vs. EE relationship, which is derived from just one activity. However, this advantage comes with the caveat that data do not seem to be collected in a reliable fashion, as observed in previous studies [29]. These results may inform researchers who have to select a suitable activity tracker for a specific research question and study design. Researches might prefer to capture a wide variety of activities relatively precisely but at a high risk of signal loss, or alternatively accept a higher underestimation of PAEE while ensuring that the signal is reliably captured.

The accuracy of PAEE estimates by the ActiGraph accelerometer differs depending on the actual activity observed. Physical activity energy expenditure during walking was captured relatively well, in line with previous studies (r = .82) [27]. By contrast, PAEE during the cycle ergometer test and household activities was largely underestimated. In the target population of older adults, walking is a preferred and highly prevalent type of physical activity [30]. Still, older adults, and women more than men, also devote a great deal of time to household activities [31]. The underestimation of PAEE in these activities can be explained for the ActiGraph and the OMRON by the waist placement of the devices. The waist receives little or no vertical acceleration during upper-body movements and physical activities, which do not involve a transfer of the centre of mass.

The extreme interindividual variation of differences between IC- and ActiGraph-PAEE during the cycle ergometer test indicates that in some persons activity is captured relatively well, while in others no acceleration at all is captured. It is well known that accelerometer-based PAEE estimation with standard equations do not produce useful results for cycling activities in older adults (e.g. Kossi et al. [32]: measurement bias 0.81 ≤ ES ≤ 1.90). In fact, the vertical axis counts in our study were surprisingly high, but the additional vector magnitude analysis showed slightly better results.

Overall, our results of the ActiGraph underestimating PAEE can also be found for free-living situations. In adults between 43 and 83 years, PAEE estimated by accelerometer was 50–73% lower than PAEE determined by the DLW method [9].

Of the commercially available activity trackers, the Fitbit Charge 3 overestimated PAEE and show very wide limits of agreement. The Bland-Altman plot may indicate that overestimation of PAEE increases with increasing PA intensity. These findings indicate that, without the GPS, the Fitbit Charge 3 does not produce acceptable estimates, especially at higher self-selected intensities. However, the findings contradict previous findings that the Fitbit Charge devices underestimate PAEE in different age groups [33]. The OMRON underestimated PAEE and showed less wide limits of agreement than the Fitbit, albeit still larger than the research-grade devices. Similarly, it has been reported that the OMRON pedometer underestimated PAEE with limited accuracy [34]. Unlike in our study, in that research the magnitude of underestimation depended on the walking speed. Generally, our results are in good agreement with the review of Fuller et al. [15] who reported that EE estimates of consumer-level activity trackers varied widely, with less than 10% of estimates falling within acceptable limits in controlled settings.

Our sample is as old as, or slightly younger than the population in published ActiGraph validation studies, which however focus on other aspects, such as cut point or equation validation [10, 35]. Their aerobic fitness seems relatively good compared to norm values for 60–69-year-old adults [36], reflecting the problem, that due to the pandemic situation, we had to skip the second part of our study and did not include participants with cardiac disease.

A potential limitation of our study is that the activities were simulated in a laboratory instead of performed in a real-world setting. Participants were encouraged to perform activities as they usually do, but the setting, duration of the activity, measurements (especially the facemask), and the presence of the investigator might have influenced their performance. It could be criticised, that the fact that different activities are grouped into one activity category in our activity protocol means that activity energy is not consistently recorded. In our study, we aimed to estimate the PAEE during short periods of activity followed by breaks, as usually found in analyses of physical activity in the age group and close to daily life (e.g. walking around the house, walking and waiting at traffic lights) [37, 38].

Another limitation concerns the selection of the formulae and algorithms to calculate PAEE, even though we chose the most current default formula (group calibration) [11], cut-points and algorithm [12] for each device to report PAEE as provided in the literature. There are studies evaluating different cut-off values and algorithms for older populations [10, 35], but within the scope of our project we decided to validate the most common ones only. Further studies will focus on the evaluation of other algorithms and cut off-values as, for example metabolic costs increase with age, but our results at least partly suggest that the problem of inaccurate PAEE estimates may only be solved by more advanced technology and intelligent pattern-detecting algorithms.

A strength of this study is its focus on women with a light to moderate cardiovascular risk, an understudied but important group, and directly comparing various devices for a variety of activities of daily living typical of the age group over 60 years against indirect calorimetry. While it may be sufficient for many clinical studies to analyse the criterion validity of steps or time spent in moderate-to-vigorous PA, this study offers a significant advantage in analysing the criterion validity of PAEE.

Conclusions

The accurate assessment of PAEE in older adults with increased cardiac risk is important in various research and clinical contexts, and is especially understudied in women. In this study, we have directly compared the validity of four different activity trackers against IC.

Overall, research-grade activity trackers provide better validity than consumer-level activity trackers, but seriously underestimate PAEE during cycling and household activities (ActiGraph), or do not reliably capture all signals (Actiheart). Clinicians and researchers should be aware of the potential limitations of these devices.

Supporting information

Acknowledgments

The authors thank all participants of the study for their generous participation.