Reader Comments

Post a new comment on this article

Effect of a ketogenic diet on energy expenditure: What conclusions can be drawn from a problematic pilot study?

Posted by mfriedman820 on 09 Dec 2019 at 19:58 GMT

Hall et al. 2016 in context

Although not identified as such in either Hall et al. papers [1, 2] or in Hall’s Formal Comment, the original Hall et al. study [1] was a pilot study. As described in the study’s protocol [3], it was intended to provide information about the magnitude and variability of various outcomes and to harmonize methods and procedures in preparation for a future larger scale, multisite study. Not surprisingly, given its status as a pilot study, Hall et al. [1] tested a relatively small number of participants (N = 17; 4-5 at each of four sites). In addition, the study had two serious limitations.

First, as a one-way crossover study, it was not randomized and had no control for the order in which subjects consumed the basal and ketogenic diets. This precludes causal inference about the effect of diet on the various outcomes and restricts comparisons to before and after the diet switch. This limitation applies not only to interpretation of the original results, but also to any secondary analysis or reanalysis, including that by Hall et al. [2], Friedman & Appel [4], Rosenbaum et al. [5], and Hall’s Formal Comment. Therefore, Hall et al. [1] and its follow-up analyses do not allow a definitive conclusion about the effect of a ketogenic diet on energy expenditure or any other outcome.

Second, according to the study protocol, participants were to be maintained in energy balance during the basal diet (BD) period by matching energy intake to expenditure and then fixing energy intake at that level after the switch to the ketogenic diet (KD). However, despite this intention to maintain energy balance, participants lost weight throughout the study because the procedure for establishing energy balance was flawed. Energy intake was matched (“titrated”) to expenditure as measured in metabolic chambers for two days each week during the baseline period. As a result, subjects were underfed as much as ~500 kcal/d relative to expenditures for five days each week when they were housed on the ward where their physical activity and energy expenditure were approximately 20% greater [1, 4]. Consequently, participants lost 3.7 +/- 1.7 kg (mean + SD; p < .000001 by paired t-test) in body weight over the course of the 8 week study. As we discussed [4], this chronic energy deficit would be expected to suppress energy expenditure, which could have restrained or counteracted any increase in expenditure measured after the diet switch. In addition, the continuing energy deficit, combined with the lack of a control for diet order, further complicates analysis and interpretation of the study results.

In his Formal Comment, Hall states that, “The subjects consumed … slightly more energy than they expended during the respiratory chamber days and they completed 90 minutes of mandatory daily [cycling] exercise … in an attempt to stabilize energy expenditure from physical activity between chamber and non-chamber days.” Although this may imply that exercise in the chambers was used to equate physical activity between chamber and non-chamber days, subjects were in fact prescribed the same amount of cycling for both chamber and non-chamber days throughout the study. Consequently, the exercise, being simply additive, did nothing to equate or “stabilize” physical activity expenditure between chamber and non-chamber days and therefore could not have prevented underfeeding on non-chamber days relative to chamber days.

Energy expenditure by doubly labeled water - adjustments in RQ

Hall’s Formal Comment centers in large part on the claim [2] that energy expenditure measured using doubly labeled water (DLW) in the original Hall et al. study [1] needs to be recalculated to adjust RQ in relation to deviations from energy balance and differences in diet. Again, the failure to maintain energy balance and the resulting cumulative loss of body weight and body fat that started during the BD period together with the lack of a control for diet order complicates interpretation of these adjusted expenditure values.

Hall et al.’s [2] adjustments in RQ, and the resulting changes in reported energy expenditure measured using DLW, are based on theoretical, not empirical, considerations despite Hall’s objection to our characterization of the adjustments as “hypothetical” (or “theoretical” elsewhere in our paper). Elia [6], on whose work the Hall et al. [2] recalculations are based, makes clear that the adjustments in RQ are theoretical in nature and this is echoed in the Hall et al. [2] reanalysis (see caption to Figure 2 in the Online Supporting Material). In fact, the adjustment in RQ must be theoretical since there are no RQ data for non-chamber days. Consequently, it is impossible to validate the adjustments in RQ and expenditures by comparison of actual and theoretical values, which further indicates that the changes in effect size associated with the calculated adjustment in RQ claimed by Hall et al. [2] and in Hall’s Formal Comment must be considered conjectural.

Rosenbaum et al. [7] found a close correspondence between changes in total daily energy expenditure in response to over- and under-feeding whether measured using either DLW or caloric titration. With caloric titration, expenditure is determined simply by the number of fed calories required to maintain a stable body weight and therefore can be used to validate measurements of energy expenditure using DLW. In this regard, Ebbeling et al. [8] recently reported results from a secondary analysis of a large randomized control trial showing that daily energy expenditure measured using caloric titration was greater in free-living, weight-reduced and weight-stable participants eating a low-carbohydrate diet than it was in those eating a high-carbohydrate diet. The magnitude of the difference in expenditure between these two diet groups was similar to that observed in the same subjects based on DLW measurements [9, 10]. In addition to providing further evidence that low-carbohydrate diets increase energy expenditure in free-living subjects, these findings reinforce Friedman & Appel’s [4] suggestion that DLW and caloric titration could be profitably used together in future studies of the effect of ketogenic diets on energy expenditure.

Identification and selection of outliers

In response to our critical evaluation [4] of Hall et al.’s [1,2] criteria for identification and selection of outliers, Hall redefines for a second time the parameters used to identify outliers. Because no criteria for identification and handling of outliers were pre-specified in the study protocol [3], such post hoc definition and redefinition gives pause, especially considering the small number of subjects and how large an effect exclusion of data from putative outliers had on the apparent magnitude of the change in DLW expenditure after the diet switch.

As described in his Formal Comment, Hall now suggests a best-fit analysis to determine body weight change over the DLW measurement period to identify outliers. Hall helpfully provides supporting information that includes individual data used in this new analysis of weight change and the resulting modeled changes in body energy stores. However, it is difficult to gauge the value of this approach without knowing the individual probability and r2 statistics to determine whether the individual regressions to establish best-fit functions for each subject are statistically significant and of meaningful predictive power.

The two participants whose data were excluded from the analysis of DLW energy expenditure in the Hall et al. studies [1,2] were considered outliers because, according to Hall, their data were incompatible with the law of energy conservation. As defined operationally, this meant that changes in body weight for both subjects during the DLW measurement period were incommensurate with the differences in their energy intake and energy expenditure. However, as we discussed in Friedman & Appel [4], five other subjects could be considered outliers by the same definition. In three of the five cases, the discrepancies between energy intakes and expenditures were substantial on an absolute basis (+465, -291 and -250 kcal/d as expenditure minus intake), especially if considered over the long-term.

Hall dismisses the idea that these five subjects are also outliers because their discrepancies between energy intake and expenditure, as a group, were small on a relative (percentage) basis compared with those of the two outliers Hall et al. [1,2] excluded from analysis. As Hall reports in his Formal Comment, adjusted non-chamber DLW energy expenditure increased by 185 +/- 508 kcal/d (p = .15) after the diet switch for all 17 subjects, but increased by only 59 +/- 354 kcal/d (mean + SD; p = .53) when data from the two outliers were excluded from the analysis. However, despite their relatively small differences between energy intake and expenditure, additionally excluding data from the five other outliers, resulted in an increase in adjusted non-chamber expenditure of 193 +/- 330 kcal/d (p = .10) in the 10 remaining subjects (based on data from Hall’s newly available dataset). This result – a complete reversal of the effect of excluding only the two outliers from analysis – highlights again how much the apparent effect size of switching the diet depends on the choice of outliers and how important it is to pre-specify criteria and procedures for identifying, selecting and handling outliers [4].

Testable mechanism or untestable method?

Hall finds the mechanisms that we propose to account for the increase in expenditure after the switch to the KD as “somewhat mysterious.” He takes issue with our suggestion that a decrease in muscle work efficiency may be involved because Hall et al. [1] found no difference between diet periods in chamber energy expenditure during cycling. In fact, we acknowledged [4] that this finding argues against our hypothesis, but did not consider it definitive because other work suggests a decrease in muscle efficiency with a very low-carbohydrate diet and because muscle work efficiency associated with other forms of physical activity besides cycling might have been affected.

We also hypothesized that greater physical activity on non-chamber days may have increased expenditure during the KD period by creating a demand for glucose that, under the condition of severe dietary carbohydrate restriction, would have to be met through the energetically expensive process of hepatic gluconeogenesis. As we discussed [4], this hypothesis is based on (i) data showing that increased energy expenditure after insulin withdrawal is associated with hyperglucagonemia and increased protein catabolism, both of which were also observed during the KD period in the original Hall et al. study; and (ii) recent estimates of the energetic cost of gluconeogenesis associated with consumption of a ketogenic diet very similar to that used by Hall et al. [1]. To counter our gluconeogenesis hypothesis, Hall offers up a study by Veldhorst et al. [11], which estimates the energy cost of gluconeogenesis under experimental conditions that differ comprehensively and radically from those employed in the original Hall et al. study and therefore appears to have little or no relevance for our hypothesis.

What is important, however, is that the mechanism of increased gluconeogenesis we suggest is eminently testable. In short, the “mystery” is solvable. In contrast, Hall’s methodological explanation for the increase in DLW energy expenditure, which is based on theoretical considerations and cannot be validated or, in other words, tested, seems likely to retain an air of mystery of its own.


The original Hall et al. study [1] was successful when considered as the pilot study it was intended to be. In keeping with the typical purposes of a pilot study, it, together with subsequent analyses of its data [2,4], highlighted problems and issues with design, methods and procedures that would need to be resolved in a rigorous full-scale trial. However, although the Hall et al. study [1] fulfilled its purpose as a pilot, it cannot be considered a hypothesis testing trial given its serious limitations, particularly with respect to its non-randomized design. While follow-up analyses such as those by Hall et al. [2], Friedman & Appel [4] and Hall in his Formal Comment may highlight issues of methodology to be addressed and mechanisms to be tested in future studies, we cannot reanalyze, recalculate, model or redefine our way to a definitive conclusion about the effect of a ketogenic diet on energy expenditure. We will need to experiment our way to it. Therefore, the only reasonable conclusion to draw from the original Hall et al. study is that another, better study needs to be done.


1. Hall KD, Chen KY, Guo J, Lam YY, Leibel RL, Mayer LES, et al. Energy expenditure and body composition changes after an isocaloric ketogenic diet in overweight and obese men. Am J Clin Nutr 2016;104: 324–33.

2. Hall KD, Guo J, Chen KY, Leibel RL, Reitman ML, Rosenbaum M, et al. Methodologic considerations for measuring energy expenditure differences between diets varying in carbohydrate using the doubly labeled water method. Am J Clin Nutr 2019;109: 1328-34.

3. Hall KD. Effect of a Eucaloric Ketogenic Diet on Energy Expenditure: A Pilot Study. NIDDK IRB Approved Protocol KEE Study Expiration Date 07-29-14.docx; 2017. Database: Open Science Framework [Internet]. Available from:

4. Friedman MI, Appel BE. Energy expenditure and body composition changes after an isocaloric ketogenic diet in overweight and obese men: a secondary analysis of energy expenditure and physical activity. PLoS ONE. 2019. e222971.

5. Rosenbaum M, Hall KD, Guo J, Ravussin E, Mayer LS, Reitman ML, et al. Glucose and lipid homeostasis and inflammation in humans following an isocaloric ketogenic diet. Obesity 2019; 27: 971-81.

6. Elia, M. Energy equivalents of CO2 and their importance in assessing energy expenditure when using tracer techniques. Am J Physiol 1991; 260: E75-78.

7. Rosenbaum M, Ravussin E, Matthews DE, Gilker C, Ferraro R, Heymsfield SB, et al. A comparative study of different means of assessing long-term energy expenditure in humans. Am J Physiol 1996; 270: R496-504.

8. Ebbeling CB, Bielak L, Lakin PR, Klein GL, Wong JMW, Luoto PK, et al. Higher energy requirement during weight-loss maintenance on a low- versus high-carbohydrate diet: secondary analyses from a randomized controlled feeding study. medRxiv [Preprint]. 2019 medRxiv 19001248 [posted 2019 July 11]: Available from: doi: 10.1101/19001248

9. Ebbeling CB, Feldman HA, Klein GL, Wong JMW, Bielak L, Steltz sk , et al. Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial. BMJ 2018;363: k4583. Available from:

10. Ludwig DS, Lakin PR, Wong WW, Ebbeling CB. Scientific discourse in the era of open science: a response to Hall et al. regarding the Carbohydrate-Insulin Model. Int J Obese 2019. Available from:

11. Veldhorst MA, Westerterp-Plantenga MS, Westerterp KR. Gluconeogenesis and energyexpenditure after a high-protein, carbohydrate-free diet. Am J Clin Nutr 2009;90: 519-26.

Competing interests declared: As described in the Friedman & Appel paper to which Dr. Hall's Formal Comment is directed, I have been and am currently employed by Nutrition Science Initiative, a 501(c)(3) medical research organization, which provided funding for the study that is the subject of the Friedman & Hall secondary analysis.