Reader Comments
Post a new comment on this article
Post Your Discussion Comment
Please follow our guidelines for comments and review our competing interests policy. Comments that do not conform to our guidelines will be promptly removed and the user account disabled. The following must be avoided:
- Remarks that could be interpreted as allegations of misconduct
- Unsupported assertions or statements
- Inflammatory or insulting language
Thank You!
Thank you for taking the time to flag this posting; we review flagged postings on a regular basis.
closeThese data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other
Posted by glantz on 28 Jun 2013 at 01:08 GMT
This study involved randomizing people who said they were using conventional cigarettes and not interested in quitting to three groups: one receiving e-cigarettes with higher nicotine (Group A), one receiving e-cigarettes with a lower level of nicotine (Group B), and a third group receiving zero nicotine e-cigarettes (Group C, which the paper considered the “control” group). Based on comparing quit rates of conventional cigarettes one year later, the authors concluded that “in smokers not intending to quit, the use of e-cigarettes, with or without nicotine, decreased consumption and elicited enduring tobacco abstinence.” This conclusion is not supported by the data in the paper for two reasons.
First, and most important, despite the fact that the title describes the paper as a “randomized control design,” bold italicthere is not a control group of people who were not using e-cigarettes that would allow assessment of spontaneous quit rates. bold italic By not having a true control group that would account for spontaneous quitting without using e-cigarettes one cannot say anything about whether e-cigarettes affected quitting.
This is a very important point because, as noted in my textbook Primer of Biostatistics (7ed, McGraw-Hill, 2012, p. 250), “To reach meaningful conclusions about the efficacy of some treatment, one must compare the results obtained in the individuals who receive the treatment with an appropriate control group that is identical to the treatment group in all respects except the treatment. Clinical studies often fail to include adequate controls. This omission generally biases the study in favor of the treatment.”
Second, ""bold italic"" there are issues with the statistical analysis which, when corrected, eliminate the reported statistically significant results. ""bold italic""
The authors state that “At week 52 quitters were 22/200 (11.0%) in Groups A-B [the two groups of nicotine e-cigarette users combined] and 4/100 (4.0%) in Group C [the zero nicotine e-cigarette users] (p = 0.04), which is the basis for the “enduring abstinence” conclusion. The authors based this conclusion on the fact that a chi-square test of a 2 x 2 contingency table (smoking or not smoking vs nicotine or non-nicotine e-cigarettes) reached statistical significance (p = 0.04, which is less than 0.05, the cutoff for conventional statistical significance). The problem is that the authors failed to include the required Yates correction* in their calculation of the chi-square test statistic and associated p value. Recalculating the test properly yields p = 0.07, which is no longer statistically significant. Thus, the correct conclusion is that there is no statistically significant difference between the nicotine and non-nicotine e-cigarettes.
Probably a more appropriate comparison – which follows the experimental design – would be to treat the data as a 2 x 3 contingency table (smoking or not smoking vs the three different kinds of e-cigarettes). The chi-square analysis of the 2 x 3 contingency table yields p = 0.08, which is even further from statistical significance.
Thus, combining the fact that there is not a non-e-cigarette control group and correcting the statistics means that the appropriate conclusion to draw about quitting smoking based on these data is that the level of nicotine in the e-cigarette (including zero nicotine) has no detectable effect on quitting smoking conventional cigarettes.
""bold italic""These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other. ""bold italic""
*NOTE ON THE YATES CORRECTION (from italicPrimer of Biostatisticsitalic, page 84):
“… when analyzing 2 × 2 contingency tables, the value of chi-square computed [from the data] and the theoretical chi-square distribution leads to P values that are smaller than they ought to be. Thus, the results are biased toward concluding that the treatment had an effect when the evidence does not support such a conclusion. The mathematical
reason for this problem has to do with the fact that the theoretical chi-square distribution is continuous whereas the set of all possible values that the chi-square test statistic can take on is not. To obtain values of the test statistic that are more compatible with the critical values computed from the theoretical chi-square distribution [for a 2 x 2 contingency table], apply the Yates correction (or continuity correction) to compute a corrected chi-square test statistic …. This correction slightly reduces the value of chi-square associated with the contingency table and compensates for the mathematical problem just described.”
ECLAT study has a control group, Yates correction would be completely inappropriate
DrFarsalinos replied to glantz on 28 Jun 2013 at 09:47 GMT
The study evaluated the efficiency of electronic cigarette use in a randomized, double blind clinical trial. Two nicotine concentrations were used, and a control group which received non-nicotine electronic cigarette was also included. The control group is not in reality a placebo group, because they received a product which mimics the act of smoking and provides visual and sensory stimulation; thus, it handles several aspects of the addiction to smoking, but does not provide nicotine. Including such a control group leads to underestimation of the true effects on the treatment group. Moreover, no psychological support was provided, which is a key issue and may explain much (if not all) of the efficacy of other smoking-cessation methods like NRTs [1].
It is extremely important that the authors used intention-to-treat analysis. The failure rate was significant, something that is common in studies of smoking cessation. The fact that all participants were not willing to quit further raises the expected lost-to-follow up rate, as is the use of an old and inefficient electronic cigarette device (the study started in 2009, technology has significantly evolved since then). Treating all these participants as failures in the final analysis was the correct thing to do, but is also another reason for underestimating the true effects of electronic cigarettes.
Yates correction for continuity [1] is a method to correct for continuity in chi-squared statistics. However, it tends to overcorrect the p-value. Moreover, it is chiefly used when at least one cell of the table has an expected count smaller than 5, and this is not the case in this study. It has also been suggested that this correction is unnecessary even when the sample size is low [2]. In fact, it has been supported that Yates correction “produces a test that is so conservative as to be almost useless” [3], and several other studies have found such a correction inappropriate [4,5].
Considering the use of an old and inefficient electronic cigarette device, the inherent lack of placebo available for electronic cigarette studies and the inclusion of unmotivated smokers (also supported by their overall good health status, which would further decrease their incentive to quit), it is extremely difficult to discard the results of this study.
1. Moore D, Aveyard P, Connock M, Wang D, Fry-Smith A, Barton P. Effectiveness and safety of nicotine replacement therapy assisted reduction to stop smoking: systematic review and meta-analysis. BMJ. 2009;338:b1024.
2. Yates, F. Contingency table involving small numbers and the χ2 test. Journal of the Royal Statistical Society 1934;1: 217–235.
3. Sokal RR, Rohlf FJ. Biometry: The Principles and Practice of Statistics in Biological Research. Oxford: W.H. Freeman 1981, ISBN 0-7167-1254-7.
4. Grizzle JE. Continuity correction in the χ2 test for 2 × 2 tables. The American Statistician 1967;21:28–32.
5. Maxwell EA. Analysis of contingency tables and further reasons for not using Yates correction in 2 × 2 tables. Canadian Journal of Statistics 1976; 4:277–290.
6. Conover WJ. Some reasons for not using the Yates continuity correction on 2×2 contingency tables. Journal of the American Statistical Association 1974;69:374–376.
RE: ECLAT study has a control group, Yates correction would be completely inappropriate
MattJHodgkinson replied to DrFarsalinos on 28 Jun 2013 at 10:40 GMT
The commenter, Dr. Konstantinos Farsalinos, is a Subject Matter Expert for the trade association American E-liquid Manufacturers’ Standards Association (AEMSA): http://www.aemsa.org/aems.... Dr Farsalinos has received research funding from the Association of Hellenic Electronic Cigarette Traders, ΣΕΕΗΤ/SEEHT: http://www.seeht.org/
RE: RE: ECLAT study has a control group, Yates correction would be completely inappropriate
DrFarsalinos replied to MattJHodgkinson on 28 Jun 2013 at 11:13 GMT
AEMSA is an OFFICIALY non-profit (501(c) declared by IRS) organisation, developed and ran by CONSUMERS with no financial relations to the e-cigarette industry.
I have been a subject matter expert on a VOLUNTEER and NON PAID basis.
I have not received any financial compensation by the greek or any other trade organisation and i have not published any research funded by SEEHT. Moreover, for any research published, i have not received any financial compensation as clearly stated in every research. The same applies for every research not published yet.
Is this considered COMPETING interest by the journal? I doubt it....
RE: RE: RE: ECLAT study has a control group, Yates correction would be completely inappropriate
MattJHodgkinson replied to DrFarsalinos on 28 Jun 2013 at 12:03 GMT
Please note the section on "Non-financial competing interests" in the PLOS competing interests policy: http://www.plosone.org/st...
Funding from SEEHT was acknowledged here: https://plus.google.com/1...
RE: RE: RE: RE: ECLAT study has a control group, Yates correction would be completely inappropriate
DrFarsalinos replied to MattJHodgkinson on 28 Jun 2013 at 13:10 GMT
Unfortunately the discussion is not about the study itself but about funding.
However, i must protect my personal integrity.
I hope the AEMSA issue has been resolved, but i have asked the AEMSA president to take part to this discussion.
The google post says: "I should thank the Greek Association of Electronic Cigarette Businesses (SEEHT) for funding all the expenses of this study, and risking their money without having any control on the experiment and the results."
As i said in my previous post: "...i have not published any research funded by SEEHT."
My google post is about a study that has not been published yet. Moreover, i have not received any grand from them (either restricted on unrestricted), they paid themselves DIRECTLY to the laboratory the expenses for some tests. I did not raise the money and i did not pay the laboratory myself. I have asked them to send me the receipts they got from the laboratory, in which my name is NOT mentioned. And i have NOT received any financial compensation from them for any study.Therefore, i have not received any funds myself, either for the expenses of the studies or for personal compensation.
Even in this study: http://informahealthcare.... i never received, touched or paid any money for the expenses. It was a transaction between the company and the laboratory which did some tests.
I read the journal's section on competing interests. And i find that i either did not violate them or the PLoS One associate editor is using double standards. For example, why is research funded by pharmaceutical industry not considered a competing interest when pharmaceutical industry is the main competitor of electronic cigarettes? Please see funding of this study (2013, Pfizer foundation) http://tobaccocontrol.bmj...
What about the millions of dollars at stake? http://www.bmj.com/rapid-...
http://www.plosmedicine.o...
And you are searching for a google+ post about a study with expenses being less than 10,000 euros, which in fact i did not even receive or pay? This is interesting....
You are transforming a comment about a study into a personal fight between researchers and between the journal editor and researchers. And this does not serve either the study or the consumers.
Instead of continuing, i an declaring my true "competing interests" and that should put everything in place.
Being an electronic cigarette user myself and, thus, wanting to know about its true effects on my health.
RE: RE: ECLAT study has a control group, Yates correction would be completely inappropriate
LouR replied to MattJHodgkinson on 30 Jun 2013 at 03:32 GMT
The American E-liquid Manufacturing Standards Association (AEMSA) is incorporated as a non-profit Professional Trade Association. AEMSA also has 501(c)(6) status: irs.gov 501(c)(6) Organizations
AEMSA was Co-Founded, and formation facilitated, by two consumer volunteer advocates with no financial involvement in the industry. Both co-founders continue to serve on the AEMSA Board of Directors in Consumer Advocate (CA) capacity roles. I am one of the consumer volunteer AEMSA Co-Founders and currently serve as President of the Association. The position of AEMSA President is a non-paid/volunteer position elected by the Association’s General Members voting category.
AEMSA SME memberships (case in point Dr. Konstantinos Farsalinos), like Consumer Advocate (CA) memberships, are voluntary positions receiving no compensation or remunerations.
Pursuant to the AEMSA By-Laws, SME members act in an advisory capacity. Our SMEs may serve on, and even Chair, Committees (e.g. Applications, Standards or Compliance Committees) and/or the Board of Directors. Committee and/or Board positions are also volunteer positions receiving no compensation or remuneration.
While SME members serving on Committees or the Board of Directors can and do vote within those roles, SME members do not vote on any General Membership voting matters and/or agenda items (e.g. new Standards, Standards amendments and/or By-Laws amendments). Voting participation for SME members, pursuant to AEMSA By Laws, is quoted below:
Article III – Voting, subsection 3.03 – Subject Matter Experts:
(a) May not cast a vote on General Membership matters
(b) SMEs serving on Boards and Committees (including Board of Directors) may cast one vote
on Board/Committee issues
AEMSA is an all-volunteer Association and currently has zero payrolls. If the Association’s Board of Directors officially requests a SME to travel to an AEMSA meeting, venue, presentation or other to speak, testify and/or advocate with or for AEMSA, said SME may submit for travel expenses reimbursement based on per-diem rates as listed on the gsa.gov website: GSA Per Diem
Given all of the above, we see no reason why any SME involvement in AEMSA might be construed to be any sort of conflict of interest to any other professional, research and/or advocacy positions of any nature.
AEMSA’s mission (paraphrased) is establishing/refining/maintaining high Standards for consumer protection(s), product stewardship, maintaining an example/model for the industry, supporting and encouraging ongoing medical and scientific research and advocacy. The AEMSA Standards are the first real step towards defining cGMPs for commercial manufacture and distribution of e-liquids and advocating for “Reasonable, Realistic and Sustainable” regulations.
Dr Farsalinos should be commended for his forward thinking and helping AEMSA in its goals of developing Standards for e-liquid production that further responsible Product Stewardship and Consumer Protection.
Lou Ritter - AEMSA President
The commentor is mixing up two issues
glantz replied to DrFarsalinos on 29 Jun 2013 at 01:11 GMT
As I pointed out in my comment, the Yates correction is used to compensate for the fact that the actual distribution of the chi-square test statistic differs from the theoretical chi-square distribution in a way that leads to p values that are too small (as happened in the ECLAT study). The Yates correction adjusts for this mismatch and gies much more accurate p values.
It has nothing to do with small numbers of expected values in the individual cells. That is a different problem (which is dealt with by using a Fisher Exact Test when the numbers are small), which is not an issue in this study.
RE: Yates correction is not recommended for ECLAT
DrFarsalinos replied to glantz on 29 Jun 2013 at 13:55 GMT
Yates correction by definition underestimates the statistical significance when the P value is close to 0.05. So, when you already know what it does, the only reason to use it is to intentionally prove that a result is not significant. It is not a correction because it always goes one way, it makes the result less significant than it really is.
I have provided several references stating that Yates correction is considered inappropriate (or useless by some). I invite everyone to read the above mentioned (and several other) references in the literature.
If someone intends to make a result statistically insignificant, there are better ways. Make a multivariate analysis and use some covariates like age, gender, or nicotine dependence scale scores, and it will almost certainly become insignificant. But this is then becoming a statistics game, intentionally trying to make the result insignificant. Moreover, in ECLAT study the comparison is between two active-treatment groups. Electronic cigarette use should be considered active treatment even when zero-nicotine liquid is used. Still, the result was statistically significant.
RE: These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other
MattJHodgkinson replied to glantz on 28 Jun 2013 at 12:25 GMT
The commenter, Stanton Glantz, is the American Legacy Foundation Distinguished Professor of Tobacco Control at UCSF, and a tobacco control activist, including being a founder of Americans for Nonsmokers' Rights: http://profiles.ucsf.edu/...
RE: RE: These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other
DrFarsalinos replied to MattJHodgkinson on 28 Jun 2013 at 13:21 GMT
So, this means that his comment cannot be challenged?
My comment is supported by referenced research. Should i try to find who has funded the authors of all these comments?
Please, do not forget, we are commenting on a clinical study. You are misleading the discussion to an issue which has nothing to do with the study. Perhaps you do it unwittingly, but you are doing it.... If someone does not like the results of this study, it does not mean that it must be discredited...
Being an electronic cigarette user myself and, thus, wanting to know about its true effects on my health.
RE: RE: RE: These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other
ojkershaw replied to DrFarsalinos on 28 Jun 2013 at 13:41 GMT
I believe Mr Hodgkinson belatedly was flagging up Mr Glantz's competing interests, something he failed to do originally when he reported what he believed were your own competing interests.
I suggest that his list of Professor Glantz's own interests are incomplete, and were he to apply the same vigor to researching him, he might come back with many more examples of competing interests than a context-free Google+ comment.
RE: These data cannot be used to support any statement about the efficacy of e-cigarettes for stimulating smoking cessation one way or the other
mbsiegel replied to glantz on 28 Jun 2013 at 22:33 GMT
In his critique of this study, Dr. Glantz makes two main arguments:
1. "There is not a control group of people who were not using e-cigarettes that would allow assessment of spontaneous quit rates. By not having a true control group that would account for spontaneous quitting without using e-cigarettes one cannot say anything about whether e-cigarettes affected quitting."
2. "The [second] problem is that the authors failed to include the required Yates correction in their calculation of the chi-square test statistic and associated p value. Recalculating the test properly yields p = 0.07, which is no longer statistically significant. Thus, the correct conclusion is that there is no statistically significant difference between the nicotine and non-nicotine e-cigarettes."
Let's take each of these arguments in turn.
First, it is true that the study did not include any control group. However, that doesn't necessarily mean that we have no idea what the quit rate would have been for smokers with no interest in quitting in the absence of electronic cigarettes. We can simply examine the ample scientific evidence reporting one-year quit rates among smokers in the absence of intervention.
However, what we really want is the one-year quit rate among smokers with no interest in quitting. To be conservative, we can use the one-year quit rate among smokers who do want to quit (and try to quit), which is about 3%.
Clearly, the quit rate observed in the high-nicotine electronic cigarette group (13%) is considerably larger than the 3% observed for the population of smokers who want to quit and make quit attempts. The natural quit rate among smokers with no desire to quit would be substantially lower than 3%.
In fact, the paper provides population-based data showing that in Italy, the spontaneous quit rate during the study period was a dismal 0.02%.
So this first argument does not invalidate the study's conclusion that electronic cigarettes did help achieve smoking cessation for some smokers. There is no scenario under which you would observe 13% of non-motivated smokers quitting within one year without any intervention.
It is also important to point out that the comparison group Dr. Glantz requests is not possible. One could not ethically conduct a clinical trial of smoking cessation in which one group is assigned to receive no intervention.
What is really needed is a head-to-head comparison of electronic cigarettes c ompared to nicotine replacement therapy (NRT).
Dr. Glantz's second argument is that the study conclusions are invalid because of failure to apply the Yates correction. This argument is invalid, however, because it is a diversion from the actual research question. It is a diversion because there is no placebo in this trial. Had this study compared electronic cigarettes with a placebo, then the appropriate analysis would indeed be to statistically compare the cessation rates in the e-cigarette vs. placebo groups. But there was no placebo. The group receiving electronic cigarettes without nicotine was in fact an intervention arm. Even without nicotine, electronic cigarettes have been shown to reduce the craving to smoke. And in fact, in this trial, the most remarkable finding was that in the zero nicotine e-cigarette group, 14% either quit or cut down by more than half. This is remarkable for a group of smokers who had no intention to quit and who received no non-tobacco cigarette nicotine for an entire year.
The paper actually acknowledges the lack of statistical significance in quit rates between the three study arms at one year. But this doesn't mean the treatment wasn't effective. It reflects the small sample size of the study along with the fact that there was some effect among the 0 nicotine e-cigarette group.
Dr. Glantz is absolutely right that the correct comparison to make to evaluate the results of the study is how the observed quit rate (about 9% for all three groups combined) compares to the spontaneous one-year quit rate among the population of smokers, except that one would have to restrict that to the population of smokers who have no interest in quitting.
If Dr. Glantz can provide data demonstrating that more than 2% of smokers with no interest in quitting achieve smoking cessation over a one-year period, then his argument about the lack of validity of this study would be reasonable. But short of such data, it cannot be denied that the results achieved in this study are better than what one would have obtained with no intervention.
Michael Siegel, MD, MPH
Professor
Department of Community Health Sciences
Boston University School of Public Health