We thank the Field, Jones and Christiansen for these stimulating comments on our meta-analysis of cognitive bias modification (CBM) for substance addictions and will take the opportunity to reply. Indeed, the results of our meta-analysis might not appear surprising based on narrative reviews. However, narrative reviews are an unreliable source of evidence as they do no try to provide a systematic snapshot of a field. They also leave a lot of action room for eventual a priori biases from their authors influencing the selection and interpretation of included studies. In what follows, we will succinctly attempt to reply to the main points raised by Field and colleagues in their comment.
The first critique involves our combination of what the authors of the comment call “laboratory research” with randomized controlled trials (RCTs). This apparent distinction is based on a current misconception about what defines an RCT. RCTs are identified as such by the existence of a random allocation of participants to two or more groups, with one receiving an intervention, and the other no intervention, a dummy treatment or no treatment. This is clearly stated by all the relevant guidelines, such as the Cochrane Handbook[1] (box 6.3a), which we have in fact closely followed for our inclusion criteria, or the NICE guidelines (http://www.nice.org.uk/gl...). In our previous review of an early version of our paper at a preeminent journal in the field, we were once again confronted with and surprised by how common this misconception is. An RCT is not defined by the presence of trial registration, by the existence of blinding, the use of a clinical samples and definitely not by the authors’ own description of the trial. An RCT is certainly not defined by whether or not it had “intended” to be an RCT. This last criteria regarding “intention” is the most misleading, as it opens the door to explicit and unaccountable data selection. In fact, by suggesting that we exclude studies that by definition would be RCTs just because their authors did not “intend” them as such, Field and colleagues effectively reprimand us for not having employed data selection. The term under which this process is known in the scientific research is “cherry-picking”, the selective exclusion of studies from a meta-analysis based on non-transparent, non-reproducible criteria, a practice that has been shown to distort results[2].
The second main critique regards our choice of outcome measures. Field and colleagues criticize us for not including alcohol consumption as measured by the “taste taste”. As it is clearly stated in our paper, we did include alcohol consumption or other measures of substance consumption (like cigarette consumption) when they were measured with a standardized instrument (such as the Alcohol Approach Avoidance Questionnaire/AAAQ, in fact developed by one of the CBM developers, or The Timeline Follow-Back diary/TLFB). We did indeed not consider behavioral measures such as the taste test. This task measures in the laboratory how much the participants drink from an alcoholic (usually beer) and respectively non-alcoholic drink (usually orange juice). We were unable to find any independent validation of this measure, particularly regarding its criterion validity. Field and co-authors claim to have demonstrated a correlation between how much participants drink in the lab and how much they drink in the laboratory. The paper they cite in the comment is their own meta-analysis showing that a form of CBM has a positive impact on food or alcohol consumption measured in the laboratory. We were not able to find the result they made reference to in that cited paper. However, in another paper, unfortunately not referenced in the comment[3] Jones and colleagues did show, as claimed in the commentary, that participants’ typical alcohol consumption was a significant predictor for their performance on the taste test. However, in the comment Field et al. omitted to mentioning other relevant results from this paper. For instance, typical alcohol consumption was a significant predictor together (i.e., in the same model) with subjective craving and the perceived pleasantness of the alcoholic drink. All these three variables were related to consumption as measured by the taste test. Also, hazardous drinking as measured by the Alcohol Use Disorders Identification Test (AUDIT) was not related to consumption as measured by the taste test. Hence, this is hardly evidence of validity. We still don’t know which one of these variables is more important in accounting for the performance in the taste test. Maybe it’s subjective preference, which would not really be related to addiction (unless of course to like beer more than orange juice is a problem). In fact, what appears evident from Jones at al. [3] is that it is not clear what behavior at the taste test means and that this behavior is most likely connected to a number of factors. Importantly, social desirability, which could have been very relevant given the nature of the problem and the study population (mostly young females) was not considered. Nonetheless, there is an even more problematic aspect in considering this paper as evidence of criterion validity. The analysis presented was not carried on an independent sample, but aggregating data from separate intervention studies. Jones et al.[3] claim to have demonstrated the construct validity of the taste test because they showed it is sensitive to experimental manipulations (mainly several forms of CBM). But this argument is circular. The authors are doing what is known as a “double-dip” in their evidence. On one hand, they use these studies as separate studies proving the efficiency of CBM interventions because of a demonstrated effect on participants’ performance in the taste test (in the single papers and in their meta-analysis[4]). On the other hand, they collate a very similar pool of studies (some are identical) to show these prove the validity of the taste test as a relevant outcome measure because it was successfully impacted by CBM interventions. A proper validation analysis would require a separate, new sample. Moreover, we would add that proper establishing of criterion validity for the taste test should not simply show that it correlates with consumption, this is of course evident. If you don’t drink beer, you will not drink it in the laboratory. Rather, validation ought to show that this behavioral measure is sensitive enough to detect relevant variations in drinking behavior, i.e., if you drink more outside the laboratory, you would tend to drink more in the taste test.
In sum, we are perplexed as to why Field and colleagues find our decision to exclude ad hoc non-validated behavioral measures confusing or arbitrary. They argue this decision was not supported by any references but this because there were no references validating the taste test as a substitute measure of addiction relevant outcomes. As we explained previously, citing the authors’ own research, it is not even clear what the taste test measures. Does it measure consumption? Does it measure preference? Does it measure the fact one simply does not like orange juice very much? As we are talking about a sensitive behavior like alcohol consumption, particularly given that a significant proportion of participants were aware of the purpose of the taste test, it would also be reasonable to question how this measure fares in regards to social desirability. Nonetheless, a more serious problem for the taste test in regard to our meta-analysis was that studies used different versions and very different scoring procedures. This is not simply a trivial problem to be just noted as a limitation, it cannot be simply filed under “the measure is not perfect” as Field and colleagues seem to suggest in their comment. This is a major issue particularly germane for a meta-analysis as it would have led to very high heterogeneity around the estimations of the effect. This means that the pooled effect size could have no longer been considered a reliable estimation of the “true” effect of the intervention. It also did not help that outcomes on the taste test are given in ml, a unit that simply cannot be aggregated with measures such as psychological scales. We do not have any standardization for this measure, so as to be able to tell how many ml of beer (or orange juice) drank in the lab are a little or a lot. As importantly, any standardization should probably take into account participant’s baseline values.
Field and colleagues also criticize us for considering craving as a more important outcome and seem to imply we did not take into account the existent limitations of measures of craving. This is inaccurate. Craving was simply a construct that was assessed in more trials and as such we could aggregate results. Some of these studies used validated scales, some used visual analogue or Likert scales. As evident from Table 2, we conducted sensitivity analysis exclusively examining the validated measures of craving and results were similar. We believe this is evidence we did acknowledge the limitations of the craving measures since we thought it necessary to verify them in sensitivity analysis. Moreover, we did not use attempt to speculate on the meaning of these results for addiction research, and simply reported them because craving was a relevant outcome measured in a sufficient number of studies. Surely, Field et al. can see that there are considerable differences between interpreting a construct like urge to drink, even when measured on a Likert scale, and interpreting the quantity of beer consumed in the laboratory, or in many cases an obscure proportion between the quantity of beer consumed and that of orange juice, as done by variations of the taste test.
To conclude, we thank Field and co-authors for these constructive criticisms. We admit that there are limitations to our meta-analysis and acknowledge the merit of their assessment. In fact, we are more than willing to recognize that if synthesizing their main critiques, their claim would be that while CBM might not work for addictions, it could be effective on making an individual drink orange juice or soda rather than beer in a laboratory setting, that might indeed be true. But we are less sure this is a healthy, desirable or practically relevant result.

Ioana Cristea, Babes-Bolyai University, Cluj-Napoca, Romania

References
1. Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] [Internet]. The Cochrane Collaboration; 2011. Available: www.cochrane-handbook.org.
2. Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014; MR000035. doi:10.1002/14651858.MR000035.pub2
3. Jones A, Button E, Rose AK, Robinson E, Christiansen P, Di Lemma L, et al. The ad-libitum alcohol “taste test”: secondary analyses of potential confounds and construct validity. Psychopharmacology (Berl). 2016;233: 917–924. doi:10.1007/s00213-015-4171-z
4. Jones A, Di Lemma LCG, Robinson E, Christiansen P, Nolan S, Tudur-Smith C, et al. Inhibitory control training for appetitive behaviour change: A meta-analytic investigation of mechanisms of action and moderators of effectiveness. Appetite. 2016;97: 16–28. doi:10.1016/j.appet.2015.11.013