http://orcid.org/0000-0001-8586-1261
Figures
Abstract
Objectives
To help implement behaviour change interventions (BCIs) in practice it is important to be able to characterize their key components. This study compared broad features of cost-effective BCIs that addressed smoking, diet, physical activity, alcohol and sexual health. It also assessed the association of these with the magnitude of the cost-effectiveness estimates.
Methods
A content analysis of 79 interventions based on 338 intervention descriptions was conducted, using the Behaviour Change Wheel (BCW) to classify intervention content in terms of intervention functions, and the BCT taxonomy to identify and categorise component Behaviour Change Techniques (BCT). Regression analysis identified the association of these with upper (pessimistic) and lower (optimistic) cost-effectiveness estimates.
Results
The most and least common functions and BCT clusters were education (82.3%) and shaping knowledge (79.7%), and coercion (3.8%) and covert learning (2.5%). Smoking interventions contained the largest ( = 12) number of BCTs and were most cost-effective. Several other factors were associated with worse (coercionfunction βupper = 36551.24; shaping knowledgeBCT βlower = 2427.78; comparison of outcomesBCT βupper = 9067.32; repetition and substitutionBCT βupper = 7172.47) and better (modellingfunction βlower = -2905.3; environmental restructuringfunction βupper = -8646.28; reward and threatBCT βupper = -5577.59) cost-effectiveness (p<0.05).
Citation: Beard E, West R, Lorencatto F, Gardner B, Michie S, Owens L, et al. (2019) What do cost-effective health behaviour-change interventions contain? A comparison of six domains. PLoS ONE 14(4): e0213983. https://doi.org/10.1371/journal.pone.0213983
Editor: Fernando A. Wilson, University of Nebraska Medical Center, UNITED STATES
Received: July 4, 2018; Accepted: March 5, 2019; Published: April 17, 2019
Copyright: © 2019 Beard et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All the available data are presented in the paper. Raw data are available from https://osf.io/2kwg6/.
Funding: This study was funded by the UK’s National Institute for Health and Care Excellence (NICE).
Competing interests: EB has received unrestricted funding from Pfizer. RW has undertaken research and consultancy for companies that develop and manufacture smoking cessation medications. LS has received honoraria for talk and travel expenses from manufacturers of medications for smoking cessation to attend meetings and workshops. RW is funded by Cancer Research UK; EB is funded by the NIHR SPHR and CRUK also provide support. RW is funded by Cancer Research UK and NIHR SPHR also provide support. SPHR is a partnership between the Universities of Sheffield; Bristol; Cambridge; Exeter; UCL; The London School for Hygiene and Tropical Medicine; the LiLaC collaboration between the Universities of Liverpool and Lancaster and Fuse; The Centre for Translational Research in Public Health, a collaboration between Newcastle, Durham, Northumbria, Sunderland and Teesside Universities. The views expressed are those of the authors(s) and not necessarily those of the NHS, NIHR, or Department of Health. No funders had any involvement in the design of the study, the analysis or interpretation of the data, the writing of the report, or the decision to submit the paper for publication. LO’s salary is funded by NICE. BG and FL have nothing to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Physical inactivity, smoking, excessive alcohol consumption, unprotected sex and poor diet cost the National Health Service (NHS) in England more than £14 billion per year [1–4]. In the US, smoking alone accounts for 6–8% of personal health expenditure and obesity for 10% of all medical costs [5, 6]. Poor health behaviours also adversely affect the local economy in terms of work productivity, sick leave, and need for social-care [7–9]. Developing behaviour change interventions (BCIs) to address this has become a key objective of public health over the last few decades. BCIs typically involve coordinated sets of activities designed to change specified behaviour patterns. There is good evidence for the effectiveness of some BCIs in some contexts, including the provision of behavioural support for smoking cessation [10], brief advice in primary care for excessive alcohol consumption [11], school based programmes to raise physical activity levels [12], interactive digital interventions for sexual health promotion [13], and behavioural support to reduce calorie intake [14]. These BCIs have also been found to be cost-effective in yielding quality-adjusted life years [15–17].
To implement these interventions in practice it is important to be able to characterize their key components. Also, different types of components may be more useful for some behavioural targets than others. This paper presents a first attempt to specify intervention content and compare this across different behavioural domains using a reliable, theory-based coding system. It focuses specifically on interventions found to be cost-effective [18]. This is because of the common challenge of translating interventions that have been found to be cost-effective in Randomised Controlled Trials (RCTs) to routine practice [19]. Many factors may contribute to this problem [20], but an inadequate specification of the key components of the intervention likely plays a role. In addition, it is often not possible to replicate an intervention precisely, and so it is important to have an understanding of what may be its essential functions so that intervention adaptations can be made without losing these functions [21]. A secondary aim was to identify the association of these key components with measures of cost-effectiveness. This will give some indication as to the degree of cost-effectiveness of different features. To our knowledge this is the first paper to conduct this type of analysis.
BCIs may be characterised in terms of both ‘content’ and ‘delivery’. Content refers to what may be thought of as the active ingredients of the intervention (akin to the chemical composition of a pharmaceutical product), while delivery refers to the manner in which this is applied (akin to the dosing regimen of the pharmaceutical product). The complexity of BCIs means that it is not possible to capture every aspect of content and delivery, but it is possible to capture some key features using coding systems that can be applied with an acceptable degree of reliability [21]. In particular, BCI content can be characterised in terms of a set of ‘intervention functions’, which capture ways in which an intervention can change behaviour: education, persuasion, incentivisation, coercion, training, restriction, environmental restructuring, modelling, and enablement (Table 1).
These functions form part of a framework for developing interventions called ‘The Behaviour Change Wheel’ (BCW). The BCW is a synthesis of 19 behaviour change frameworks that draw on a wide range of disciplines and approaches and has been used in a variety of contexts [22–26]. In brief, the BCW is a behavioural system, the hub of which specifies that for behaviour change to occur one needs three conditions: capability, opportunity and motivation (COM-B). Around this hub, nine intervention functions are positioned which aim to address deficits in one or more of these conditions. These intervention functions can then be implemented in an intervention using one of 93 proposed Behaviour Change Techniques (BCTs) [21, 27]. BCTs represent observable and irreducible intervention components that serve to perform one or more of these functions that is, a technique is proposed to be an ‘active ingredient’ (e.g., feedback, self-monitoring, reinforcement) [28]. For example, motivation can be increased through the function ‘persuasion’ and the application of the BCT ‘salience of consequences’, which uses methods to emphasize the consequences of changing the behaviour. This might involve showing people hard hitting images of the consequences of smoking, such as diseased lungs. These coding frameworks have been demonstrated to be able to be used to reliably code descriptions of interventions, and they appear to cover most if not all the BCIs that have been evaluated in RCTs to date [21, 27, 29–35].
It is possible that different behaviours require different approaches, and so these essential functions may differ across behavioural domains. If there are systematic differences, this may provide clues as to the mechanisms of action that commonly need to be targeted in each case. It is possible that effective components of interventions targeting addictive behaviours such as smoking and alcohol consumption may have less of an impact on sexual health, physical activity or diet interventions [36, 37]. For example, pharmacological aids are commonly used for smoking cessation and alcohol dependency, to a lesser extent for weight loss, and infrequently for encouraging exercise [38, 39]. If on the other hand, similar features are found across behavioural domains this may support the use of multiple behaviour change interventions [40]. Identifying whether disparities exist across health behaviours can also be helpful in identifying whether implicit theoretical assumptions exist regarding causes of behaviour. Traditionally, the biomedical model has been used in the treatment of addiction, with recognition only recently of the role of psychological and social factors [37].
The current study aimed to:
- Characterise BCIs according to their intervention functions [27, 31] and BCTs taxonomy [21]. In order to help understand the findings, this study also classified the interventions according to a range of contextual factors such as setting and intensity.
- Compare the intervention functions and BCTs used to address smoking, diet, physical activity, alcohol, sexual health, and multiple health behaviours.
- Assess the association between the features and content of BCIs (in terms of functions and BCTs) with estimates of cost-effectiveness.
Methods
Stage 1: Identification and retrieval of source material
Nineteen National Institute for Health and Care Excellence (NICE) guidance documents which included economic modelling and cost-effectiveness reviews and which assessed behaviour change in at least one of six behavioural domains (smoking, diet, physical activity, alcohol, sexual health, or multiple health behaviours) were identified in November/December 2012 through a systematic search conducted by NICE (see Table 2). These reports were searched for interventions which focused on individuals aged 16 years and older and which showed evidence of being cost-effective. A total of 79 interventions were considered to be cost-effective; of which, estimates could be calculated for 72 (see Table 2 and S1 Fig).
Cost-effectiveness was determined using cost-utility analysis. This considers someone’s quality of life and the length of life they will gain as a result of an intervention. The health benefits are expressed as quality-adjusted life years (QALYs). An intervention was classified as cost-effective if its incremental cost-effectiveness ratio (ICER), which is the difference in cost between two possible interventions, divided by the difference in their effect, was below the NICE threshold of £20,000 per quality-adjusted life year (QALY) [58]. Cost-effectiveness estimates were derived directly or calculated from figures in the reports and, where necessary, converted into GBP at time of original analysis. Both lower and upper limits of estimates were recorded.
After identification of cost-effective interventions in economic reports, the sources of effectiveness estimates used in economic analyses of these interventions were identified. These could either be publications reporting primary data, or summaries located in systematic reviews/meta-analyses. A total of 115 relevant source documents were initially identified from economic analyses. In cases where insufficient detail was provided on intervention content in systematic reviews/meta-analyses, original studies were retrieved, resulting in a total of 338 papers/reviews being reviewed (66 reviews and 272 original papers; see S1 Material).
Stage 2: Characterisation of interventions
The content of interventions was characterised using two methods. The first identified their intervention functions as defined in the BCW framework [31] (see Table 1). Interventions were also coded using a taxonomy of 93 BCTs [BCT Taxonomy v1 [21]], divided into 16 clusters derived from hierarchical cluster analysis (see [21] for more details).
Following Michie et al’s [21] guidelines, BCTs were coded only where coders believed that there was unequivocal evidence of their inclusion in a given intervention. All articles were coded by EB, with a subset of articles (n = 66, 20%) coded in batches by BG, with disagreements resolved through discussion after each batch. Agreement was 97%, with a mean Cohen’s Kappa of 0.74 (95%CI 0.67 to 0.82), indicating good inter-rater reliability [59].
Interventions were also categorized in terms of a range of factors relating to their context and delivery: intervention level (e.g. individual vs. population), delivery agent type (e.g. nurses vs. physicians), and intensity (e.g. high vs. low) [60–62] (see S2 Table).
Analysis
All data were extracted into a data extraction form and then transferred into IBM SPSS v.20. Interventions were split into six categories: alcohol, diet, smoking, physical activity, sexual-health interventions, and interventions targeting multiple health behaviours. Differences according to intervention characteristics were analysed using t-tests or one-way ANOVAs and Chi-square (χ2) or Fisher Exact tests for continuous and categorical variables, respectively. The Tukey correction was applied in post hoc analyses.
The factors associated with cost-effectiveness were then assessed using generalised linear modelling specifying the Gaussian family. Only the intervention features, functions and BCT clusters were considered due to sample size. Unadjusted and stepwise adjusted models are reported for all interventions combined for which cost-effectiveness estimates were available (n = 72). Stepwise methods were used to select the most relevant variables for the adjusted analysis based on the Akaike information criterion (AIC). Associations were assessed with both lower (most optimistic) and upper (most pessimistic) limits of cost-effectiveness estimates taken directly from the reports. In cases where only a single cost-effectiveness estimate was recorded this was included as both the lower and upper limit.
Results
Broad characterization of interventions
Overall.
The majority of interventions were classified as being of high intensity, were set in primary care or the community and were delivered by health professionals (Table 3). They also mostly targeted individuals from the general population. Incentives were used in 15.2% of interventions and pharmacological support in 34.2%.
By behavioural domain.
The broad intervention features used varied according to behavioural domain. While diet interventions were mostly set in the community, interventions to improve sexual health were predominantly based in primary care. Post-hoc analysis also showed that smoking interventions were generally delivered by health professionals or physicians. Interventions for alcohol consumption and those targeting multiple health behaviours were more often population-wide than those for other target behaviours, which mostly focused on individuals or groups. Smoking cessation interventions were also more likely to involve pharmacological support than other interventions.
Intervention functions
Overall.
Overall, the most common functions, identified in over two thirds of interventions, were education (82.3%), and enablement (75.9%) (Table 4). Only 3.8% of interventions used coercion and 7.6% incentivisation.
By behavioural domain.
Intervention functions also differed significantly according to the health behaviour targeted. In contrast to other types of interventions, alcohol interventions had a weaker focus on education, enablement and training and a stronger focus on restrictions (Table 4). While alcohol and diet interventions also had less of a focus on persuasion, the use of environmental restructuring was particularly uncommon in smoking cessation and sexual health interventions. There were no differences between interventions in terms of incentivisation, coercion or modelling in post-hoc analyses.
Identification of BCT clusters
Overall.
Around three-quarters of BCIs included the BCT cluster ‘shaping knowledge’ (79.7%), which includes the BCTs reattribution, antecedents, behavioural experiments and instruction on how to perform the behaviour, and the cluster ‘goals and planning’ (73.4%, Table 5). More than 60% of the interventions also included ‘social support’ and ‘antecedents’ while only 3.8% included ‘scheduled consequences’, and 2.5% ‘covert learning’.
By behavioural domain.
The prevalence of clusters differed among the behavioural domains (Table 5). All of the diet interventions included shaping knowledge and antecedents, while fewer than 50% of alcohol interventions included these two clusters. Goals and planning and social-support were most prevalent in smoking and multiple health behaviour interventions and least prevalent in diet interventions which also did not feature comparison of outcomes, which includes the BCTs persuasive source, pros and cons and comparative imagining of future outcomes. The BCT cluster regulation was most common in smoking interventions which were also the only interventions that included scheduled consequences and covert learning, this involves the BCTs imaginary punishment and reward and vicarious consequences. All the multiple health behaviour interventions included feedback and monitoring.
Identification of individual BCTs
Overall.
The mean number of BCTs identified per BCI was 10 (range 2 to 39). Smoking cessation interventions included the largest number of BCTs on average (mean 11.8, median = 8, mode = 6, range 3 to 39), followed by interventions targeting multiple behaviours (mean = 9.1, median = 8, mode = 7, range 6 to 14),physical activity interventions (mean = 8.4, median = 7.5, mode = 4, range 4 to 14), alcohol interventions (mean = 7.7, median = 5.5, mode = 3, range 2 to 21) and interventions to improve sexual health (mean = 7.6, median = 5, mode = 5, range 2 to 15). Diet interventions included the smallest number of BCTs (mean = 4.7, median = 5, mode = 5, range 2 to 8).
A total of 45 BCTs were identified in at least three BCIs (see Fig 1). Instructions on how to perform a behaviour, social support (unspecified), information about health consequences and problem solving, which involves analyzing factors influencing the behaviour and generating strategies that include overcoming barriers and/or increasing facilitators, were included in over half of all the BCIs. A further four BCTs were coded in at least two interventions [mental rehearsal of successful performance, self-incentive, information about other’s approval, imaginary punishment] and a further nine BCTs were included in at least one intervention [addition of self-monitoring of outcome(s) of behaviour, monitoring outcome(s) of behaviour by others without feedback, remove aversive stimulus, satiation, restructuring the social environment, distraction, information about antecedents, incompatible beliefs, identity associated with changed behaviour, anticipated regret]. The other 34 BCTs were not identified in any BCI.
By behavioural domain.
A total of 55 distinct BCTs were identified in the smoking cessation interventions. The most prevalent BCT (recorded in 90% of smoking cessation interventions) was the inclusion of instructions on how to perform a behaviour, for example, advise the person how to use smoking cessation medication correctly (see S1 Table). Thirteen BCTs were found in the dietary interventions. Instructions on how to perform a behaviour was the most common BCT. Physical activity interventions included 29 BCTs. The most common BCT was discussion of body changes, which involves altering body structure, functioning or support directly to facilitate behaviour change. Thirty BCTs were identified in effective alcohol interventions, and 23 in interventions to improve sexual health. All sexual-health interventions included instructions on how to perform a behaviour. Finally, in the interventions targeting multiple behaviours, 23 BCTs were identified.
Factors associated with cost-effectiveness
In bivariate analyses, sexual health interventions were found to be less cost-effective than smoking cessation interventions (βlower = 7422.54, p<0.001 and βupper = 25190.52, p<0.001), while interventions implemented in nonstandard settings were more cost effective than those set in primary care (βupper = -8799.06, p = 0.036). Several functions and BCT clusters were also associated with lower cost-effectiveness: restriction (βlower = 3322.8, p = 0.008 and βupper = 13144.1, p = 0.004), coercion (βlower = 9947.6, p<0.001 and βupper = 45665.9, p<0.001) and identity (βlower = 2226.4, p = 0.044 and βupper = 9882.2, p = 0.014) (see Table 6).
In adjusted stepwise analyses, diet (βupper = 14683.11, p = 0.002), physical activity (βlower = 2739.99, p = 0.027), sexual health (βlower = 7012.09, p<0.001) and multiple health behaviour interventions (βupper = 15431.68, p = 0.006) were found to be less cost-effective than smoking cessation interventions. Those delivered through use of media or nonstandard means were more cost-effective than those delivered by a physician (βlower = -5562.95, p = 0.032 and βlower = -3986.23, p = 0.005). Recruitment of a general population sample was associated with higher cost-effectiveness for the lower cost-effectiveness limit and lower cost-effectiveness for the upper cost-effectiveness limit (βlower = -1855.58, p = 0.023 and βupper = 5588.12, p = 0.044), while use of incentives was associated with higher cost-effectiveness and social-marketing with lower cost-effectiveness (βlower = -3402.56, p = 0.010 and βlower = 3216.09, p = 0.045). Several intervention functions were also associated with lower (coercion βlower = 6679.31, p = 0.002 and βupper = 36551.24, p<0.001) and higher (modelling βlower = -2905.3, p = 0.008; environmental restructuring βupper = -8646.28, p = 0.025) cost-effectiveness. The BCT clusters shaping knowledge (βlower = 2427.78, p = 0.021), comparison of outcomes (βlower = -2026.43, p = 0.019 and βupper = 9067.32, p = 0.002) repetition and substitution (βupper = 7172.47, p = 0.012) were associated with lower cost-effectiveness, while reward and threat (βupper = -5577.59, p = 0.034) with higher cost-effectiveness.
Discussion
This study compared broad features of cost-effective behaviour change interventions (BCIs) that addressed smoking, diet, physical activity, alcohol and sexual health. It also assessed the association of these with cost-effectiveness estimates among cost-effective interventions.
Most interventions were high intensity, set in primary care and delivered by health-care professionals. Education and enablement were the most commonly used intervention functions while incentivisation and coercion were rarely used. There was large variation across behavioural domains. While education, enablement, persuasion and training were less prominent, restriction was more prominent for alcohol than other behavioural targets. The majority of interventions included around 10% of all the potential BCTs in the taxonomy, with the most common BCT clusters being shaping knowledge and goals and planning. Few studies adopted scheduled consequences or covert learning. There was substantial variability across behavioural domains. For example, the use of pharmacological support, persuasive source, social-support and goal–setting were most prevalent in smoking cessation interventions. Body changes featured commonly in diet and physical activity interventions, restructuring the physical environment in alcohol interventions and providing feedback on behaviour in sexual health interventions and interventions targeting multiple behaviours. The BCTs shaping knowledge, comparison of outcomes, repetition and substitution, and the intervention function coercion were associated with lower cost-effectiveness, while the BCT reward and threat and intervention functions modelling and environmental restructuring, were associated with higher cost-effectiveness estimates.
Extensive evidence exists for the effectiveness of the most prevalent intervention functions and BCT clusters. In terms of education and shaping knowledge, educational materials have been shown to increase the uptake of cervical cancer screening [63] and to change attitudes towards excessive alcohol consumption [64]. However, providing knowledge and education alone is often not sufficient for enduring behaviour change. A phenomenon known as the knowledge-behaviour gap is commonly observed whereby what we believe we should do does not also tally with what we actually do in practice [65, 66]. For this reason, further support needs to be provided in the form of enablement (e.g. behavioural support and medications which are effective tools for helping smokers to stop [67]) or in the form of implementation intentions [68], which can involve action and goal planning one of the most commonly reported BCT clusters [69].
In our study, the wide variation in use of intervention functions across behavioral domains is unsurprising. For example, alcohol control policy has historically focused on reducing availability though licensing laws, minimum pricing for alcohol and age-of-sale restrictions [70]. In contrast, tobacco control has focused on a range of measures including educational and training approaches, in addition to coercive techniques such as increased taxation. In 1998 a national network of stop smoking services was set up in England with the aim of providing every smoker in the country who wanted help with stopping with access to evidence based behavioural support. This support includes the promotion of knowledge of the harms of smoking and training in relapse prevention, on top of a prescription for a smoking cessation medication, as recommended in clinical practice guidelines [71]. Persuasion has also been highly prevalent, most notably through the application of tobacco mass media campaigns [72, 73].
In our study, a relatively small number of BCTs were coded in the cost-effective interventions examined (an average of 10 out of a possible 93). This could be because only a small number of BCTs are effective or cost-effective, or because of insufficient intervention descriptions in published/available information [74]. Alternatively, it may reflect some intervention developers’ implicit theoretical assumptions regarding causes of behaviour and how it might be changed, or providers’ norms, historical bias and/or lack of training in intervention design [27, 75, 76]. Intervention design is also often governed and influenced by political and social priorities, and the goals of the funding source, which may impact on which BCTs are chosen [77, 78]. It is of particular interest that few interventions used BCTs based on operant learning (i.e the BCT clusters covert learning and reward and threat) [79]. This includes techniques which involve manipulation of environmental contingencies such as rewarding behaviour, using prompts and cues, agreeing on a behavioural contract and encouraging practice. Operant conditioning techniques have been applied successfully [80, 81] and are argued to underpin much of human behaviour [82, 83]. It is possible that interventions are inadvertently implementing such principles, with extinction and poor knowledge of schedules of reinforcement perhaps responsible for the failure to achieve maintainable behaviour change [83]. The fact that some BCTs were particularly common to specific health behaviour interventions is largely consistent with previous studies [29, 30] and can be systematically linked to theories of human behaviour [82, 84–86]. For example, the provision of social-support and goal-setting, which featured commonly in smoking cessation interventions, form part of Goal-Setting Theory [84], Social Learning Theory [85] and the Health Belief Model[86].
In this study, environmental restructuring, modelling of behaviour and threat and reward were associated with higher cost-effectiveness. The focus of interventions on removing or adding objects to the environment has been advocated by the popular book ‘Nudge’ [87]. Although it has proven efficacy [88], concerns have been raised that a reliance on its principles eschews the use of other efficacious BCW intervention functions [27]. Demonstration of behaviour and social comparison form part of several behaviour change theories including Social Comparison Theory and Social Learning Theory [85, 89], and have been associated previously with smoking cessation success [90], perception of alcohol-related negative consequences [91] and greater weight loss [92]. Threat and reward underpins the principles of Operant conditioning, which comprises some of the most underused BCTs’. Several factors influence the effectiveness of conditioning in practice, including the form of reward or punishment. Studies suggest that the optimal presentation of rewards should follow a ‘variable ratio schedule’ rather than ‘fixed ratio’ where a response is reinforced after an unpredictable number of responses [93].
In contrast, the function ‘coercion’ and the BCTs ‘shaping knowledge’, ‘comparison of outcomes’ and ‘repetition and substitution’, although present in cost-effective interventions, were associated with lower cost-effectiveness overall. ‘Coercion’ involves raising the financial cost of a behaviour whose incidence you wish to reduce. This might be through fiscal measures such as taxation or legislation. Intervention cost may increase with the inclusion of coercive methods as they place more emphasis on external influences which require manipulation than personal agency. However, there are instances where legislative measures can have a low per person cost compared with face-to-face interventions. There is substantial evidence for coercive measures in behaviour change. For example, raising the unit price of tobacco and alcohol products generates reductions in their use and healthcare costs [94, 95]. ‘Comparison of outcomes’ involves the use of persuasive arguments, summarising the pros and cons and comparative imagining of future outcomes, while ‘shaping knowledge’ covers instructions on how to perform a behaviour and use of antecedents and reattribution techniques. Finally, ‘repetition and substitution’ involves behavioural practice, substitution, habit formation or reversal and graded tasks. There are several explanations why these may incur higher cost, including the need for delivery by a trained professional and one-to-one support.
These findings have a number of implications. First, they may aid evidence-based practice and the application of BCIs in the public domain. Although part of the failure to implement interventions in the real world results from differences in choice of control conditions and resources, the ability to duplicate the components of the original intervention may also play a key role [19, 20], and this paper provides some of the key BCTs contained in cost-effective interventions. Secondly, studying the types of components of behavioural interventions in this manner may help enable scientific replication, by clearly specifying which components have been employed previously [33]. Replication is important both for ascertaining the generalisability of interventions and for increasing confidence in conclusions regarding their efficacy [96]. Thirdly, elucidating and summarising the components of cost-effective interventions may be a valuable resource to intervention designers, with guidelines recommending a full literature review of the components of efficacious interventions before development [97]. These findings may also help to encourage clearer reporting of intervention content and reveal gaps in the literature which need addressing [98]. For example, few studies used the BCT cluster coercion, reasons for this need to be identified and addressed. This is particularly important with the growing movement towards developing machine-readable papers based on highly specified ontologies to improve evidence synthesis [99]. Finally, this paper gives some indication as to the BCTs and intervention features which may be associated with greater cost-effectiveness. Future studies should aim to assess the contribution of these either in isolation or using factorial type designs.
To our knowledge, this is the first attempt to synthesise cost-effective BCIs in terms of their functions and ‘active ingredients’. However, this study also has several limitations. First, the BCT taxonomy coding approach was applied conservatively, in that a technique was coded as present only when there was unequivocal evidence from written materials that it was used. Because many intervention reports are poorly specified there was some uncertainty about inclusion of BCTs [29]. However, this approach did lead to greater specificity in identifying BCTs. Secondly, it is not possible to make a causal attribution of cost-effectiveness to specific BCTs. Although the regression analysis can be used to help discern these effects, caution should be taken because the BCIs typically contained many BCTs and the analyses were likely underpowered to detect small associations [100]. Thirdly, this paper did not compare effective with ineffective interventions or effective interventions which were cost-effective and not cost-effective, so we do not know how far these identified BCTs are unique to the interventions included in the current paper [88]. We also cannot make conclusions as to which BCTs may be ineffective or not cost-effective, only the degree to which they are cost-effective. It will be important to ascertain in future studies the features of interventions deemed ineffective and cost-ineffective in order to draw firmer conclusions. Nonetheless, this study provides, to our knowledge, the first indication of BCTs which are commonly applied in interventions deemed to be cost-effective in the United Kingdom (UK). Fourthly, this study focused on controlled studies, mainly RCTs, and many BCIs (e.g. tax increases) are not readily evaluated using this method. Fifthly, the process for identifying studies through using the NICE economic reports may have resulted in some interventions being missed and new studies are being published every week so the picture may change over time. Further studies are needed which assess the use of BCTs in other behavioural domains (e.g. cancer screening attendance and use of illicit drugs). Sixthly, for the behavioural domains other than smoking cessation only relatively small numbers were identified. This may reflect a tendency on the part of the research community not to conduct cost-effectiveness analyses for those behaviours. Several guidelines exist for the reporting of cost-effectiveness and researchers should be encouraged to follow these, with the quality of reporting of economic evaluations varying widely [101–103]. This has several implications, including the possibility that those interventions which fail to report such analyses are more likely to contain certain intervention functions and BCTs. Seventhly, this report is limited by the quality and time-frame of the economic analyses which provided the evidence-based for interventions included in this BCT analysis. Economic modelling itself is open to a number of limitations, such as uncertainty about temporal discounting, adjustment for quality of life and the use of disparate methodologies (e.g. assumptions) across reports [104]. This paper used the NICE threshold of cost effectiveness of £20,000-£30,000 per QALY. However, there is debate about the correct level of this threshold which should be used [105] and this varies enormously between countries [106] and even within countries. Eighthly, the adjusted model for the association with cost-effectiveness was derived using stepwise regression in order to prevent over-parameterization of the model. There are several criticisms of this procedure, including the production of biased regression coefficients and artificially narrow confidence intervals [107–110]. Alternative subjective methods such as hierarchical regression could not be used as we did not have any pre-specified theoretical hypotheses as to which associations may be present. Caution should also be taken when interpreting the results from the bivariate and adjusted regression models as they did not take into account sampling error. This can be achieved with meta-regression but unfortunately the sample sizes were too small for this technique [100]. Ninthly, a number of contextual features of interventions which may be associated with cost-effectiveness were not considered as they were beyond the scope of the current review. These include geographical location and more specific socio-demographic and cultural features of the samples included [111]. This will be an important avenue for future research. Finally, due to the small sample sizes for some of the analyses it is possible that we were underpowered to detect effects and associations. It will be important to update these findings as more literature becomes available.
In conclusion, this study reliably categorized and coded the BCTs used in cost-effective BCIs. These interventions heavily relied on education and enablement and most used relatively few BCTs. However, substantial variations were found in the content of interventions targeting the six health behaviour domains of interest, with alcohol interventions focusing less on education and enablement and more on restriction, and smoking interventions featuring most BCTs. There are a number of explanations for this, including the use of common sense models of human behaviour, poor reporting and variations in underlying etiology. Several intervention functions and BCTs were associated with higher cost-effectiveness: modelling, environmental restructuring and reward and threat. These findings will be of interest to intervention developers and policy makers attempting to implement BCIs in the real world.
Supporting information
S1 Material. References for included studies.
https://doi.org/10.1371/journal.pone.0213983.s002
(DOCX)
S1 Table. Most prevalent BCTs across the various health-behaviours.
https://doi.org/10.1371/journal.pone.0213983.s003
(DOCX)
S2 Table. Broad categorisation of interventions.
https://doi.org/10.1371/journal.pone.0213983.s004
(DOCX)
References
- 1. Scarborough P, Bhatnagar P, Wickramasinghe KK, Allender S, Foster C, Rayner M. The economic burden of ill health due to diet, physical inactivity, smoking, alcohol and obesity in the UK: an update to 2006–07 NHS costs. J Public Health 2011;33(4):527–35.
- 2.
Nash R, Featherstone H. Cough up: Balancing tobacco income and costs in society 2010 [1]. http://www.policyexchange.org.uk/publications/category/item/cough-up-balancing-tobacco-income-and-costs-in-society.
- 3.
NHS National Treatment Agency for Substance Misuse. Alcohol treatment in England 2011–12 London: NHS National Treatment Agency for Substance Misuse,; 2013 [https://www.nice.org.uk/guidance/ph49/resources/costing-statement-69190813.
- 4.
Development Economics. Unprotected nation: The fiancial and economic impacts of restricted contraceptive and sexual health services 2013 [http://www.fpa.org.uk/sites/default/files/unprotected-nation-sexual-health-full-report.pdf.
- 5. Finkelstein EA, Trogdon JG, Cohen JW, Dietz W. Annual medical spending attributable to obesity: payer-and service-specific estimates. Health affairs (Project Hope). 2009;28(5):w822–31.
- 6. Warner KE, Hodgson TA, Carroll CE. Medical costs of smoking in the United States: estimates, their validity, and their implications. Tob Control. 1999;8(3):290–300. pmid:10599574
- 7. Van Duijvenbode D, Hoozemans M, Van Poppel M, Proper K. The relationship between overweight and obesity, and sick leave: a systematic review. Int J Obes. 2009;33(8):807–16.
- 8. de Vocht F, Heron J, Angus C, Brennan A, Mooney J, Lock K, et al. Measurable effects of local alcohol licensing policies on population health in England. J Epidemiol Community Health. 2016;70(3):231–7. pmid:26555369
- 9. Parrott S, Godfrey C, Raw M. Costs of employee smoking in the workplace in Scotland. Tob Control. 2000;9(2):187–92. pmid:10841855
- 10. Stead LF, Koilpillai P, Lancaster T. Additional behavioural support as an adjunct to pharmacotherapy for smoking cessation. Cochrane Database Syst Rev. 2015(10):CD009670. pmid:26457723
- 11. Kaner EF, Beyer F, Dickinson HO, Pienaar E, Campbell F, Schlesinger C, et al. Effectiveness of brief alcohol interventions in primary care populations. Cochrane Database Syst Rev. 2007(2):CD004148. pmid:17443541
- 12. Dobbins M, Husson H, DeCorby K, LaRocca RL. School-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6 to 18. Cochrane Database Syst Rev. 2013(2):Cd007651. pmid:23450577
- 13. Bailey JV, Murray E, Rait G, Mercer CH, Morris RW, Peacock R, et al. Interactive computer-based interventions for sexual health promotion. Cochrane Database Syst Rev. 2010;9(CD006483).
- 14. Shaw K, O’Rourke P, Del Mar C, Kenardy J. Psychological interventions for overweight or obesity. Cochrane Database Syst Rev. 2005(2):Cd003818. pmid:15846683
- 15.
National Institute for Health and Care Excellence. Behaviour Change: individual approaches PH49 2013 [https://www.nice.org.uk/guidance/ph49/chapter/about-this-guidance.
- 16. Gordon L, Graves N, Hawkes A, Eakin E. A review of the cost-effectiveness of face-to-face behavioural interventions for smoking, physical activity, diet and alcohol. Chronic illness. 2007;3(2):101–29. pmid:18083667
- 17. Shepherd J, Kavanagh J, Picot J, Cooper K, Harden A, Barnett-Page E, et al. The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13–19: a systematic review and economic evaluation. Health technology assessment (Winchester, England). 2010;14(7):1–206, iii–iv.
- 18. McCabe C, Claxton K, Culyer AJ. The NICE cost-effectiveness threshold. Pharmacoeconomics. 2008;26(9):733–44. pmid:18767894
- 19. Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev Sci. 2004;5(1):47–53. pmid:15058912
- 20. Clarke GN. Improving the transition from basic efficacy research to effectiveness studies: methodological issues and procedures. J Consult Clin Psychol. 1995;63(5):718. pmid:7593864
- 21. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behaviour change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behaviour change interventions. Ann Behav Med. 2013;46(1):81–95.
- 22. Tombor I, Shahab L, Brown J, Crane D, Michie S, West R. Development of SmokeFree Baby: a smoking cessation smartphone app for pregnant smokers. Translational Behavioral Medicine. 2016;6(4):533–45. pmid:27699682
- 23. Suntornsut P, Wongsuwan N, Malasit M, Kitphati R, Michie S, Peacock SJ, et al. Barriers and Recommended Interventions to Prevent Melioidosis in Northeast Thailand: A Focus Group Study Using the Behaviour Change Wheel. PLoS Negl Trop Dis. 2016;10(7):e0004823. pmid:27472421
- 24. Fulton EA, Brown KE, Kwah KL, Wild S, editors. StopApp: using the behaviour change wheel to develop an app to increase uptake and attendance at NHS Stop Smoking Services. Healthcare; 2016: Multidisciplinary Digital Publishing Institute.
- 25. Webb J, Foster J, Poulter E. Increasing the frequency of physical activity very brief advice for cancer patients. Development of an intervention using the behaviour change wheel. Public health. 2016;133:45–56. pmid:26822162
- 26. Mc Sharry J, Murphy P, Byrne M. Implementing international sexual counselling guidelines in hospital cardiac rehabilitation: development of the CHARMS intervention using the Behaviour Change Wheel. Implementation Science. 2016;11(1):134. pmid:27724957
- 27. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implementation Science. 2011;6(1):42.
- 28. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behaviour change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behaviour change interventions. Annals of behavioral medicine. 2013;46(1):81–95.
- 29. Lorencatto F, West R, Stavri Z, Michie S. How well is intervention content described in published reports of smoking cessation interventions? nicotine & tobacco research. 2013;15(7):1273–82.
- 30. Michie S, Abraham C, Whittington C, McAteer J, Gupta S. Effective techniques in healthy eating and physical activity interventions: a meta-regression. Health Psychology. 2009;28(6):690. pmid:19916637
- 31.
Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. 2014.
- 32. Michie S, Hyder N, Walia A, West R. Development of a taxonomy of behaviour change techniques used in individual behavioural support for smoking cessation. Addictive behaviors. 2011;36(4):315–9. pmid:21215528
- 33. Michie S, West R. Behaviour change theory and evidence: a presentation to Government. Health Psychology Review. 2013;7(1):1–22.
- 34. Michie S, Whittington C, Hamoudi Z, Zarnani F, Tober G, West R. Identification of behaviour change techniques to reduce excessive alcohol consumption. Addiction. 2012;107(8):1431–40. pmid:22340523
- 35. Hackshaw L, McEwen A, West R, Bauld L. Quit attempts in response to smoke-free legislation in England. Tob Control. 2010;19(2):160–4. pmid:20378592
- 36. Baban A, Craciun C. Changing health-risk behaviors: A review of theory and evidence-based interventions in health psychology. Journal of Cognitive and Behavioral Psychotherapies. 2007;7(1):45–67.
- 37.
West R, Brown J. Theory of addiction: John Wiley & Sons; 2013.
- 38. Beard E, Brown J, Michie S, Kaner E, Meier P, West R. Use of aids for smoking cessation and alcohol reduction: A population survey of adults in England. BMC Public Health. 2016;16:1237. pmid:27931202
- 39. Rueda-Clausen CF, Padwal RS. Pharmacotherapy for weight loss. BMJ 2014;348.
- 40. Sweet SN, Fortier MS. Improving Physical Activity and Dietary Behaviours with Single or Multiple Health Behaviour Interventions? A Synthesis of Meta-Analyses and Reviews. Int J Environ Res Public Health. 2010;7(4):1720–43. pmid:20617056
- 41.
National Institute for Health and Care Excellence. Brief Interventions and Referral for Smoking Cessation in Primary Care and Other Settings NICE Public Health Intervention Guidance (PH1) 2006 [https://www.nice.org.uk/guidance/ph1.
- 42.
National Institute for Health and Care Excellence. Smoking: workplace interventions (PH5) 2006 [https://www.nice.org.uk/guidance/ph5.
- 43.
National Institute for Health and Care Excellence. Stop Smoking Sevices (PH10) 2008 [https://www.nice.org.uk/guidance/ph10.
- 44.
National Institute of Health and Care Excellence. Cardiovascular disease: identifying and supporting people most at risk of dying early (PH15) 2008 [https://www.nice.org.uk/guidance/ph15/resources.
- 45.
National Institute for Health and Care Excellence. Weight management before, during and after pregnancy (PH27). 2010.
- 46.
National Institute for Health and Care Excellence. Maternal and Child Nutrition (PH11) 2008 [https://www.nice.org.uk/guidance/ph11.
- 47.
National Institute for Health and Care Excellence. Four commonly used methods to increase physical activity (PH2) 2006 [https://www.nice.org.uk/guidance/ph2.
- 48.
National Institute for Health and Care Excellence. Physical activity and the environment (PH8) 2008 [https://www.nice.org.uk/guidance/ph8.
- 49.
National Institute for Health and Care Excellence. Physical activity in the workplace (PH13). 2008.
- 50.
National Institute for Health and Care Excellence. Physical activity for children and young people (PH17) 2009 [https://www.nice.org.uk/guidance/ph17.
- 51.
National Institute for Health and Care Excellence. Sexually transmitted infections and under-18 conceptions: prevention (PH3) 2007 [https://www.nice.org.uk/guidance/ph3.
- 52.
National Institute for Health and Care Excellence. Alcohol-use disorders: prevention (PH24) 2010 [https://www.nice.org.uk/guidance/ph24.
- 53.
National Institute for Health and Care Excellence. HIV testing: increasing uptake in black Africans (PH33) 2011 [https://www.nice.org.uk/guidance/ph33.
- 54.
National Institute for Health and Care Excellence. HIV testing: increasing uptake in men who have sex with men (PH34) 2011 [https://www.nice.org.uk/guidance/ph34.
- 55.
National Institute for Health and Care Excellence. Behaviour change: general approaches (PH6) 2007 [https://www.nice.org.uk/guidance/ph6.
- 56.
National Institute for Health and Care Excellence. Cardiovascular disease prevention (PH25) 2010 [https://www.nice.org.uk/guidance/ph25.
- 57.
National Institute for Health and Care Excellence. Type 2 diabetes prevention: population and community-level interventions (PH35) 2011 [https://www.nice.org.uk/guidance/ph35.
- 58.
Towse A, Pritchard C, Devlin NJ. Cost-effectiveness thresholds: economic and ethical issues: King’s Fund; 2002.
- 59. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. pmid:843571
- 60.
National Institute for Health and Care Excellence (NICE). Behaviour change at population, community and individual levels (Public Health Guidance 6). London; 2007.
- 61. Prochaska JJ, Prochaska JO. A Review of Multiple Health Behaviour Change Interventions for Primary Prevention. American journal of lifestyle medicine. 2011;5(3):
- 62. Glazier RH, Bajcar J, Kennie NR, Willson K. A systematic review of interventions to improve diabetes care in socially disadvantaged populations. Diabetes Care. 2006;29(7):1675–88. pmid:16801602
- 63. Forbes CA, Jepson RG, Martin-Hirsch PP. Interventions targeted at women to encourage the uptake of cervical screening. The Cochrane Library. 2002.
- 64. Kelly-Weeder S, Phillips K, Rounseville S. Effectiveness of public health programs for decreasing alcohol consumption. Patient intelligence. 2011;2011(3):29. pmid:23180975
- 65. Sheikh I, Ogden J. The role of knowledge and beliefs in help seeking behaviour for cancer: a quantitative and qualitative approach. Patient education and counseling. 1998;35(1):35–42. pmid:9832895
- 66. Jagdeo T. Social structure reproductive health and the knowledge-behaviour gap. PROMOTION ET EDUCATION. 1996;3(3):11–4.
- 67. Cahill K, Stead LF, Lancaster T. Nicotine receptor partial agonists for smoking cessation. The Cochrane Library. 2008.
- 68. Gollwitzer PM. Implementation intentions: Strong effects of simple plans. American psychologist. 1999;54(7):493.
- 69. Gollwitzer PM, Sheeran P. Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in experimental social psychology. 2006;38:69–119.
- 70. Li J, Lovatt M, Eadie D, Dobbie F, Meier P, Holmes J, et al. Public attitudes towards alcohol control policies in Scotland and England: Results from a mixed-methods study. Soc Sci Med. 2017;177:177–89. pmid:28171817
- 71. West R, May S, West M, Croghan E, McEwen A. Performance of English stop smoking services in first 10 years: analysis of service monitoring data. BMJ: British Medical Journal. 2013;347.
- 72. Langley T, Lewis S, McNeill A, Szatkowski L, Gilmore A, Salway R, et al. The 2010 freeze on mass media campaigns in England: A natural experiment of the impact of tobacco control campaigns. The European Journal of Public Health. 2013;23(suppl 1):ckt126. 55.
- 73. Langley T, Szatkowski L, Lewis S, McNeill A, Gilmore AB, Salway R, et al. The freeze on mass media campaigns in England: a natural experiment of the impact of tobacco control campaigns on quitting behaviour. Addiction. 2014;109(6):995–1002. pmid:24325617
- 74. Makic MBF, VonRueden KT, Rauen CA, Chadwick J. Evidence-based practice habits: putting more sacred cows out to pasture. Critical care nurse. 2011;31(2):38–62. pmid:21459864
- 75. Michie S, Prestwich A. Are interventions theory-based? Development of a theory coding scheme. Health psychology: official journal of the Division of Health Psychology, American Psychological Association. 2010;29(1):1–8.
- 76. Abraham C, Kelly MP, West R, Michie S. The UK National Institute for Health and Clinical Excellence public health guidance on behaviour change: a brief introduction. Psychol Health Med. 2009;14(1):1–8. pmid:19085307
- 77. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. Relationship between funding source and conclusion among nutrition-related scientific articles. PLOS Med. 2007;4(1):e5. pmid:17214504
- 78. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326(7400):1167–70. pmid:12775614
- 79. Skinner BF. Operant behaviour. American Psychologist. 1963;18(8):503.
- 80.
Walker S. Learning theory and behaviour modification: Routledge; 1984.
- 81.
Sarafino EP. Behaviour modification: Wiley Online Library; 2001.
- 82. Tombor I, Shahab L, Brown J, West R. Positive smoker identity as a barrier to quitting smoking: findings from a national survey of smokers in England. Drug Alcohol Depend. 2013;133(2):740–5. pmid:24075070
- 83. Johnston M. What more can we learn from early learning theory? The contemporary relevance for behaviour change interventions. British journal of health psychology. 2016;21(1):1–10. pmid:26482915
- 84. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American psychologist. 2002;57(9):705.
- 85.
Bandura A, McClelland DC. Social learning theory. 1977.
- 86. Rosenstock IM. Historical origins of the health belief model. Health education monographs. 1974;2(4):328–35.
- 87.
Thaler Richard H, Sunstein Cass R. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press; 2008.
- 88. Martin J, Chater A, Lorencatto F. Effective behaviour change techniques in the prevention and management of childhood obesity. International Journal of Obesity. 2013;37(10):1287–94. pmid:23756676
- 89. Festinger L. A theory of social comparison processes. Human Relations. 1954;7(2):117–40.
- 90. Gerrard M, Gibbons FX, Lane DJ, Stock ML. Smoking cessation: Social comparison level predicts success for adult smokers. Health psychology: official journal of the Division of Health Psychology, American Psychological Association. 2005;24(6):623.
- 91. Litt DM, Lewis MA, Stahlbrandt H, Firth P, Neighbors C. Social Comparison as a Moderator of the Association Between Perceived Norms and Alcohol Use and Negative Consequences Among College Students. J Stud Alcohol Drugs. 2012;73(6):961–7. pmid:23036214
- 92. Hartmann-Boyce J, Johns D, Jebb S, Aveyard P. Behavioural Weight Management Review Group. Effect of behavioural techniques and delivery mode on effectiveness of weight management: systematic review, meta-analysis and meta-regression. Obes Rev. 2014;15(7):598–609. pmid:24636238
- 93. Yukl G, Wexley KN, Seymore JD. Effectiveness of pay incentives under variable ratio and continuous reinforcement schedules. J Appl Psychol. 1972;56(1):19.
- 94. Contreary KA, Chattopadhyay SK, Hopkins DP, Chaloupka FJ, Forster JL, Grimshaw V, et al. Economic impact of tobacco price increases through taxation: a community guide systematic review. Am J Prev Med. 2015;49(5):800–8. pmid:26188686
- 95. Holmes J, Meng Y, Meier PS, Brennan A, Angus C, Campbell-Burton A, et al. Effects of minimum unit pricing for alcohol on different income and socioeconomic groups: a modelling study. The Lancet. 2014;383(9929):1655–64.
- 96. LeBel EP, Peters KR. Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology. 2011;15(4):371.
- 97. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Bmj. 2008;337.
- 98.
Robinson KA, Saldanha IJ, Mckoy NA. Frameworks for determining research gaps during systematic reviews. 2011.
- 99. Michie S, Thomas J, Johnston M, Mac Aonghusa P, Shawe-Taylor J, Kelly MP, et al. The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implement Sci. 2017;12(1):121. pmid:29047393
- 100. Thompson SG, Higgins J. How should meta-regression analyses be undertaken and interpreted? Statistics in medicine. 2002;21(11):1559–73. pmid:12111920
- 101. Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness analyses. JAMA. 1996;276(16):1339–41. pmid:8861994
- 102. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Cost Eff Resour Alloc. 2013;11(1):6. pmid:23531194
- 103. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—Explanation and elaboration: A report of the ISPOR Health Economic Evaluations Publication Guidelines Task Force. Value Health. 2013;16.
- 104. Raftery J. Methodological limitations of cost-effectiveness analysis in health care: implications for decision making and service provision. Journal of evaluation in clinical practice. 1999;5(4):361–6. pmid:10579700
- 105. Appleby J, Devlin N, Parkin D. NICE’s cost effectiveness threshold. BMJ. 2007;335(7616):358. pmid:17717337
- 106. Eichler H-G, Kong SX, Gerth WC, Mavros P, Jönsson B. Use of cost-effectiveness analysis in health-care resource allocation decision-making: how are cost-effectiveness thresholds expected to emerge? Value health. 2004;7(5):518–28. pmid:15367247
- 107. Lewis-Beck MS. Stepwise regression: A caution. Political Methodology. 1978:213–40.
- 108. Henderson DA, Denison DR. Stepwise regression in social and psychological research. Psychol Rep. 1989;64(1):251–7.
- 109. Whittingham MJ, Stephens PA, Bradbury RB, Freckleton RP. Why do we still use stepwise modelling in ecology and behaviour? J Anim Ecol. 2006;75(5):1182–9. pmid:16922854
- 110. Olusegun AM, Dikko HG, Gulumbe SU. Identifying the limitation of stepwise selection for variable selection in regression analysis. Am Theoretical and Appl Statistics. 2015;4(5):414–9.
- 111.
Craig P, Frohlich K, Mykhalovskiy E, White M. on behalf of the Canadian Institutes of Health Research (CIHR) National Institute for Health Research (NIHR) Context Guidance Authors Group.(2018) Taking account of context in population health intervention research: guidance for producers, users and funders of research. NIHR Evaluation, Trials and Studies Coordinating Centre.