Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

The efficacy of Personalized Normative Feedback interventions across addictions: A systematic review and meta-analysis

  • Jenny Saxton ,

    Roles Formal analysis, Methodology, Writing – original draft, Writing – review & editing

    ‡ These authors share first authorship on this work.

    Affiliation School of Population Health, University of Auckland, Auckland, New Zealand

  • Simone N. Rodda ,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

    ‡ These authors share first authorship on this work.

    Affiliations School of Population Health, University of Auckland, Auckland, New Zealand, School of Psychology, Deakin University, Geelong, Australia

  • Natalia Booth,

    Roles Data curation, Methodology, Project administration, Writing – review & editing

    Affiliation School of Population Health, University of Auckland, Auckland, New Zealand

  • Stephanie S. Merkouris,

    Roles Formal analysis, Methodology, Supervision, Writing – review & editing

    Affiliation School of Psychology, Deakin University, Geelong, Australia

  • Nicki A. Dowling

    Roles Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing

    Affiliations School of Psychology, Deakin University, Geelong, Australia, Melbourne Graduate School of Education, University of Melbourne, Parkville, Australia


Personalized Normative Feedback (PNF) may help address addictive disorders. PNF highlights discrepancies between perceived and actual peer norms, juxtaposed against self-reported behavior. PNF can be self-directed and cost-efficient. Our study estimates the efficacy of PNF alone, and in combination with other self-directed interventions, to address frequency and symptom severity of hazardous alcohol use, problem gambling, illicit drug and tobacco use. We searched electronic databases, grey literature, and reference lists of included articles, for randomized controlled trials published in English (January 2000-August 2019). We assessed study quality using the Cochrane Risk of Bias tool. Thirty-four studies met inclusion criteria (k = 28 alcohol, k = 3 gambling, k = 3 cannabis, k = 0 tobacco). Thirty studies provided suitable data for meta-analyses. PNF alone, and with additional interventions, reduced short-term alcohol frequency and symptom severity. PNF with additional interventions reduced short-term gambling symptom severity. Effect sizes were small. PNF did not alter illicit drug use. Findings highlight the efficacy of PNF to address alcohol frequency and symptom severity. The limited number of studies suggest further research is needed to ascertain the efficacy of PNF for gambling and illicit drug use. Cost-effectiveness analyses are required to determine the scale of PNF needed to justify its use in various settings.


Addictive behaviors associated with alcohol, tobacco, illicit drugs and gambling can have considerable negative consequences for individuals, families and the wider society. Alcohol use is the leading global risk factor for death in men and women aged 15–49 [1]. Alcohol use also causes substantial harm, attributed to 8.9% and 2.3% of disability adjusted life years in men and women 15–49, respectively [1]. Tobacco control measures and the widespread adoption of the World Health Organization’s 2003 Framework Convention on Tobacco Control have been linked to impressive reductions in tobacco use [2, 3]. Yet, global tobacco use prevalence remains substantial at 25%, and tobacco use was attributed to 6.4 million deaths in 2015 alone [4]. In terms of illicit drugs, cannabis dependence is the most common substance use disorder, with an estimated 22·1 million cases globally in 2016, and 31·8 million disability adjusted life years attributed to drug use overall [5]. Gambling is a common practice worldwide, though lack of data examining its relationship to health and mortality makes it difficult to estimate population level harms through metrics such as disability adjusted life years [6]. As an alternative measure, Browne and colleagues assessed eight domains of gambling-related harm to estimate decrements in health-related quality of life amongst gamblers in New Zealand [6]. Their findings indicated health-related quality of life reductions of 0.18, 0.37 and 0.54 for low-risk, moderate-risk and problem gamblers respectively, equivalent to 2.5 times the harm caused by diabetes. Gambling problems are estimated at an average of 2.3% internationally [7].

The latest edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) now classifies these addictive behaviors as Substance-Related and Addictive Disorders. The update takes into account new research suggesting that gambling disorder shares features of substance use disorders in terms of brain origins, clinical and physiological manifestations, comorbidity and effective treatment options [8]. Moreover, there is considerable comorbidity between these addictive behaviors [913]–. Psychological interventions such as cognitive-behavioral therapies (CBT) and motivational interviewing demonstrate good outcomes for individuals reporting these addictive behaviors [1417]. Unfortunately, only 10–20% of people with addictive behaviors access face-to-face treatments, and those that do tend to have the most severe problems [18]. People with heavy, but less severe use, rarely seek help despite experiencing associated negative consequences [19].

Screening and brief interventions (SBIs), such as the Alcohol Use Disorders Identification Test (AUDIT), provide both an opportunity for identifying people with problems who are not yet seeking help and a suite of interventions that can be delivered in primary care at low cost [5]. Other Screening and brief interventions (SBIs) delivered via the internet usually last 5–20 minutes and have been reported as effective in reducing hazardous and harmful drinking [20, 21], as well as illicit drug use [22, 23] and gambling problems [24]. A 2017 Cochrane review found internet-based tobacco use interventions were significantly more effective than non-active controls at six months [25]. Effect sizes are generally smaller than more intensive, longer treatments, or those involving a clinician [2527], but their brevity and ease of access appear attractive to those with lower levels of symptom severity [28, 29]. So whilst Screening and Brief Interventions (SBIs) may be somewhat less effective than gold standard treatments, such as CBT for gambling [16], it may still be worth offering them due to their greater cost efficiency and wider reach.

Social norms approaches are a potentially powerful and cost-effective way to promote behavior change [30] and may be incorporated into SBIs to help address the burden of addictive behaviors [31, 32]. Social norms approaches include various distinct interventions, operationalized from different aspects of social norms theory e.g., social marketing, fear-based methods, and personalized normative feedback (PNF) [30].

PNF was first developed in the United States in response to college student drinking. At this time, multiple studies had indicated that college students over-estimated the quantitative and frequency of alcohol consumption in peer groups [33]. This bias appeared stronger for personally relevant social groups (e.g., college fraternities) compared with all students or adults in general. Early studies delivering PNF with mail out questionnaires and feedback (also delivered by mail) indicated an impact on the frequency and amount of alcohol consumed [34]. The delivery mechanism changed in the early 2000’s with the emergence of computer-delivered interventions. This delivery mechanism vastly increased the capacity to deliver PNF in real time (rather than waiting for mail feedback) and allowed PNF to be more readily tailored and presented in more sophisticated formats (e.g., using graphics).

Personalized normative feedback (PNF) interventions can be considered as a subset of personalized feedback interventions (PFIs). Personalized feedback interventions (PFIs) aim to ‘increase the salience of normative and personal standards in order to promote thoughtful consideration’ about one’s own behavior [35]. Personalized normative feedback interventions (PNF) make use of injunctive and/or descriptive normative information to elicit behavior change. Injunctive norms refer to social judgements about a particular behavior by an individual’s peer group; descriptive norms refer to the prevalence of a particular behavior amongst an individual’s peer group [36]. The premise of PNF is that individuals misperceive (i.e., over or underestimate) consumption levels or judgements of their peers, which contributes to maintaining their own problematic behavior. PNF asks individuals to provide information about their own consumption then presents them with the true injunctive and/or descriptive norms for their peer group. The theory behind PNF is that when confronted with their misperception of their peer group’s behavior and/or the social disapproval of their peer group, an individual will adjust their own behavior towards the newly realized norm [30].

Review rationale

Given the DSM-5 considers hazardous alcohol use, tobacco use, illicit drug and problem gambling as Substance-Related and Addictive Disorders, which are often co-occurring and share common mechanisms [37], we seek to understand whether PNF is a candidate intervention for each of these four disorders. We have chosen to focus on self-directed PNF interventions, as this will enable us to determine the efficacy of a low-cost, low resource intervention with a potentially wide reach.

Existing systematic reviews of social norms approaches to promote behavior-change, including PNF, have treated distinct personalized feedback interventions (PFIs) as though they were the same [3840]. This prevents an understanding of which social norms approaches work best, under what circumstances, and for which problems. PNF is also frequently implemented as part of a multicomponent intervention when examining the evidence, including additional information such as official guidelines for ‘safe’ levels of use, or self-help strategies. In these multicomponent interventions, the mechanisms underpinning PNF could be undermined by one of the other elements (e.g., fear-based messaging) [30]. To fully understand the utility of PNF, it is important to differentiate between ‘pure’ PNF (i.e., PNF alone) and PNF in combination with other approaches (‘mixed PNF’ interventions).

While several systematic reviews have explored the efficacy of PNF to reduce alcohol, illicit drug use and gambling, largely in college populations, no single review has considered different addictive disorders with a range of sample types, and none have included tobacco as a target behavior. Furthermore, none of these systematic reviews have isolated the efficacy of self-directed PNF (alone or in combination with other self-directed interventions) to address each of the four substance-related and addictive disorders we focus on in the current review. Most existing reviews focus on PNF or other norms-based interventions delivered in-person [38, 4148], do not target people with problems [47, 49, 50], and/or only include college student samples or samples of young people for a single addiction type [46, 47, 49, 50], which limits the generalizability of their findings. Several reviews also include studies with active control conditions [45, 47, 49, 50], which can reduce statistical power to identify intervention effects [43], whilst others allow non-randomized controlled trial (RCT) designs [49, 50], which are at higher risk of confounding and bias than RCTs [51].

Review aims

  1. To examine the efficacy of ‘pure PNF’ interventions (i.e., no other intervention implemented) for hazardous alcohol use, problem gambling, illicit drug and tobacco use, relative to passive control groups, in reducing frequency of use and symptom severity.
  2. To examine the efficacy of PNF plus self-directed interventions (‘mixed PNF interventions’) for hazardous alcohol use, problem gambling, illicit drug and tobacco use, relative to passive control groups, in reducing frequency of use and symptom severity.
  3. To examine whether addictive disorder type, setting (e.g., university environment), and type of additional intervention components explain the variability in the magnitude of the PNF intervention effects.
  4. To examine the extent to which methodological risk of bias characteristics influence PNF intervention effects.


Our reporting of this systematic review is compliant with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). We published the protocol for this review in the PROSPERO database of systematic reviews [CRD42018093549], which we updated in May 2019 (original version, August 2018) [52]. Differences between the updated PROSPERO protocol and the published review include: (1) We included tobacco use as an additional addictive disorder for review; (2) We amended one study exclusion criterion: instead of excluding studies where not all participants were exposed to normative feedback, we excluded studies where not all participants were given the opportunity to access the PNF intervention. This was due to several studies reporting incomplete intervention exposure amongst participants (e.g., not all participants downloaded and used a PNF app), despite giving all participants the opportunity to do so; (3) We conducted additional sub-group analyses to explore the influence of setting, and additional intervention components on PNF efficacy; and (4) We conducted additional sensitivity analyses to assess whether papers for which we converted medians to means affected our findings.

Search strategy

Our systematic search included an electronic database search of English language articles in EMBASE, MEDLINE, PsycINFO and the Cochrane Library Databases from January 1st 2000 (consistent with the advent of computer-delivered PNF) to August 28th 2019. Our search strategy included a combination of keywords and wildcards. This combination was intervention type (e.g., social norms, personalized feedback) AND the addictive behaviors (e.g., gambling, alcohol, drug, tobacco) AND treatment (e.g., intervention, trial). We hand-searched the reference lists of included studies. Our grey literature search comprised: (1) searching for otherwise unpublished trial data with the search terms ‘alcohol and personalized feedback’, ‘gambling and personalized feedback’, ‘drug or substance and personalized feedback’ and ‘smoking or tobacco and personalized feedback’ in the following trial registers: US, Metaregister of controlled trials, and WHO International clinical trials registry platform search portal (with the exception of tobacco use, as the portal was no longer available); and (2) using the same search terms, conducting four Google searches for reports of funded projects, where the first 100 entries were examined for each search. Search terms for each database are available in S1 Appendix.

Study eligibility criteria

Studies were eligible for inclusion if they met the following criteria: (1) were RCTs; (2) at least one arm used a PNF component, in which the feedback needed to include reference to the participant’s own alcohol, gambling, illicit or prescription drug or tobacco use, and descriptive and/or injunctive normative information about alcohol, gambling, illicit drug or tobacco use; (3) the PNF intervention was delivered to individuals (not groups); (4) the PNF intervention was self-directed, however interventions where researchers instructed participants to look at personalized normative feedback were included [53], as were study debriefs by researchers [54]; (5) study samples consisted of adults (18 years and older), or mixed groups of adults and adolescents who were 16 years or older; (6) study samples consisted of individuals with some level of problematic use of alcohol, gambling, drugs or tobacco use at baseline, as determined by a screening tool, health professional, or standard definition by researchers that attempted to identify regular moderate-heavy consumption or binge consumption; (7) the study included a passive control group (i.e., no intervention, assessment only, or generic health feedback that did not include gambling, alcohol, illicit drug use or tobacco use feedback); we consider that PNF could be primarily delivered as an SBI for users not yet engaged with other interventions, for whom absolute efficacy estimates (with passive controls) are more relevant than relative efficacy estimates (with active controls); and (8) the article was published in a peer-reviewed journal in English between January 2000 and August 2019, or was identified through grey literature searches of Google or any of the three controlled trial registers listed in the search strategy section covering the same time period; and (9) studies included at least one outcome measure of alcohol, gambling, illicit drug use or tobacco use (i.e., frequency or symptom severity).

We focused on frequency and severity outcomes as they are indices that can consistently be applied across all of the included substance and behavioral addictions. Frequency (Outcome 1) was defined as how often or how frequently participants engaged in drinking, gambling, illicit drug or tobacco use in a given reference period. This may be termed as the number of drinking/gambling/illicit drug use/smoking days, occasions or episodes. We excluded outcomes measuring quantity-type frequency variables (e.g., number of drinks per week, number of bets placed) and binge-related outcomes, as they may be measuring a different construct. Measures of subjective change in drinking were also excluded, as it was unclear whether they referred to quantity or frequency (e.g., if participants were asked if their drinking had increased, decreased, or stayed the same in the past month). Symptom severity (Outcome 2) was defined as any standardized or un-standardized measure of problem severity or harm in relation to drinking, gambling, illicit drug or tobacco use. We excluded quantity measures (e.g., number of drinks consumed, dollars spent on gambling) and blood alcohol content (BAC) as they are addiction-specific and not applicable across the range of substance and behavioral addictions our review included. We also excluded measures of attitudinal change because our aim was to assess the effect of PNF on behavior change.

Articles were excluded from the current review if the: (1) PNF intervention primarily targeted weight loss, or any other health behaviors that did not include alcohol, gambling, illicit drug or tobacco use; (2) intervention was labelled as PNF, but descriptive or injunctive normative feedback was not provided; (3) PNF intervention targeted people with specific physical or psychological comorbidities (e.g., war veterans with post-traumatic stress disorder), which are less generalizable to other populations; (4) PNF intervention was delivered in a group setting; (5) PNF intervention was a prevention program designed only for people not yet engaged in problem alcohol consumption, gambling, illicit drug or tobacco use (sometimes referred to as ‘at risk’ groups) or if subgroup analyses based on severity level were conducted, where groups were identified after randomization [55]; (6) participants received any formal treatment (psychological or drug therapy) in conjunction with PNF, or who were within 12 months of completing formal treatment for problem use/addictive disorder; (7) participants were mandated to complete the program (e.g., for legal reasons); (8) sample comprised only children or adolescents younger than 16 years; (9) studies did not assess relevant outcome measures (i.e., alcohol, gambling, drug or tobacco frequency or severity); (9) studies provided insufficient information about the intervention for it to be categorized or provided no usable data; (10) article was a review, conference proceeding, abstract, book or book chapter, or protocol; (11) studies included active comparison/control groups (e.g., relevant health information given as leaflet, or where participants rated the usefulness of relevant self-help information); (12) studies required in-person contact with a researcher, facilitator or health professional for all intervention arms, or where participants were given verbal feedback about their scores; we deliberately excluded interventions requiring in-person contact as our interest was focused on estimating the efficacy of PNF as a very low cost, low resource intensive intervention; and (13) participants in the intervention arm were not all given the opportunity to take part in the PNF intervention, or where it was not possible to verify that the majority of participants were given access to the intervention.

Article screening, data extraction, and quality assessment

Two reviewers independently screened all titles and abstracts of articles retrieved by the literature searches against the inclusion and exclusion criteria. Discrepancies between reviewers were resolved through discussion or by a third reviewer if an agreement could not be reached. Full text review was also conducted by two reviewers independently. Discrepancies were again resolved by a third reviewer if necessary.

Data were independently extracted by two reviewers using a standardized extraction sheet in Microsoft Excel. The data extracted included comprehensive details about the study characteristics (e.g., year of publication, sample type), full descriptions of the intervention and control groups and outcome measures (e.g., type of measure employed, means and standard deviations). Any variations in the data extracted by the two reviewers were resolved through discussion, involving a third reviewer when necessary. Studies with missing data were not requested from study authors.

We used the Cochrane Risk of Bias tool, version 2.0 [56] to assess studies for risk of bias. The tool assesses papers for potential bias on five domains: (1) the randomization process; (2) bias due to deviations from the intended interventions; (3) bias due to missing outcome data; (4) bias in measurement of the outcome; and (5) bias in selection of the reported result. Each domain is scored as low, some concerns or a high risk of bias. If there are some concerns for one domain only, the overall paper is judged to have some concerns; if the paper has some concerns in two or more domains, or high risk in one or more domain, the overall judgement is the paper is at high risk of bias.

Two reviewers independently assessed the same one-third of the papers to ensure consistent application of the tool. After resolving any discrepancies, the remaining two-thirds of papers were assessed by one reviewer. All papers judged to have some concerns or a high risk of bias were also double-assessed. We used original articles and their published protocols (where available) to arrive at our Risk of Bias judgements.

Data synthesis

Description of included studies and meta-analysis.

We briefly described the characteristics and tabulated the results of all included studies, considering pure and mixed PNF studies separately. We then ran a series of meta-analyses of studies providing suitable data, which were performed in Review Manager (version 5.3), with forest plots created in STATA (version 13). Our main analyses consisted of random-effects models with the inverse variance method to generate standardized mean differences (SMD), with 95% confidence intervals, based on follow up means and standard deviations (SDs) for each of the continuous outcomes (frequency and symptom severity), four follow up periods, and for the pure and mixed PNF studies separately. Conventional thresholds were used to label effect sizes as small (0.2), medium (0.5) or large (0.8) [57]. Heterogeneity among studies was estimated using Chi square and associated P-value, and the I2 statistic. Adapting guidance from the Cochrane Review Handbook, we considered heterogeneity to be minimal if I2 was 0–40% and the Chi square p-value was not significant (p>0.1), moderate if I2 was 41–60% with a significant Chi square p-value (p≤0.10), and substantial if I2 was 61–100% with a significant Chi square p-value [58]. A minimum of two estimates were required to conduct a meta-analysis.

Our full list of decision rules for meta-analytic estimates and statistical conversions is provided in S2 Appendix, but the key rules are summarized as follows: (1) If more than one frequency or severity outcome was reported in the same article, preference was given to measures employed more frequently, followed by complete tools rather than sub-scales, then standardized over unstandardized tools, and multi-dimensional over single dimensional tools (for symptom severity) or ‘days’ followed by ‘occasions’, followed by ‘episodes’ (for frequency); (2) If no means or SDs were available, we calculated them as per the conversion formulae available in the Cochrane handbook, when possible [59]. Using the same guidance material, we calculated single intervention means, SDs and numbers of participants where multiple intervention means, SDs and numbers of participants were reported. Where possible, we also converted medians to means according to Hozo’s guidance [60] but given that conversions of medians to means is not yet standard practice for systematic reviews, we conducted sensitivity analyses omitting these studies to assess any difference in results. We also obtained estimates of relevant data from published graphs if necessary. Where data could not be converted or estimated, studies were excluded from the meta-analysis and reported in the description of studies section only; (3) Intention to treat data were preferred over completer data; (4) Data from the least adjusted models were preferred over more adjusted; (5) Overall results were preferred over males and females separately; (6) If an article presented two or more values within a single follow up period (as defined in our review), we used data from the longest follow up period; (7) If multiple papers were available for the same data set, we used the article reporting our preferred outcomes, then with the longest follow up period, then with the least adjusted results, then using the most robust measurement tool.

Subgroup analyses

Pre-specified subgroup analyses were performed to investigate potential differences in PNF intervention effects according to the following study characteristics: (1) addiction type (alcohol, gambling, illicit drugs, tobacco use); (2) sample type (university/college students; non-university/college students); and (3) type of additional intervention component included, using common categories emerging from the articles (mixed PNF studies only).

Sensitivity analyses

To examine the influence of methodological characteristics on the PNF intervention effects, we excluded articles rated overall as having ‘some concerns’ or ‘high risk of bias’ from the main analyses. We also sought to examine whether excluding papers for which we had converted medians to means changed the findings from our main analyses.


Search results

5,171 articles were screened from primary and secondary (reference list) searches after the removal of duplicates, with 232 full text articles reviewed. Of these, 34 studies were included in the final review, with 30 of these studies providing sufficient data for inclusion in the meta-analyses. Full details of the literature search results are presented in the PRISMA diagram in Fig 1.

Fig 1. PRISMA diagram for systematic review and meta-analysis.

Characteristics of included studies

Pure and mixed PNF study characteristics, study results and risk of bias assessments are presented in Tables 1 and 2 respectively. Further details of specific intervention elements and the number of studies using them are provided in S3 Appendix.

Table 1. Characteristics of included pure Personalized Normative Feedback (PNF) studies (k = 13).

Table 2. Characteristics of included mixed Personalized Normative Feedback (PNF) studies (k = 24).

Number of studies and focus behaviors

Thirteen studies examined the efficacy of at least one pure PNF arm against a passive control. Twelve of these focused on alcohol, and one on gambling. We did not identify any pure PNF studies of illicit drug or tobacco use that met our inclusion criteria.

Twenty-four studies tested the efficacy of PNF combined with other self-directed intervention components against a passive control. Three of these also included pure PNF arms [65, 70, 90]. Of these, 19 studies focused on alcohol, two on gambling, and three on illicit drug use, specifically cannabis. We did not identify any mixed PNF studies of tobacco use that met our inclusion criteria.

Risk of bias

The majority of pure PNF studies were judged to be at low risk of bias (k = 10, 76.9%), with three exceptions, which were all found to have ‘some concerns’ [32, 63, 70]. Most mixed PNF studies were judged to be at low risk of bias (k = 18, 75%), with five studies judged to have ‘some concerns’ overall [53, 70, 73, 85, 88] and one study judged to be at overall high risk of bias [87]. Domain 1 –risk of bias about the randomization process—was the most common area where papers were considered to have some risk of bias, and this was generally due to lack of detail rather than authors actively stating inappropriate group allocation.


The results of the main meta-analyses are as forest plots presented in Figs 25. These figures include the SMDs and 95% CIs for individual studies within each analysis, as well as the pooled effects and heterogeneity statistics for each of the main analyses. The results of all subgroup analyses related to addiction type, setting and additional intervention components (mixed PNF studies) and the sensitivity analyses are presented in S4 Appendix for pure PNF studies, and S5 Appendix for mixed PNF studies.

Fig 2. The efficacy of pure PNF on frequency across follow-up periods, b.

a Number of participants in all studies: 0–3 months PNF n = 2128, control n = 800; 4–11 months PNF n = 2043, control n = 807; 12–23 months PNF n = 1330, control n = 316. b Insufficient studies were available for meta-analyses at 24 months+.

Fig 3. The efficacy of pure PNF on symptom severity across follow-up periodsa, b.

a Number of participants in all studies: 0–3 months PNF n = 3396, control n = 1364; 4–11 months PNF n = 2753, control n = 950; 12–23 months PNF n = 1330, control n = 316. b Insufficient studies were available for meta-analyses at 24 months+.

Fig 4. The efficacy of mixed PNF on frequency across follow-up periodsa, b.

a Number of participants in all studies: 0–3 months PNF n = 3192, control n = 2532; 4–11 months PNF n = 3722, control n = 3639; 12–23 months PNF n = 444, control n = 478b Insufficient studies were available for meta-analyses at 24 months+.

Fig 5. The efficacy of mixed PNF on symptom severity across follow-up periodsa, b.

a Number of participants in all studies: 0–3 months PNF n = 3145, control n = 2369; 4–11 months PNF n = 4149, control n = 4068; 12–23 months PNF n = 444, control n = 478. b Insufficient studies were available for meta-analyses at 24 months+.

Pure PNF vs control

Frequency: Main analyses.

As shown in Fig 2, there were no significant differences between pure PNF intervention and the control groups on frequency at 0–3 months (k = 6), 4–11 months (k = 6) or 12–23 months (k = 2) post-baseline. Heterogeneity for 0–3 and 4–11 month follow up periods was substantial, and was minimal for 12–23 months. There were no studies available for the follow up period ≥24 months.

Symptom severity: Main analyses.

As shown in Fig 3, there were significantly lower symptom severity scores in the pure PNF group versus the control at the 0–3 month follow up period, with a small effect size. This analysis included eleven studies (ten alcohol studies and one gambling study), where heterogeneity was minimal. Results were non-significant for further follow up periods of 4–11 months (k = 7) and 12–23 months (k = 2).

Mixed PNF vs control

Frequency: Main analyses.

As shown in Fig 4, eleven studies were included in the meta-analysis for the 0–3 month follow up period: seven alcohol studies, two gambling studies and two illicit drug use studies. The pooled SMD indicated small but significantly lower frequency in the mixed PNF group compared to the control groups, with minimal heterogeneity. At 4–11 months, seven studies (six focused on alcohol and one focused on illicit drugs) were meta-analyzed, again showing small but significantly lower frequency in the mixed PNF group than the control groups, where heterogeneity was moderate. At 12–23 months, only two studies were available for meta-analyses, both on alcohol. This analysis showed no significant difference between mixed PNF and control groups for frequency, and heterogeneity was minimal.

Symptom severity: Main analyses.

As shown in Fig 5, there was no significant difference in symptom severity amongst mixed PNF participants compared to controls for the 0–3 month follow-up period. Fifteen studies were available for this meta-analysis (11 alcohol studies, 2 gambling studies, and 2 illicit drug studies), and heterogeneity was minimal. Results were also non-significant for the 4–11 month follow up period, the analysis comprising 9 alcohol studies and one illicit gambling study, with substantial heterogeneity. The 12–23 month analysis included 2 alcohol studies with minimal heterogeneity, and indicated a significant between group difference that favored the control group.


Main findings

This is the first systematic review and meta-analysis evaluating the effectiveness of PNF interventions to address four addictive behaviors: hazardous alcohol use, problem gambling, illicit drug and tobacco use. Our review provides mixed evidence for the use of PNF to address common Substance-Related and Addictive Disorders, but there was a notable lack of studies focused on illicit drugs, few on gambling, and none on tobacco use. This limited our statistical power and our results should be interpreted with caution. Without additional studies in these areas, we cannot draw firm conclusions about the utility of PNF across addictive behaviors. Our findings do provide some support for the use of PNF to address alcohol frequency and symptom severity in college/university settings, and to a lesser extent, gambling symptom severity in a broader range of settings. Mixed PNF studies appear to have slightly more enduring effects for some comparisons (up to 11 months) than pure PNF studies (0–3 months). With so few longer-term studies available we cannot draw firm conclusions about PNF’s longevity. There is no evidence from our review that PNF can reduce frequency and symptom severity from cannabis use, and no studies were available for other illicit drugs or tobacco use.

Comparison to the wider literature


Tanner-Smith and Lipsey [47] reported similar effect sizes to ours for frequency in two meta-analyses of interventions which overlap with PNF (norms referencing and personalized feedback), with young people 19–30 years, where their outcome (alcohol consumption) included frequency, quantity and blood alcohol content. They reported a slightly larger and more persistent effect size for symptom severity (up to one year). Their inclusion criteria did allow face-to-face contact with health professionals though, which our review did not, and which are generally associated with larger effect sizes [41]. A review by Smedslund et al. [50] of prevention studies (i.e., not yet problem users) also identified similar effect sizes to ours in short and longer-term (≥6 months) studies, though their estimates were based on low quality evidence. A review of reviews by Stockings et al. [46] reported a small reduction in problematic alcohol use (defined as heavy use, which might cause harm to self or others) amongst 10–24 year olds in response to social norms feedback, but the authors considered the size of effect to be of no meaningful benefit. Schmidt et al. [45] reported smaller effect sizes than ours, which were non-significant at short term follow up, for interventions involving printed or computer-generated feedback seeking to reduce numbers of heavy drinking episodes. Finally, Foxcroft et al. [39] reported modest reductions in alcohol frequency, with smaller effect sizes than ours, for web/computer based normative feedback interventions; they reported no effect for mailed normative feedback. Both groups of interventions permitted the inclusion of non-personalized norms, and participants who were not necessarily problem drinkers, which could explain their smaller effect sizes. Finally, a review of standalone PNF also observed small, but significant reductions in alcohol related harm, with a larger effect size than ours, though their sole focus was on college students for whom PNF seems to be the most effective [49].


Our non-significant findings for gambling frequency contrast with a narrative review by Marchica and Derevensky [38], which reported promising results for personalized feedback interventions, including PNF. This difference could be explained by their inclusion of a mixture of face-to-face and self-directed PFI interventions in their analyses, whilst ours was restricted to self-directed. Meta-analyses by Goslar et al. [41] identified small significant effect sizes in the short term for self-guided treatments (CBT-based workbooks, and personalized feedback), but where effect sizes were close to zero for longer follow up periods. The greater variety of interventions (including non-PNF) they tested together could explain the difference in results compared to our review. Finally, Quilty et al. [43] identified short-term improvements in gambling behavior and associated problems in response to brief interventions (including PNF, MI/enhancement, personalized feedback and brief advice). Again, this contrasts with our finding, though the outcomes are not directly comparable, and Quilty et als gambling behavior variable included presence/absence of gambling and severity, as well as frequency.

In line with our findings for gambling symptom severity, Quilty et al. [43] detected small and significant improvements in gambling problems of a similar effect size in their meta-analysis of brief face-to-face gambling interventions. In contrast, Goslar et al. [41] identified non significant small effect sizes for global severity in post-treatment and follow up periods.

Illicit drug use.

In line with our findings, Stockings et al. [46] concluded that social norms feedback is ineffective for addressing heavy drug use amongst people 10–24 years, based on low levels of evidence. Similarly, Smedslund et al. [50] reviewed prevention studies and observed no impact of computerized brief interventions on cannabis use in short and longer terms, as well as noting the general lack of studies for this behavior. Possible explanations for the ineffectiveness of SBIs in general for illicit drug use are offered by a narrative review, which considers illicit drug use as a different category of behavior than alcohol, for example [44]. The author highlights that drugs are taken, despite widespread knowledge that they are (often) illegal and socially unacceptable, and users may not respond to normative information in the same way as legal and socially sanctioned behaviors such as alcohol.

Additional intervention components

Our findings are in line with meta-analyses by Tanner-Smith and Lipsey [47], who considered various intervention components similar to the categories emerging in our papers, which were not associated with significantly larger or smaller benefits on their alcohol outcomes.

Though we did not formally compare the efficacy of pure and mixed PNF studies, both produced similar effect sizes. We did observe that mixed PNF studies reported significant findings for two medium term follow up periods (4–11 months) whereas pure PNF studies did not. It is possible that the additional components in mixed PNF studies led to more meaningful interventions (and therefore more enduring effects) from a participant perspective, but there were also far fewer pure PNF studies than mixed PNF studies, and in the absence of formal comparisons of the two types of PNF interventions, it is difficult to draw firm conclusions about whether mixed approaches are superior.

It was notable that in the PNF studies overall, very few studies (k = 1 pure, k = 2 mixed) made use of injunctive norms, so for that specific dimension, PNF interventions remain largely untested.


College/university settings were by far the most common intervention environment. There were insufficient studies from non-college/university settings to enable equivalent subgroup analyses. In other reviews, Tanner-Smith and Lipsey [47] found that whilst university, primary health care and remote/online settings gave similar results, emergency room settings did not result in significant benefits for young adults’ alcohol consumption or related problems. Conversely, Schmidt et al. [45] focused their meta-analyses on the efficacy of brief interventions (including PNF) in emergency departments on various alcohol outcomes (consumption quantity, intensity and number of heavy drinking episodes), and the majority of their comparisons indicated modest but significant effect sizes for up to 12 months. Further studies in non-university environments could help clarify the efficacy of PNF in a wider range of settings.

Follow up period

Most comparisons that were significant in our meta-analyses showed a weakening of effects over time, though one analysis (mixed PNF studies and alcohol symptom severity) saw a short-term effect favoring the intervention group turn into a long-term effect favoring the control group. The general weakening of effects we observed is in line with other reviews [45, 47]. Weakening effects are unsurprising given the brevity of PNF, but it highlights the potential benefit of repeating interventions before 12 months for sustained behavior change, though the evidence-base for whether repeating interventions is worthwhile is currently scarce.

Effect sizes for PNF versus other brief interventions

The modest significant effect sizes we identified for PNF appear to be similar to other brief interventions assessing frequency and symptom severity. For example, Tanner-Smith and Lipsey [47] compared the following SBIs to controls, where effect sizes are shown in parentheses for alcohol consumption and alcohol related problems respectively: CBT (0.13 and 0.10), Motivational Enhancement Therapy (0.20 and 0.17), Psycho-Educational Therapy (0.16 and 0.13). Combining Motivational Enhancement Therapy and CBT was counterproductive (0.03 and 0.00) whilst Expectancy Challenge resulted in the strongest effect sizes (0.36 and 0.34).

Limitations of the current evidence base

Firstly, our use of follow up means, rather than change from baseline could have over or underestimated PNF effects, depending on whether there were baseline imbalances in the outcome and which group they favored, though a recent paper suggests that this would not necessarily have changed our conclusions [92]. Several papers we included did not report baseline values, and as we were relying on published estimates without consulting authors of primary articles, we preferred to include these articles in our meta-analyses rather than omit them due to missing baseline data. As we only included RCTs, we also expected any baseline imbalances to be random, affecting intervention and control groups approximately equally. A second limitation to our review is the exclusion of non-English language articles which could have increased the number of studies included in our review. Thirdly, many of the included studies did not publish protocols prior to their trials. Since 2005, medical journals have required health related RCTs, including those that are behavioral treatments or educational programs, to be registered with an appropriate registry [93], but this has not been extended to the majority of addiction-related journals. In the absence of protocols, we relied upon the congruency of hypotheses and analysis plans with the results reported in the original articles, which could have overestimated study quality and affected our conclusions. Finally, in the context of other PFI and PNF systematic reviews, which have some degree of overlap with the present review, the impact of this paper may be incremental. However, it is also the first review to provide evidence about the efficacy of PNF across multiple addictive behaviors and settings, and we hope is useful for practitioners and users seeking to access a ‘ready to go’ low cost SBI.

Implications for clinical practice

How meaningful are our effect sizes? In a review of reviews of social norms interventions to reduce risky alcohol use in young people, Stockings et al. [46], concluded that though these led to reduced alcohol use, the small associated effect sizes raise questions as to the benefit at policy and practice levels. Whilst their inclusion criteria were broader than ours and are not specific to PNF, they raise an important question about whether it is worthwhile delivering interventions with such small effects. We cautiously concur with Tanner-Smith and Lipsey [47] who address the issue of modest effects and offer a different interpretation. They conclude that such interventions were ‘potentially worthwhile given their brevity and low cost’, and go on to say that brief interventions are not usually intended as full treatments but as a precursor to further interventions for those who need them, as well as to motivate and provide participants with tools and resources to manage their behaviors. If adopting this perspective, future research could investigate whether PNF is a useful kick start to more intensive intervention as necessary, or as a standalone intervention for people at the lower end of the risk continuum to assist with motivation and consideration of behavior change. Schmidt et al. [45] consider that the small effect sizes they observed in emergency departments warrant a ‘more cautious approach to widespread implementation’ of brief interventions in those settings. They do suggest that very brief and/or computerized approaches are preferred over more resource intensive brief interventions given resource and time pressures in emergency departments, and here there is a potential place for computerized PNF in further research.

Whilst there is not strong support for implementing interventions with such small effect sizes in environments with extreme resource constraints, cost-effectiveness studies would provide useful insight as to the true value of implementing interventions, such as computerized PNF, in a variety of settings. Available cost-effectiveness studies are scarce, and we are not aware of any assessing computerized PNF, though some studies are available for other brief interventions. One UK-based study using alcohol health workers to implement brief interventions to excessive drinkers attending sexual health clinics found mixed effects on alcohol outcomes at 6 months, with a mean cost of £12.60 per participant, which they considered was not a cost-effective use of resources [94] Conversely, one 2001 study in Australian primary health care estimated the cost of brief interventions for alcohol reduction at AUD$19.14-$21.50 and reported marginal costs per additional life year saved as below AUD$1873, which they describe as ‘highly encouraging’ [95]. A more recent modelling study concluded that national screening and brief intervention programs would be a cost effective way to reduce alcohol related morbidity and mortality in most EU countries [96].

The evidence is clear from longer term follow up studies that there is a time-limited effect of PNF, which largely disappears by 12 months, or sooner. This positions PNF more as a potentially useful beginning to further intensive intervention for those who need it, which could be incorporated into general health screening. PNF could be used to alert people to problem behavior, and be used to prompt motivation and engagement for addressing it, rather than PNF being a standalone intervention expected to produce sustainable change. It may be however that PNF is efficacious for alcohol than other addictions because of differing levels of importance and readiness to change. For example, people with gambling, illicit drugs and tobacco use report higher levels of perceived importance to change consumption compared to those engaged in hazardous alcohol use [9799]. These studies indicate perceived importance of change is predictive of outcomes when combined with confidence to change. Future research should investigate the impact of SBIs that combine awareness raising and strategies for action on varying levels of readiness for change. It is unclear whether PNF booster sessions would improve the longevity of benefits. Arguably, PNF could still be used as standalone, single intervention in college/university settings to minimize harm from alcohol consumption in the short-term.

Concluding statement

Our review provides evidence for the short-term efficacy of self-directed PNF to reduce alcohol frequency and symptom severity, and to a lesser extent gambling symptom severity based on a small number of studies. Our review does not provide evidence that self-directed PNF can be beneficial for addressing cannabis use, though again the number of available studies was very small and there were no studies of other illicit drugs. There were no studies addressing tobacco use. PNF studies tended to be conducted in college/university populations of young people, with predominantly White participants, and at the lower end of the spectrum of problem use which limits the generalizability of our findings. Whilst effects from mixed PNF studies appear to be slightly more enduring than pure PNF studies, our analyses did not provide support for the use of additional components to enhance PNF, again limited by the number of available studies. All significant effect sizes were small, but comparable with other more costly face-to-face interventions in primary care and emergency department settings. Cost effectiveness studies will help to resolve questions about whether implementing self-directed PNF at large scale is worthwhile.

Supporting information

S1 Appendix. Search terms for literature search.


S2 Appendix. Decision rules for the review and meta-analysis.


S4 Appendix. Sub-group and sensitivity analyses for pure PNF studies.


S5 Appendix. Sub-group and sensitivity analyses for mixed PNF studies.



We acknowledge Brenna Knaebe and Matt Brittain for their valuable help with data extraction.


  1. 1. Global Burden of Disease Alcohol Collaborators. Alcohol use and burden for 195 countries and territories, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet (London, England). 2018;392(10152):1015–35. pmid:30146330
  2. 2. Gravely S, Giovino GA, Craig L, Commar A, D’Espaignet ET, Schotte K, et al. Implementation of key demand-reduction measures of the WHO Framework Convention on Tobacco Control and change in smoking prevalence in 126 countries: an association study. The Lancet Public Health. 2017;2(4):e166–e74. pmid:29253448
  3. 3. WHO. Management of Substance Abuse: Screening and brief intervention for alcohol problems in primary health care. 2003.
  4. 4. Reitsma MB, Fullman N, Ng M, Salama JS, Abajobir A, Abate KH, et al. Smoking prevalence and attributable disease burden in 195 countries and territories, 1990–2015: a systematic analysis from the Global Burden of Disease Study 2015. The Lancet. 2017;389(10082):1885–906. pmid:28390697
  5. 5. Global Burden of Disease Alcohol and Drug Use Collaborators. The global burden of disease attributable to alcohol and drug use in 195 countries and territories, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. The lancet Psychiatry. 2018;5(12):987–1012. pmid:30392731
  6. 6. Browne M, Bellringer M., Greer N., Kolandai-Matchett K., Rawat V., Langham E., Rockloff M., Palmer Du Preez K., Abbott M. Measuring the Burden of Gambling Harm in New Zealand. 2017.
  7. 7. Williams RJ, Volberg Rachel A., Stevens Rhys M. G., Williams Robert J., Volberg Rachel A., Stevens Rhys M. G. The population prevalence of problem gambling: methodological influences, standardized rates, jurisdictional differences, and worldwide trends. Report prepared for the Ontario Problem Gambling Research Centre and the Ontario Ministry of Health and Long Term Care.; 2012 May 8, 2012.
  8. 8. American Psychiatric Association. DSM5: Substance-related and Addictive disorders. 2013.
  9. 9. Dowling NA, Cowlishaw S, Jackson AC, Merkouris SS, Francis KL, Christensen DR. Prevalence of psychiatric co-morbidity in treatment-seeking problem gamblers: A systematic review and meta-analysis. The Australian and New Zealand journal of psychiatry. 2015;49(6):519–39. pmid:25735959
  10. 10. Moss HB, Goldstein RB, Chen CM, Yi H-Y. Patterns of use of other drugs among those with alcohol dependence: Associations with drinking behavior and psychopathology. Addictive Behaviors. 2015;50:192–8. pmid:26151585
  11. 11. Sussman S, Lisha N, Griffiths M. Prevalence of the Addictions: A Problem of the Majority or the Minority? Evaluation & the Health Professions. 2011;34(1):3–56. pmid:20876085
  12. 12. Adams S. Psychopharmacology of Tobacco and Alcohol Comorbidity: a Review of Current Evidence. Curr Addict Rep. 2017;4(1):25–34. pmid:28357192
  13. 13. McGrath DS, Barrett SP. The comorbidity of tobacco smoking and gambling: A review of the literature. Drug and Alcohol Review. 2009;28(6):676–81. pmid:19930023
  14. 14. Magill M, Ray LA. Cognitive-behavioral treatment with adult alcohol and illicit drug users: a meta-analysis of randomized controlled trials. Journal of studies on alcohol and drugs. 2009;70(4):516–27. pmid:19515291
  15. 15. Connor JP, Haber PS, Hall WD. Alcohol use disorders. The Lancet. 2016;387(10022):988–98. pmid:26343838
  16. 16. Cowlishaw S, Merkouris S, Dowling N, Anderson C, Jackson A, Thomas S. Psychological therapies for pathological and problem gambling. Cochrane Database of Systematic Reviews. 2012(11). pmid:23152266
  17. 17. Neumann T, Rasmussen M, Ghith N, Heitmann BL, Tønnesen H. The Gold Standard Programme: smoking cessation interventions for disadvantaged smokers are effective in a real-life setting. Tobacco Control. 2013;22(6):e9. pmid:22705716
  18. 18. Grant BF, Goldstein RB, Saha TD, Chou SP, Jung J, Zhang H, et al. Epidemiology of DSM-5 alcohol use disorder: results from the National Epidemiologic Survey on Alcohol and Related Conditions III. JAMA psychiatry. 2015;72(8):757–66. pmid:26039070
  19. 19. Caldeira KM, Kasperski SJ, Sharma E, Vincent KB, O’Grady KE, Wish ED, et al. College students rarely seek help despite serious substance use problems. Journal of substance abuse treatment. 2009;37(4):368–78. pmid:19553064
  20. 20. O’donnell A, Anderson P, Newbury-Birch D, Schulte B, Schmidt C, Reimer J, et al. The impact of brief alcohol interventions in primary healthcare: a systematic review of reviews. Alcohol and alcoholism. 2013;49(1):66–78. pmid:24232177
  21. 21. Donoghue K, Patton R, Phillips T, Deluca P, Drummond C. The effectiveness of electronic screening and brief intervention for reducing levels of alcohol consumption: a systematic review and meta-analysis. Journal of medical Internet research. 2014;16(6):e142. pmid:24892426
  22. 22. Boumparis N, Karyotaki E, Schaub MP, Cuijpers P, Riper H. Internet interventions for adult illicit substance users: a meta‐analysis. Addiction. 2017;112(9):1521–32. pmid:28295758
  23. 23. Tait RJ, Spijkerman R, Riper H. Internet and computer based interventions for cannabis use: a meta-analysis. Drug and alcohol dependence. 2013;133(2):295–304. pmid:23747236
  24. 24. Petry NM, Ginley MK, Rash CJ. A systematic review of treatments for problem gambling. Psychology of Addictive Behaviors. 2017;31(8):951. pmid:28639817
  25. 25. Taylor GMJ, Dalili MN, Semwal M, Civljak M, Sheikh A, Car J. Internet-based interventions for smoking cessation. The Cochrane database of systematic reviews. 2017;9(9):CD007078-CD. pmid:28869775
  26. 26. McHugh RK, Hearon BA, Otto MW. Cognitive behavioral therapy for substance use disorders. Psychiatr Clin North Am. 2010;33(3):511–25. pmid:20599130
  27. 27. Sundström C, Blankers M, Khadjesari Z. Computer-Based Interventions for Problematic Alcohol Use: a Review of Systematic Reviews. International Journal of Behavioral Medicine. 2017;24(5):646–58. pmid:27757844
  28. 28. Tucker JA, Simpson CA. The recovery spectrum: from self-change to seeking treatment. Alcohol Res Health. 2011;33(4):371–9. pmid:23580021
  29. 29. Rodda SN, Hing N, Lubman DI. Improved outcomes following contact with a gambling helpline: the impact of gender on barriers and facilitators. International Gambling Studies. 2014;14(2):318–29.
  30. 30. Dempsey RC, McAlaney J, Bewick BM. A Critical Appraisal of the Social Norms Approach as an Interventional Strategy for Health-Related Behavior and Attitude Change. Frontiers in Psychology. 2018;9(2180). pmid:30459694
  31. 31. Bertholet N, Cunningham JA, Adam A, McNeely J, Daeppen J-B. Electronic screening and brief intervention for unhealthy alcohol use in primary care waiting rooms–A pilot project. Substance Abuse. 2019:1–9.
  32. 32. Miller MB, DiBello AM, Carey KB, Pedersen ER. Blackouts as a Moderator of Young Adult Veteran Response to Personalized Normative Feedback for Heavy Drinking. Alcoholism: Clinical and Experimental Research. 2018;42(6):1145–53. pmid:29602274
  33. 33. Baer JS, Stacy A, Larimer M. Biases in the perception of drinking norms among college students. Journal of studies on alcohol. 1991;52(6):580–6. pmid:1758185
  34. 34. Agostinelli G, Brown JM, Miller WR. Effects of normative feedback on consumption among heavy drinking college students. Journal of drug education. 1995;25(1):31–40. pmid:7776148
  35. 35. Miller MB, Meier E, Lombardi N, Leffingwell TR. Theories of behaviour change and personalised feedback interventions for college student drinking. Addiction Research & Theory. 2015;23(4):322–35.
  36. 36. Lapinski MK, Rimal RN. An Explication of Social Norms. Communication Theory. 2006;15(2):127–47.
  37. 37. Kim HS, Hodgins DC. Component Model of Addiction Treatment: A Pragmatic Transdiagnostic Treatment Model of Behavioral and Substance Addictions. Frontiers in psychiatry. 2018;9:406-. pmid:30233427
  38. 38. Marchica L, Derevensky JL. Examining personalized feedback interventions for gambling disorders: A systematic review. Journal of Behavioral Addictions. 2016;5(1):1–10. pmid:28092190
  39. 39. Foxcroft DR, Moreira MT, Almeida Santimano NM, Smith LA. Social norms information for alcohol misuse in university and college students. The Cochrane database of systematic reviews. 2015;1:Cd006748. pmid:25622306
  40. 40. Moreira MT, Smith LA, Foxcroft D. Social norms interventions to reduce alcohol misuse in university or college students. The Cochrane database of systematic reviews. 2009(3):Cd006748. pmid:19588402
  41. 41. Goslar M, Leibetseder M, Muench HM, Hofmann SG, Laireiter A-R. Efficacy of face-to-face versus self-guided treatments for disordered gambling: A meta-analysis. Journal of Behavioral Addictions. 2017;6:142–62. pmid:28662618
  42. 42. Peter SC, Brett EI, Suda MT, Leavens ELS, Miller MB, Leffingwell TR, et al. A Meta-analysis of Brief Personalized Feedback Interventions for Problematic Gambling. Journal of Gambling Studies. 2019;35:447–64. pmid:30610506
  43. 43. Quilty LC, Wardell JD, Thiruchselvam T, Keough MT, Hendershot CS. Brief interventions for problem gambling: A meta-analysis. PLOS ONE. 2019;14(4):e0214502. pmid:30995229
  44. 44. Saitz R. Screening and Brief Intervention for Unhealthy Drug Use: Little or No Efficacy. Frontiers in Psychiatry. 2014;5(121). pmid:25228887
  45. 45. Schmidt CS, Schulte B, Seo H-N, Kuhn S, O’Donnell A, Kriston L, et al. Meta-analysis on the effectiveness of alcohol screening with brief interventions for patients in emergency care settings. Addiction. 2016;111(5):783–94. pmid:26637990
  46. 46. Stockings E, Hall WD, Lynskey M, Morley KI, Reavley N, Strang J, et al. Prevention, early intervention, harm reduction, and treatment of substance use in young people. The lancet Psychiatry. 2016;3(3):280–96. pmid:26905481
  47. 47. Tanner-Smith EE, Lipsey MW. Brief Alcohol Interventions for Adolescents and Young Adults: A Systematic Review and Meta-Analysis. Journal of Substance Abuse Treatment. 2015;51:1–18. pmid:25300577
  48. 48. Young MM, Stevens A, Galipeau J, Pirie T, Garritty C, Singh K, et al. Effectiveness of brief interventions as part of the Screening, Brief Intervention and Referral to Treatment (SBIRT) model for reducing the nonmedical use of psychoactive substances: a systematic review. Systematic Reviews. 2014;3(1):50. pmid:24887418
  49. 49. Dotson KB, Dunn ME, Bowers CA. Stand-Alone Personalized Normative Feedback for College Student Drinkers: A Meta-Analytic Review, 2004 to 2014. PloS one. 2015;10(10):e0139518-e.
  50. 50. Smedslund G, Wollscheid S, Fang L, Nilsen W, Steiro A, Larun L. Effects of early, computerized brief interventions on risky alcohol use and risky cannabis use among young people. Campbell Systematic Reviews. 2017;13(1):1–192.
  51. 51. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):iii-x, 1–173. pmid:14499048
  52. 52. Rodda S, Saxton J., Booth N., Merkouris S.S., Dowling N. The efficacy of personalized normative feedback interventions for problem gambling, alcohol and illicit drug use: a systematic review and meta-analyses. PROSPERO International prospective register of systematic reviews. 2019.
  53. 53. Butler L, Correia C. Brief alcohol intervention with college student drinkers: face-to-face versus computerized feedback. Psychology of addictive behaviors. 2009, 23(1):163–7. pmid:19290702
  54. 54. Neighbors C, Lewis MA, LaBrie J, DiBello AM, Young CM, Rinker DV, et al. A multisite randomized trial of normative feedback for heavy drinking: Social comparison versus social comparison plus correction of normative misperceptions. Journal of Consulting & Clinical Psychology. 2016;84(3):238–47. pmid:26727407
  55. 55. Cunningham JA, Koski-Jannes A, Wild T, Cordingley J. Treating alcohol problems with self-help materials: A population study. Journal of Studies on Alcohol. 2002;63(6):649–54. pmid:12529064
  56. 56. Sterne J.A.C. SJ, Page M.J., Elbers R.G., Blencowe N.S., Boutron I., et al. ROB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019, 366. pmid:31462531
  57. 57. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed: Routledge; 1988.
  58. 58. Deeks JJ HJ, Altman DF (editors). Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions Chichester, UK: John Wiley & Sons; 2008.
  59. 59. Higgins JPT. Data extraction for continuous outcomes. Cochrane Handbook for Systematic Reviews of Interventions Version 510: The Cochrane Collaboration, 2011; 2011.
  60. 60. Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Medical Research Methodology. 2005;5(1):13.
  61. 61. Collins SE, Kirouac M, Lewis MA, Witkiewitz K, Carey KB. Randomized controlled trial of web-based decisional balance feedback and personalized normative feedback for college drinkers. Journal of Studies on Alcohol and Drugs. 2014;75(6):982–92. pmid:25343656
  62. 62. LaBrie JW, Lewis MA, Atkins DC, Neighbors C, Zheng C, Kenney SR, et al. RCT of web-based personalized normative feedback for college drinking prevention: Are typical student norms good enough? Journal of Consulting and Clinical Psychology. 2013;81(6):1074–86. pmid:23937346
  63. 63. Lewis M, Neighbors C. Optimizing personalized normative feedback: the use of gender-specific referents. Journal of studies on alcohol and drugs. 2007; 68(2):[228–237 pp.]. pmid:17286341
  64. 64. Lewis MA, Neighbors C, Oster-Aaland L, Kirkeby BS, Larimer ME. Indicated prevention for incoming freshmen: Personalized normative feedback and high-risk drinking. Addictive Behaviors. 2007;32(11):2495–508. pmid:17658695
  65. 65. Lewis MA, Patrick ME, Litt DM, Atkins DC, Kim T, Blayney JA, et al. Randomized controlled trial of a web-delivered personalized normative feedback intervention to reduce alcohol-related risky sexual behavior among college students. Journal of Consulting and Clinical Psychology. 2014;82(3):429–40. pmid:24491076
  66. 66. Neighbors C, Larimer M, Lewis M. Targeting misperceptions of descriptive drinking norms: efficacy of a computer-delivered personalized normative feedback intervention. Journal of consulting and clinical psychology. 2004; 72(3):[434–47 pp.]. pmid:15279527
  67. 67. Neighbors C, Lewis M, Bergstrom R, Larimer M. Being controlled by normative influences: self-determination as a moderator of a normative feedback alcohol intervention. Health psychology. 2006; 25(5):[571–9 pp.]. pmid:17014274
  68. 68. Neighbors C, Lewis MA, Atkins DC, Jensen MM, Walter T, Fossos N, et al. Efficacy of web-based personalized normative feedback: A two-year randomized controlled trial. Journal of Consulting and Clinical Psychology. 2010;78(6):898–911. pmid:20873892
  69. 69. Neighbors C, DiBello AM, Young CM, Steers MLN, Rinker DV, Rodriguez LM, et al. Personalized normative feedback for heavy drinking: An application of deviance regulation theory. Behaviour Research and Therapy. 2018. pmid:30580836
  70. 70. Young CM, Neighbors C. Incorporating Writing into a Personalized Normative Feedback Intervention to Reduce Problem Drinking Among College Students. Alcohol Clin Exp Res. 2019;43(5):916–26. pmid:30817010
  71. 71. Neighbors C, Rodriguez LM, Rinker DV, Gonzales RG, Agana M, Tackett JL, et al. Efficacy of personalized normative feedback as a brief intervention for college student gambling: A randomized controlled trial. Journal of Consulting and Clinical Psychology. 2015;83(3):500–11. pmid:26009785
  72. 72. Andersson C. Comparison of WEB and Interactive Voice Response (IVR) methods for delivering brief alcohol interventions to hazardous-drinking university students: A randomized controlled trial. European Addiction Research. 2015;21(5):240–52. pmid:25967070
  73. 73. Baldin YC, Sanudo A, Sanchez ZM. Effectiveness of a web-based intervention in reducing binge drinking among nightclub patrons. Revista de Saude Publica. 2018;52:2. pmid:29364357
  74. 74. Bendtsen P, Bendtsen M, Karlsson N, White IR, McCambridge J. Online alcohol assessment and feedback for hazardous and harmful drinkers: Findings from the AMADEUS-2 randomized controlled trial of routine practice in Swedish universities. Journal of Medical Internet Research. 2015;17(7):No Pagination Specified. pmid:26159179
  75. 75. Bertholet N, Cunningham JA, Faouzi M, Gaume J, Gmel G, Burnand B, et al. Internet-based brief intervention for young men with unhealthy alcohol use: a randomized controlled trial in a general population sample. Addiction. 2015;110(11):1735–43. pmid:26173842
  76. 76. Bertholet N, Godinho A, Cunningham JA. Smartphone application for unhealthy alcohol use: Pilot randomized controlled trial in the general population. Drug and Alcohol Dependence. 2019;195:101–5. pmid:30611977
  77. 77. Cunningham JA, Hendershot CS, Murphy M, Neighbors C. Pragmatic randomized controlled trial of providing access to a brief personalized alcohol feedback intervention in university students. Addiction Science & Clinical Practice. 2012;7:21.
  78. 78. Johnson NA, Kypri K, Saunders JB, Saitz R, Attia J, Latter J, et al. Effect of electronic screening and brief intervention on hazardous or harmful drinking among adults in the hospital outpatient setting: A randomized, double-blind, controlled trial. Drug & Alcohol Dependence. 2018;191:78–85. pmid:30096637
  79. 79. Kypri K, Hallett J, Howat P, McManus A, Maycock B, Bowe S, et al. Randomized controlled trial of proactive web-based alcohol screening and brief intervention for university students. Archives of Internal Medicine. 2009;169(16):1508–14. pmid:19752409
  80. 80. Kypri K, McCambridge J, Vater T, Bowe SJ, Saunders JB, Cunningham JA, et al. Web-based alcohol intervention for Maori university students: Double-blind, multi-site randomized controlled trial. Addiction. 2013;108(2):331–8. pmid:22925046
  81. 81. Kypri K, Vater T, Bowe SJ, Saunders JB, Cunningham JA, Horton NJ, et al. Web-based alcohol screening and brief intervention for university students: A randomized trial. JAMA: Journal of the American Medical Association. 2014;311(12):1218–24. pmid:24668103
  82. 82. Murphy J, Dennhardt A, Yurasek A, Skidmore J, Martens M, MacKillop J, et al. Behavioral economic predictors of brief alcohol intervention outcomes. Journal of consulting and clinical psychology. 2015; 83(6):[1033–43 pp.] pmid:26167945
  83. 83. Ridout B, Campbell A. Using Facebook to deliver a social norm intervention to reduce problem drinking at university. Drug and alcohol review. 2014; 33(6):[667–73 pp.]. pmid:24689339
  84. 84. Wagener TL, Leffingwell TR, Mignogna J, Mignogna MR, Weaver CC, Cooney NJ, et al. Randomized trial comparing computer-delivered and face-to-face personalized feedback interventions for high-risk drinking among college students. Journal of Substance Abuse Treatment. 2012;43(2):260–7. pmid:22197301
  85. 85. Walters ST, Bennett ME, Miller JH. Reducing alcohol use in college students: A controlled trial of two brief interventions. Journal of Drug Education. 2000;30(3):361–72. pmid:11092154
  86. 86. Walters ST, Vader AM, Harris TR, Field CA, Jouriles EN. Dismantling motivational interviewing and feedback for college drinkers: a randomized clinical trial. Journal of Consulting & Clinical Psychology. 2009;77(1):64–73.
  87. 87. Luquiens A, Tanguy M-L, Lagadec M, Benyamina A, Aubin H-J, Reynaud M. The efficacy of three modalities of Internet-based psychotherapy for non-treatment-seeking online problem gamblers: A randomized controlled trial. Journal of Medical Internet Research. 2016;18(2):1–13. pmid:26878894
  88. 88. Martens MP, Arterberry BJ, Takamatsu SK, Masters J, Dude K. The efficacy of a personalized feedback-only intervention for at-risk college gamblers. Journal of Consulting and Clinical Psychology. 2015;83(3):494–9. pmid:25664642
  89. 89. Elliott JC, Carey KB, Vanable PA. A preliminary evaluation of a web-based intervention for college marijuana use. Psychology of Addictive Behaviors. 2014;28(1):288–93. pmid:24731118
  90. 90. Lee C, Neighbors C, Kilmer J, Larimer M. A brief, web-based personalized feedback selective intervention for college student marijuana use: a randomized clinical trial. Psychology of addictive behaviors. 2010; 24(2):[265–73 pp.]. pmid:20565152
  91. 91. Palfai TP, Saitz R, Winter M, Brown TA, Kypri K, Goodness TM, et al. Web-based screening and brief intervention for student marijuana use in a university health center: pilot study to examine the implementation of eCHECKUP TO GO in different contexts. Addictive Behaviors. 2014;39(9):1346–52. pmid:24845164
  92. 92. Fu R, Holmer HK. Change score or follow-up score? Choice of mean difference estimates could impact meta-analysis conclusions. Journal of clinical epidemiology. 2016;76:108–17. pmid:26931293
  93. 93. Laine C, Horton R, DeAngelis CD, Drazen JM, Frizelle FA, Godlee F, et al. Clinical Trial Registration—Looking Back and Moving Ahead. New England Journal of Medicine. 2007;356(26):2734–6.
  94. 94. Crawford MJ, Sanatinia R, Barrett B, Byford S, Dean M, Green J, et al. The clinical effectiveness and cost-effectiveness of brief intervention for excessive alcohol consumption among people attending sexual health clinics: a randomised controlled trial (SHEAR). Health Technol Assess. 2014;18(30):1–48. pmid:24813652
  95. 95. Wutzke SE, Shiell A, Gomel MK, Conigrave KM. Cost effectiveness of brief interventions for reducing alcohol consumption. Social science & medicine (1982). 2001;52(6):863–70. pmid:11234861
  96. 96. Angus C, Thomas C, Anderson P, Meier PS, Brennan A. Estimating the cost-effectiveness of brief interventions for heavy drinking in primary health care across Europe. European Journal of Public Health. 2016;27(2):345–51.
  97. 97. Bertholet N, Gaume J, Faouzi M, Gmel G, Daeppen JB. Predictive value of readiness, importance, and confidence in ability to change drinking and smoking. BMC Public Health. 2012 Dec 1;12(1):708.
  98. 98. Rodda SN, Lubman DI, Iyer R, Gao CX, Dowling NA. Subtyping based on readiness and confidence: The identification of help‐seeking profiles for gamblers accessing web‐based counselling. Addiction. 2015 Mar;110(3):494–501. pmid:25393315
  99. 99. Abar B., Baumann B.M., Rosenbaum C., Boyer E., Boudreaux E. D. Readiness to change alcohol and illicit drug use among a sample of emergency department patients. J Subst Use 2012; 17: 260–8.