Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Does attention bias modification reduce anxiety in socially anxious college students? An experimental study of potential moderators and considerations for implementation

Abstract

According to cognitive models, preferential attention to social threat contributes to maintenance of social anxiety. Socially anxious individuals are known to show attention biases to threatening stimuli, although there is inconsistency in the literature with regards to the type of attentional biases they present. This study examines the effect of attention bias modification (ABM) for social anxiety in non-treatment-seeking college students meeting social anxiety disorder criteria, taking into consideration previous mixed results regarding its effectiveness. Attention bias levels and types (i.e. vigilance vs avoidance) at baseline were examined and considered as potential moderators of ABM effects. Sixty-eight socially anxious individuals were randomly allocated to ABM vs placebo groups. A structured interview and self-report assessment were completed at pre-treatment and post-treatment. Results showed half of the participants presented few attention biases at baseline, and the rest presented either vigilance or avoidance. Participants with low attention biases scored higher in social anxiety than those showing avoidance and there was no difference between those showing vigilance vs avoidance. No significant effects from pre to post treatment were observed in attention biases, self-report or structured interview of anxiety in the ABM group. Baseline attention biases did not moderate these effects. Results are discussed with regards to implications for future research towards the creation of more effective protocols, based on the needs of heterogeneous social anxiety sub-groups.

Introduction

Social anxiety disorder (SAD) is highly prevalent with rates estimated around 5% of the general population [1]. It is considered as the second most common psychological disorder in the general US [2] and European [3] population, and has been related to economic costs, work absenteeism and high health care utilization [4]. According to DSM-5, individuals with SAD can be divided into those demonstrating primarily performance or public speaking fears and those with general social fears, who are the majority of cases [5,6]. Individuals with either sub-type have similar age of onset, family history and sociodemographic characteristics [2]. SAD is typically chronic and disabling [7], a risk factor for the development of additional psychopathology, and shares high comorbidity with other anxiety disorders and depression [8,9].

Cognitive-behavioral therapy is the most effective treatment for adults with social anxiety [10]. However, a high percentage of them, that could reach over 80%, do not seek or receive treatment [1,11]. In addition to other documented reasons, such as distance from psychological services, or unwillingness to share personal information with strangers, individuals with SAD avoid treatment because of fears of negative evaluation from the therapist [12]. These considerations create the need for novel treatment approaches that may be more widely acceptable, either as stand alone, or as complements to cognitive-behavioral therapy [13].

Attention bias modification (ABM) is a promising intervention which could potentially help overcome these obstacles. This is a computerized intervention, eliminating concerns about face-to-face interactions, at least during early stages of treatment when anxiety is highest. It is suggested to be more easily accessible, as it requires no or limited therapist involvement and can be completed quickly. ABM aims to reduce anxiety by changing attentional processes: It trains anxious individuals to focus their attention on emotionally neutral or pleasant stimuli, based on the assumption that attention biases towards threatening stimuli play a role in the etiology and maintenance of anxiety disorders [14].

This assumption is supported by different theoretical models [15,16], which have been used to support the idea that ABM can be a useful stand-alone component in the overall treatment of SAD, or an auxiliary to highly effective cognitive-behavioral interventions. The Cognitive-Behavioral model of anxiety [16] notes that audience feedback e.g. a perceived angry face that socially anxious individuals pay preferential attention to, reinforces the negative image they hold of how they may look to others, creating more anxiety. Therefore, allocating less attention to such cues, and increased attention to neutral or positive audience feedback may be crucial in changing negative self-images. At the same time, attentional bias is believed to moderate the impact of negative life events on stress and anxiety [17,18]. Anxious individuals appear to allocate preferential attention to even milder threat conditions compared to non-anxious individuals, and this may prolong and perpetuate stress, making the world appear less safe, with ramifications for health and wellbeing. For these reasons, further investment on procedures that may modify these biases is worthwhile, as a means to complement existing effective treatments.

In the case of social anxiety, ABM trains individuals to focus attention less on angry or other threatening faces [13,19] and more on neutral or happy faces instead. However, compared to the large number of studies supporting the effectiveness of ABM for other anxiety disorders and trait anxiety [2022], a smaller number of studies found supportive evidence when it comes to social anxiety, [e.g. 13,19,23–27], with some studies finding non-significant changes from pre to post-treatment on attention biases and SAD symptoms, suggesting no or little effectiveness [2832; see metanalysis in 33]. It is possible that reasons for these mixed results pertain to the nature of attention biases associated with social anxiety.

Although the association between anxiety and attentional biases towards threat have been frequently reported in the past [34], more recent evidence casts doubt on their robustness, [35], at least when it comes to the dot-probe task. Results are somewhat more mixed when it comes to social anxiety, with some studies finding no evidence of attention biases [36,37], but several studies showing the expected pattern of greater attention towards social threatening information [38]. However, in addition to the typical hyper-vigilance to threat found in other anxiety conditions, studies find additional attention bias patterns in SAD, including difficulty in disengaging from threat, and threat avoidance, suggesting heterogeneity of bias in this anxiety disorder [36,39]. According to more recent theorizing, this heterogeneity, but also the absence of more consistent attention bias findings, may reflect individual differences that remain stable across situations, or a dynamic shift in attention biases within the same individual, as the emotional situation evolves [40,41]. To some degree, reliability issues of the way attention bias is calculated within the dominantly used dot-probe task, may account for these inconsistencies [40,41]. According to a recent review [37], additional moderators of the type of attention bias shown by socially anxious individuals include the type and timeframe of stimulus presentation, as well as social anxiety level.

More specifically, in addition to hyper-vigilance toward socially threatening information, including signs of disapproval from others [13,39], socially anxious individuals also have difficulties with disengaging their attention from threat [42]. This process refers to a delayed withdrawal of attention from a stimulus because of its ability to hold attention [43]. In fact, it was suggested that this may be the only attentional bias relevant to SAD [44]. In addition, there is evidence that socially anxious individuals show attentional avoidance of threatening stimuli, a pattern that is more uniquely related to SAD than other anxiety disorders, [37], and it is dominant in several theoretical models of SAD [15,4548].

Given these considerations, it is useful to further examine the role of heterogeneity in attention biases in social anxiety, in the mixed results regarding the effectiveness of ABM. Focusing to start with on heterogeneity at the level of individuals, it is possible, that traditional ABM works only for the subset of socially anxious participants with strong attentional vigilance [29,49]. Indeed, there is evidence that participants with limited vigilance toward threat do not show substantial gains [50]. These individual differences in the degree attentional bias towards threat, as well as the existence of other types of attention bias as part of the psychopathology of SAD, on which ABM may be less effective, have been largely ignored in most studies [see 50 and 51 for exceptions] creating the need for additional research.

In addition to the above, the mixed findings regarding the effectiveness of ABM for SAD may be attributed to methodological differences among studies. Because of the promising results for other anxiety conditions, the majority of ABM studies recruited treatment-seeking participants, and presented ABM as an effective new treatment [e.g. 24,26], which may have created demand effects on outcomes. Strong expectancy effects may overshadow specific treatment effects, leading to similar reduction in reported anxiety, in both control and intervention group. Only one study [24] showing significant effectiveness of ABM recruited non-treatment seeking participants (college students with high levels of social anxiety). It appears that more experimental studies on non-treatment seeking samples are required to establish the effects of ABM on various components of attention biases, before this intervention can be presented as an effective part of treatments for SAD.

This study aims to examine the effects of ABM, in comparison to a placebo condition, on social anxiety symptoms and pre-intervention heterogeneity in attention biases, in non-treatment seeking students with high levels of social anxiety. The widely used dot-probe task was used to assess attention biases and deliver ABM [18,52]. Because of the possibility that differences in levels of attention biases towards, vs avoidance of social threat, shown by each individual may moderate effectiveness, attention bias levels, assessed at baseline, were examined. Specifically, baseline level of vigilance vs avoidance was assessed, using widely available calculation methods [e.g., 14,52,53] for purposes of comparability to the studies showing mixed effectiveness results. In order to reduce expectancy effects of presenting this as an intervention for SAD, any terminology including ‘intervention’, ‘therapy’ etc. was avoided. Instead, participants were told that they would participate in an experiment to see if an attentional task aiming to change attentional processes can produce changes in their anxiety levels. Further examining the effectiveness of ABM in well-controlled experimental studies and addressing the role of baseline attention bias can have important implications for clinical research, towards the design of more effective ABM protocols that are specific to the mechanisms sustaining anxiety among socially anxious individuals.

Taking extant research into consideration, it was expected that: 1) heterogeneity at the individual level could be observed in attention biases at baseline, so that some participants would show more vigilance, and others would show more avoidance of threat; different levels of social anxiety may be associated with different degrees of vigilance vs avoidance, although prior hypotheses cannot be set due to the mixed findings, 2) the ABM group, in comparison to the placebo control group at post-treatment would present: a) reduction in vigilance to threat as assessed by reaction time (RT) to the dot-probe, b) reduction in SAD symptoms, as measured by structured interview and self-report, 3) baseline levels of attention bias would moderate ABM effects, with decreased attention to threat and reduced SAD symptoms [self-report and interview] found to a greater degree among participants with more vigilance to threat at baseline, compared to those with greater avoidance.

Materials and method

Participants

The sample was comprised of undergraduate students who took part in the experiment for extra credit or participation in a lottery for a tablet. Participants who scored above the clinical cut-off of 28 on the Difference subscale of the Social Phobia and Anxiety Inventory–23 (SPAI-23), or scored one standard deviation above the mean on the social anxiety subscale (M = 21, SD = 12) [54] and agreed to participate in the study, were interviewed using the Anxiety Disorders Interview Schedule adult version (ADIS-IV) [54] to confirm that their anxiety was in the clinical range. Only those who met ADIS-IV SAD criteria were included, to ensure high levels of symptoms. Exclusion criteria were the presence of current: 1) suicidal intent, 2) substance abuse, 3) meeting diagnostic criteria for post-traumatic stress disorder, obsessive-compulsive disorder, 4) past schizophrenia, bipolar disorder, organic mental disorder, and 5) any concurrent psychotherapy, 6) changes in medication during the 12 weeks prior to study, and 7) CBT during the 6 months before the study. The first four criteria were assessed with ADIS-IV and the last three criteria were assessed through a short interview developed for this study.

In total 68 participants were selected from the screening sample for their high SPAI-23 scores. Eight of them were removed for not meeting ADIS-IV criteria for SAD. Finally, the ABM group consisted of 32 participants and the placebo group of 28 participants (total N = 60), to which they were double-blindly assigned. Participants met criteria for SAD (20 participants) or the SAD specifier in DSM-5, i.e. performance anxiety (40 participants) on the basis of ADIS-IV.

Procedure

The screening study was announced in classes, along with an invitation to the experimental phase. Additional students were recruited during mental health screening days of the University Mental Health Center, where students filled out the SPAI-23 as part of screening and were informed about the study. Students meeting SPAI-23 criteria from either sample were invited to the study and those who consented were interviewed as above. Interviewers in all cases were trained, doctoral level clinical psychology students. At the end of the data collection, every interview was checked between interviewers and the primary researcher to confirm that inclusion and exclusion criteria were met. Informed consent was obtained for all study stages, which were approved by the National Bioethics Committee.

Upon arrival to the lab, participants were informed that they would perform an attention task. They were also told that the study would assess if the task can result in changes of their attentional processes (where they focus their attention) depending on their group allocation, and that it would assess if these changes in attention focus can result in changes of their anxiety levels, which they knew were considered high relative to the general population from the discussion of their ADIS-IV results. At the end of the experiment, participants were asked about their assumptions with regards to the training that they received, in order to assess their blindness to the ABM vs placebo condition. Most of them (90%) selected the ‘do not know’ option, suggesting blindness to group assignment.

To complete the study, participants came to the lab for 8 sessions: The first session (lasting approximately 1.5 hours) included: interview with the ADIS-IV; participants who did not meet clinical levels (8 participants) were provided an explanation with regards to the study’s criteria and were dismissed. Those meeting inclusion criteria completed the baseline attention bias assessment through the dot-probe task. Next, they completed a package of questionnaires assessing their baseline anxiety levels (described below). Lastly, they received either the active ABM (training away from threat) or placebo training depending on their group assignment.

Over the next 3-weeks, participants received the ABM or placebo training, depending on their group, a total of 6 times, twice a week, preceded by computerized instructions. Sessions lasted about 7 minutes. During the last (8th) session, participants received again the ABM or placebo training as usual; next they completed the self-report questionnaires (post-intervention assessment) and their attentional biases were re-assessed using the dot-probe task similar to baseline. Lastly, participants were re-assessed with the ADIS-IV. The last session lasted approximately 1.5 hours.

Measures

A battery of self-reported and clinician-rated measures of anxiety and dysfunction were the dependent variables at pre and post intervention.

Anxiety Disorders Interview Schedule adult version (ADIS-IV) [55], is a structured diagnostic interview assessing anxiety disorders according to DSM–IV. It also assesses/screens for depression, somatoform disorders and substance abuse, psychotic symptoms and family psychiatric history. This interview shows excellent reliability for social anxiety [56]. It was administered to confirm that participants had significant (clinical) levels of social anxiety and to rule out other diagnoses according to the study’s exclusion criteria.

Social Phobia and Anxiety Inventory-23 (SPAI-23) [57], is a shortened version of the SPAI [58] measuring symptoms of social anxiety. It consists of two sub-scales: social phobia (16 items) and agoraphobia (7 items). Each item is rated on a 5-point scale from 0 (never) to 4 (always). The Agoraphobia sub-scale can be subtracted from the social phobia scale, producing a Difference score. A Difference score ≥28 is indicative of clinical levels of SAD. The two scales have good internal consistency and adequate test re-test reliability over 5 1/2 weeks (r = 0.72; social phobia sub-scale, α = .95; agoraphobia sub-scale, α = .85) [57,58], and good internal consistency in Greek-Cypriot adolescents (social phobia sub-scale, α = .95; agoraphobia sub-scale, α = .85) [59]. For the current study, subscale internal consistency was as follows. Pre-treatment: social phobia, α = .90; agoraphobia, α = .85. Post-treatment: social phobia, α = .93; agoraphobia, α = .86).

Liebowitz Social Anxiety Scale Test (LSAS) [60] is a 24 item self-report measure rated on a 4-point scale. It assesses fear and avoidance in social and performance situations during the past week. It yields an overall total score and six sub-scale scores: total fear, fear of social interactions, fear of performance situations, total avoidance, avoidance of social interaction, avoidance of performance situations. In the current study, only the total score was used. Scores of 55–65 indicate moderate, 65–80 marked, 80–95 severe social anxiety and score greater than 95 is indicative of marked social phobia. The LSAS has overall good psychometric properties (test–retest reliability, convergent and discriminant validity), with internal consistency α = 0.95, and is sensitive to treatment change [61]. It also showed very good internal consistency, and replicated factor structure in a Greek-Cypriot student sample collected by the researcher (unpublished data; Cronbach’s α = 0.94). For this study, pre-treatment α = 0.95, post-treatment α = 0.96.

The dot probe attention task

The dot-probe task was adopted from the Tel Aviv University/ National Institute of Mental Health (TAU/NIMH) study. Specifically, the task includes face photographs of 20 individuals (10 male, 10 female) taken from the NimStim stimulus set [62], with the exception of one female picture taken from the Matsumoto and Ekman set [63]. Emotionally valent (anger) and neutral faces were selected. Threatening facial expressions are considered an ancient signal for submissiveness in evolutionary history [64], and are therefore fearful, especially to individuals with social anxiety. Angry faces are often used in studies of SAD [for such studies using the TAU/NIMH protocol see 23,25,65]. During the task, pairs of stimuli, angry-neutral faces or neutral-neutral faces of the same actor are presented vertically in the centre of the screen. Two different sets of pictures (A or B) are used, one for assessment and the other during training, counterbalanced between subjects. Faces are presented against a plain black background as in the Matsumoto and Ekman set and each picture is 45mm width x 34mm height. Each face is distanced from its pair by14mm. Photographs are centered vertically, with equal distance to the top and bottom of the fixation cross. The top face is 20mm from the top edge of screen. Photographs are surrounded by a single 58mm wide x 94mm tall white rectangle, which shows the area of the screen that the participant needs to focus on. Stimulus presentation was done with E-Prime 2 (PST, Pittsburgh, PA). Each participant was tested individually at 70cm distance from a 15-inch computer screen.

Attention bias assessment.

Attention bias assessment using the dot-probe task was done at pre and post-treatment. The task consists of 120 trials (80 angry-neutral and 40 neutral-neutral). Each trial begins with a fixation cross (500 ms; white cross 1*1 cm at the centre of the screen), on which the participants are instructed to focus their gaze. Then, a face pair display of 500ms duration follows. Next, a small visual probe (< or >) appears at the place of one of the two faces. Participants must determine which symbol appeared by clicking right or left on the mouse, as quickly as possible. The target probe remains on the screen until there is a response, which starts a new trial, after an inter-trial interval of 500ms. Presentation of angry-face location, probe type, location and actor are fully counterbalanced. Less than 70% accuracy determines experiment abortion. In this case, a warning gives the opportunity for the experiment to start from the beginning. No participant had to start over in this study. Task completion takes about 5 minutes.

Although the optimal way to analyze dot-probe data continues to be discussed, in the current study, for purposes of comparability to previous ABM work on SAD, we relied on the most widely used method to assess attention biases, i.e. the bias score [e.g., 14,48,49], despite concerns expressed for its reliability [40,41]. The reaction time on stimuli which replace the threatening face when it is presented with a neutral (i.e., congruent trials) is subtracted from reaction time on stimuli which replace non-threat (neutral face) when it is presented with threat (angry face) (i.e., incongruent trials). A positive number reflects faster identification of threat. However, a negative number reflects quick detection of neutral stimuli and is interpreted as avoidance of threat [23,25,34,66].

ABM and placebo protocols.

ABM and placebo protocols consist of 160 trials (120 angry-neutral and 40 neutral-neutral presentations) of a similar dot-probe task. In the placebo condition, angry-face location, probe location, and actor are fully counterbalanced in presentation. The only aspect that changes in the ABM protocol in relation to placebo is that in all angry-neutral pairs the probe is presented only after neutral faces, providing training away from threat. Additionally, probe type (< or >) is not factorially counterbalanced but there is equal probability of presentation for each of the following: angry-face location, probe location, or actor. Every 40 trials there is a short break. If accuracy is < 70%, a warning is presented during the break, giving an opportunity for the experimenter to remind the participant not to compromise accuracy. The protocol takes about 7 minutes for completion.

Reliability of trials in the current sample was good at pre-treatment (baseline assessment): neutral-neutral trials: Cronbach’s alpha = 0.94 and split-half reliability Spearman r = 0.90, angry-neutral congruent trials: Cronbach’s alpha = 0.94 and split-half reliability Spearman r = 0.85, angry-neutral incongruent trials: Cronbach’s alpha = 0.94 and split-half reliability Spearman r = 0.86 as well as at post-treatment: neutral-neutral trials Cronbach’s alpha = 0.83 and split-half reliability Spearman r = 0.77, angry-neutral congruent trials: Cronbach’s alpha = 0.81 and split-half reliability Spearman r = 0.75, angry-neutral incongruent trials: Cronbach’s alpha = 0.85 and split-half reliability Spearman r = 0.80. However, test re-test reliability was low for neutral-neutral trials r (60) = 0.40, p < 0.01, angry-neutral congruent trials r (60) = 0.33, p < 0.01, and angry-neutral incongruent trials r (60) = 0.37, p < 0.01.

Additionally, split half reliability of bias score was low r = -.22 at pre-treatment and post-treatment r = .08. Similarly, there was low test re-test reliability of bias score r (60) = -0.11, p = 0.70. The same levels of reliability have been found in studies [67,68] in undergraduate students and studies [27,69] in socially anxious individuals. There is an exception study [70] in a civilian population after war, which found high split-half reliability of bias score. Nonetheless, in an effort to be comparable with previous studies [23,25] with the same version of the dot-probe task, as well as all the studies assessing ABM effectiveness [exemption is 27], the traditional way of calculation was used for the current study.

Data analysis

Initially, frequency analysis and one-way ANOVA was done to examine the attention bias levels of participants and differences in their social anxiety levels respectively. T-test on assessed characteristics was used to verify participants’ random assignment to group. To be consistent with other studies [e.g., 23] Generalized Estimating Equations (GEE) [71,72], was used in to examine ABM effects. This analysis accounts for correlated repeated measurements and accommodates missing data under the missing- at-random assumption. Specifically, it computes estimated marginal means, thus is considered as an intention-to-treat analysis including data from all randomized participants who gave at least one data point. To represent within-subject dependencies in the models, we specified an unstructured covariance matrix. Effect of changes on attention bias score and social anxiety levels were estimated using models containing main effects of group and time, and their interaction. The moderating role of initial attention bias level (continuous variable) was assessed using moderation analysis in the PROCESS macro of SPSS (SPSS Inc., Chicago, Illinois, USA), with Group (ABM, placebo) as predictor variable and difference score on social anxiety measures between pre- and post-treatment as outcome variables [as in 25]. Finally, based on [73] in order to achieve power of 0.80 with α = 0.05, two tailed, and obtain the smallest effect found in previous studies (d = 0.58) [26] a sample size of 26 participants was indicated to capture changes in pre to post-treatment effects.

Results

Descriptive analyses

Frequencies showed that most of the participants, N = 33, presented limited attention bias (i.e. attention bias score close to zero (M = 1.73, SD = 7.30). Fifteen of them presented mostly Avoidance (M = -26.21, SD = 15.82) and 18 of them presented mostly Vigilance (M = 28.54, SD = 14.55) (1/2 SD below or above the Mean). There was a statistically significant difference on SPAI-23 difference score between groups formed on the basis of their attention biases, one way-ANOVA, F (2, 58) = 3.26, p<0.05. Specifically, participants with attention bias scores close to zero presented the highest social anxiety levels (M = 34.22, SD = 8.71), followed from those with vigilance (M = 32.65, SD = 6.14) and those with avoidance (M = 27.73, SD = 8.28). Post-hoc tests showed that this difference was significant only between those with limited attention bias and avoidance, p < 0.05 and not between those with vigilance and avoidance p = 0.26. Differences on the LSAS total were not significant, F (2, 51) = 0.41, p<0.66.

Intervention Group equivalence

Preliminary analyses indicated no Group (ABM, placebo) differences in baseline attention bias level t(58) = 1.33, p = 0.19, SPAI-23 difference score t(57) = -1.75, p = 0.09, or LSAS total t(50) = -0.80, p = 0.43 (see Table 1).

Pre to post treatment effects

GEE showed no effect of Time, Wald χ2(1) = 0.02, p = 0.88 and no main effect of Group, Wald χ2(1) = 1.22, p = 0.27 on attentional biases, and no statistically significant interaction between Group and Time, Wald χ2(1) = 0.93, p = 0.34.

Results also showed no significant changes in social anxiety from pre to post treatment on any of the self-report measures: Specifically, there were no significant effects of Time, Wald χ2(1) = 0.49, p = 0.48, Group, Wald χ2(1) = 2.51, p = 0.11, or Group x Time interaction, Wald χ2(1) = 0.29, p = 0.59 on SPAI-23 difference scores. Similarly, there were no significant effects of Time, Wald χ2(1) = 1.81, p = 0.18, Group, Wald χ2(1) = 0.37, p = 0.54, or Group x Time, Wald χ2(1) = 0.96, p = 0.33 on LSAS total.

Finally, there was no significant change in ADIS-IV diagnostic status from pre to post treatment, as all participants maintained the same SAD status (general, performance).

Moderating role of baseline attention biases

The moderation analysis using the command process.spd, with Group as the predictor variable (ABM, placebo), difference SPAI-23 score between pre-treatment and post-treatment as the outcome, and baseline attention bias level (continuous) as the moderator variable, showed that the model was not statistically significant F (3, 45) = 0.88, p = 0.46. Similarly, a non-statistically significant model was obtained when moderation was examined for LSAS difference score from pre- to post-treatment, as the outcome variable, F(3, 45) = 0.88, p = 0.46.

Discussion

Previous studies show mixed results with regards to the effectiveness of ABM for social anxiety, an effect that needs further clarification, especially in light of evidence that not all socially anxious individuals show the expected levels of hyper-vigilance to threat, as demonstrated in previous studies [38,41], as well as by the present data. Although heterogeneity in the type of attention bias associated with SAD is still the subject of discussion, with some authors strongly suggesting that it may occur dynamically within individuals, over the duration of the task, rather than at an individual difference level [40,41], for the current study we focused on baseline attention bias as an individual difference between participants, in order to examine whether this moderates ABM effects, in an effort to be comparable with almost all the studies measuring ABM effectiveness [exemption is 27]. To the degree that individuals may differ in the degree to which they are characterized by vigilance vs avoidance, ABM may prove more helpful for some and not for other socially anxious individuals [33]. Consideration of the effects of baseline tendencies towards one rather than the other type of attention bias can lead to more effective protocols. Studying such effects in a controlled experimental setting, with individuals who are not treatment seekers can allow for the control of more factors than in a clinical trial, that may contribute towards identifying the mechanisms of effects.

We found that almost half of the participants presented with limited attention bias to threatening faces, with scores close to zero when taking their average bias score across trials, in comparison to the neutral stimuli; of the remaining participants almost half presented vigilance and half avoidance, indicating that it is only a subset who show clearly and uniquely bias towards threatening faces. As the participants with close to zero scores seemed to have the highest levels of social anxiety, it should be noted that they may have presented equivalent degrees of both vigilance and avoidance (i.e. as per dominant vigilance-avoidance models [48,49,74], which cancel each other out in the typical way attention biases are calculated. This tentative interpretation is consistent with previous arguments of dynamic, within individual variation in attention bias types, that may occur at the trial-by-trial level, an effect which is disguised when taking the average over large blocks of trials [40,41], and which requires serious consideration in future ABM work.

Notably, and in accord with a substantial subset of similar studies, ABM was not effective in changing threat bias and social anxiety levels of socially anxious college students. Specifically, no attentional biases (threat bias) changes were found from pre- to post-treatment, unlike several previous studies, and as a result there were no significant changes from pre- to post-treatment with regards to social anxiety status on the ADIS or self-reported anxiety. Against initial hypotheses baseline attention bias level did not moderate ABM effects. It is possible, that future work that takes into account the dynamic changes in attention biases within individuals can better clarify the patterns of bias on which ABM is most effective, and the relation of these patterns to experienced and reported social anxiety symptoms.

This study is inconsistent with limited evidence of effectiveness in a college sample [21]. However, the current study is not the only one finding no effectiveness of ABM for SAD (and anxiety in general), and negative findings like this may be important in delineating the circumstances under which the treatment is indeed effective. Furthermore, most studies showing treatment gains with regards to social anxiety levels, observed these on clinician administered measures [23,26] or state anxiety levels measured by behavioral assessment and self-report [19] and not on self-report measures of social anxiety, or clinical interviews at post-treatment. However, even when self-report measures of social anxiety showed changes from pre to post-treatment, social anxiety levels were still above the clinical cut-off at post-treatment [13]. Although more studies would be needed to make this claim, it is possible that removing the effect of treatment-seeking and the potential expectancy effect of believing that one receives an effective treatment, may be associated with null or reduced effectiveness. Our negative findings are in accordance with studies that found no differences between the ABM and placebo group on self-report, clinician interview and behavioral measures of social anxiety [30], and with studies that used a different delivery method, delivering ABM online, which may be less comparable to the present study [e.g. 28,29,31,75]. In all, this study builds on an accumulating body of evidence that ABM may be less effective for SAD than other types of anxiety, for which much stronger effects were obtained. However, results prompt the need for further research on potential moderators of effectiveness, including via a more specific delineation of attention bias heterogeneity.

With regards to effectiveness of ABM in changing attention biases from pre-treatment, out null results are more difficult to explain, given the positive results in many previous studies [76]. Although our study is not the only one reporting no attention bias changes from pre- to post-treatment [e.g. 23], the reasons for the negative effects deserve further examination. It has been argued that participants with limited attentional biases toward threat do not present with gains in comparison to the control group [50], suggesting that attention biases may not be the mechanism sustaining anxiety in this case, and the need for alternative treatment options. However, we had also observed that individuals showing limited attention bias at baseline, in fact, scored higher in social anxiety, which may indicate that they should be the ones demonstrating greater gains. As we hypothesized, heterogeneity of attention biases in social anxiety and the type of bias favored by each participant may have affected present results, although moderation was not supported. This may be because attention bias should be assessed using a dynamic approach at the trial level, an approach that would yield rich data to help identify who, and under what circumstances responds best to ABM [41].

Results of the present study should be seen in light of some limitations. First, with regards to examining the effects of baseline attention biases, a larger sample may have been necessary to detect significant interactions, therefore, the absence of effects here should be considered as preliminary. The sample was comprised of participants with general social fears and performance anxiety; the mixed nature of the group may have affected the effect of ABM, which may be more potent in the more severe form of social anxiety, the general subtype [2]. It should be noted that an exploratory examination of ABM effectiveness separately for the two groups, not reported here for brevity, showed no difference between them. Additionally, the pre-selection of participants may need to be considered crucial at the examination of AB at the individual difference level, so that larger and equal numbers of participants primarily showing vigilance vs avoidance take part, in order to ensure adequate power to detect moderation effects.

A more general issue pertains to the use of the dot-probe task, which, despite its widespread use, has been questioned in recent studies for its reliability [e.g. 40,41] as well as it has shown low reliability in the current study. Reliability issues may account for mixed findings regarding attentional bias and anxiety reduction, and the null finding for the current study. Adopting newer, and potentially more reliable ways of calculating attention biases on the dot-probe task [41,77], over the duration of the task rather than as an overall mean may be warranted. Therefore, it may be useful in future studies to take such a dynamic perspective in analysing dot-probe task data, to more accurately reflect heterogeneity in attention biases in social anxiety, and its role in predicting ABM effectiveness. Nonetheless, the reliability of dot-probe task needs more investigation with regards to the length of it and how this might affect its reliability [78] e.g. the dynamic approach of analysing dot-probe task creates a higher number of trials which in turn might be related with the good reliability. In defence of our reliance on the more traditional approach to analysis in the current study, we aimed for comparability to the previous work on SAD, that yielded mixed effects of ABM, in order to address the role of variables like individual differences, treatment seeking and types of outcome measures. Adding to that, relying on methods beyond reaction time, such as eye-tracking may be another way to circumvent such concerns. A final point with regards to limitations, is that several participants reported fatigue by the end of the task, which may have altered their attention allocation. Generally, participants’ experience with the dot-probe task is an issue that limited studies examined [79], a topic that may prove useful to consider in future work [80].

Conclusions

In sum, this experimental study of the effects of ABM in social anxiety found no significant improvement in either attention biases or in self-report and clinical interview measures of social anxiety. Additionally, baseline attention biases did not moderate social anxiety changes from pre- to post treatment, even though they strongly suggested the presence of individual differences in whether individuals show mostly a vigilance or an avoidance pattern. The heterogeneity of attention biases, together with methodological differences from previous studies that found effectiveness of ABM in SAD should be taken into consideration in future research to produce more conclusive results as to if, under what circumstances, and for whom, this form of training is an effective intervention.

References

  1. 1. Grant BF, Hasin DS, Blanco C, Stinson FS, Chou SP, Goldstein RB, et al. The epidemiology of social anxiety disorder in the United States: results from the National Epidemiologic Survey on Alcohol and Related Conditions. The Journal of clinical psychiatry. 2005 Nov 15;66(11):1351–61. pmid:16420070
  2. 2. Kessler RC, Stein MB, Berglund P. Social phobia subtypes in the National Comorbidity Survey. American Journal of Psychiatry. 1998 May 1;155(5):613–9.
  3. 3. Wittchen HU, Jacobi F, Rehm J, Gustavsson A, Svensson M, Jönsson B, et al. The size and burden of mental disorders and other disorders of the brain in Europe 2010. European neuropsychopharmacology. 2011 Sep 1;21(9):655–79. pmid:21896369
  4. 4. Stuhldreher N, Leibing E, Leichsenring F, Beutel ME, Herpertz S, Hoyer J, et al. The costs of social anxiety disorder: the role of symptom severity and comorbidities. Journal of affective disorders. 2014 Aug 20;165:87–94. pmid:24882183
  5. 5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th (edn) Arlington. VA, American Psychiatric Association. 2013.
  6. 6. Grant BF, Hasin DS, Stinson FS, Dawson DA, Chou SP, Ruan WJ, et al. Co-occurrence of 12-month mood and anxiety disorders and personality disorders in the US: results from the national epidemiologic survey on alcohol and related conditions. Journal of psychiatric research. 2005 Jan 1;39(1):1–9. pmid:15504418
  7. 7. DeWit DJ, Ogborne A, Offord DR, MacDonald K. Antecedents of the risk of recovery from DSM-III-R social phobia. Psychological Medicine. 1999 May;29(3):569–82. pmid:10405078
  8. 8. Fehm L, Beesdo K, Jacobi F, Fiedler A. Social anxiety disorder above and below the diagnostic threshold: prevalence, comorbidity and impairment in the general population. Social psychiatry and psychiatric epidemiology. 2008 Apr 1;43(4):257–65. pmid:18084686
  9. 9. Schneier FR, Johnson J, Hornig CD, Liebowitz MR, Weissman MM. Social phobia: Comorbidity and morbidity in an epidemiologic sample. Archives of general psychiatry. 1992 Apr 1;49(4):282–8. pmid:1558462
  10. 10. Albano AM, DiBartolo PM. Cognitive-behavioral therapy for social phobia in adolescents: Stand up, speak out. Oxford University Press; 2007 Apr 12.
  11. 11. Browne MA, Elisabeth Wells J, Mcgee MA, New Zealand Mental Health Survey Research Team. Twelve-month and lifetime health service use in te Rau Hinengaro: the New Zealand mental health survey. Australian & New Zealand Journal of Psychiatry. 2006 Oct;40(10):855–64.
  12. 12. Cuijpers P, Schuurmans J. Self-help interventions for anxiety disorders: an overview. Current psychiatry reports. 2007 Aug 1;9(4):284–90. pmid:17880859
  13. 13. Amir N, Beard C, Taylor CT, Klumpp H, Elias J, Burns M, et al. Attention training in individuals with generalized social phobia: A randomized controlled trial. Journal of consulting and clinical psychology. 2009 Oct;77(5):961. pmid:19803575
  14. 14. Bar-Haim Y, Lamy D, Pergamin L, Bakermans-Kranenburg MJ, Van Ijzendoorn MH. Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study. Psychological bulletin. 2007 Jan;133(1):1. pmid:17201568
  15. 15. Clark DM. A cognitive model. Social phobia: Diagnosis, assessment, and treatment. 1995 Oct 27:69–73.
  16. 16. Rapee RM, Heimberg RG. A cognitive-behavioral model of anxiety in social phobia. Behaviour research and therapy. 1997 Aug 1;35(8):741–56. pmid:9256517
  17. 17. MacLeod C, Hagan R. Individual differences in the selective processing of threatening information, and emotional responses to a stressful life event. Behaviour research and therapy. 1992 Mar 1;30(2):151–61. pmid:1567344
  18. 18. MacLeod C, Rutherford E, Campbell L, Ebsworthy G, Holker L. Selective attention and emotional vulnerability: assessing the causal basis of their association through the experimental manipulation of attentional bias. Journal of abnormal psychology. 2002 Feb;111(1):107. pmid:11866165
  19. 19. Amir N, Weber G, Beard C, Bomyea J, Taylor CT. The effect of a single-session attention modification program on response to a public-speaking challenge in socially anxious individuals. Journal of abnormal psychology. 2008 Nov;117(4):860. pmid:19025232
  20. 20. Hakamata Y, Lissek S, Bar-Haim Y, Britton JC, Fox NA, Leibenluft E, et al. Attention bias modification treatment: a meta-analysis toward the establishment of novel treatment for anxiety. Biological psychiatry. 2010 Dec 1;68(11):982–90. pmid:20887977
  21. 21. Dennison TD. A Meta-Analysis Examining Attention Bias Modification Treatment and the Reduction of Psychological Distress.
  22. 22. Linetzky M, Pergamin‐Hight L, Pine DS, Bar‐Haim Y. Quantitative evaluation of the clinical efficacy of attention bias modification treatment for anxiety disorders. Depression and anxiety. 2015 Jun;32(6):383–91. pmid:25708991
  23. 23. Lazarov A, Marom S, Yahalom N, Pine DS, Hermesh H, Bar-Haim Y. Attention bias modification augments cognitive–behavioral group therapy for social anxiety disorder: a randomized controlled trial. Psychological medicine. 2018 Oct;48(13):2177. pmid:29258631
  24. 24. Li S, Tan J, Qian M, Liu X. Continual training of attentional bias in social anxiety. Behaviour Research and Therapy. 2008 Aug 1;46(8):905–12. pmid:18538305
  25. 25. Naim R, Kivity Y, Bar-Haim Y, Huppert JD. Attention and interpretation bias modification treatment for social anxiety disorder: A randomized clinical trial of efficacy and synergy. Journal of Behavior Therapy and Experimental Psychiatry. 2018 Jun 1;59:19–30. pmid:29127945
  26. 26. Schmidt NB, Richey JA, Buckner JD, Timpano KR. Attention training for generalized social anxiety disorder. Journal of abnormal psychology. 2009 Feb;118(1):5. pmid:19222309
  27. 27. Huppert JD, Kivity Y, Cohen L, Strauss AY, Elizur Y, Weiss M. A pilot randomized clinical trial of cognitive behavioral therapy versus attentional bias modification for social anxiety disorder: An examination of outcomes and theory-based mechanisms. Journal of Anxiety Disorders. 2018 Oct 1;59:1–9. pmid:30103187
  28. 28. Boettcher J, Leek L, Matson L, Holmes EA, Browning M, MacLeod C, et al. Internet-based attention bias modification for social anxiety: a randomised controlled comparison of training towards negative and training towards positive cues. PloS one. 2013 Sep 30;8(9):e71760. pmid:24098630
  29. 29. Boettcher J, Berger T, Renneberg B. Internet-based attention training for social anxiety: A randomized controlled trial. Cognitive Therapy and Research. 2012 Oct;36(5):522–36.
  30. 30. Bunnell BE, Beidel DC, Mesa F. A randomized trial of attention training for generalized social phobia: Does attention training change social behavior?. Behavior Therapy. 2013 Dec 1;44(4):662–73. pmid:24094791
  31. 31. Carlbring P, Apelstrand M, Sehlin H, Amir N, Rousseau A, Hofmann SG, et al. Internet-delivered attention bias modification training in individuals with social anxiety disorder-a double blind randomized controlled trial. BMC psychiatry. 2012 Dec;12(1):1–9. pmid:22731889
  32. 32. Neubauer K, von Auer M, Murray E, Petermann F, Helbig-Lang S, Gerlach AL. Internet-delivered attention modification training as a treatment for social phobia: A randomized controlled trial. Behaviour Research and Therapy. 2013 Feb 1;51(2):87–97. pmid:23262116
  33. 33. Heeren A, Mogoașe C, Philippot P, McNally RJ. Attention bias modification for social anxiety: a systematic review and meta-analysis. Clinical psychology review. 2015 Aug 1;40:76–90.
  34. 34. Cisler JM, Koster EH. Mechanisms of attentional biases towards threat in anxiety disorders: An integrative review. Clinical psychology review. 2010 Mar 1;30(2):203–16. pmid:20005616
  35. 35. Kruijt AW, Parsons S, Fox E. A meta-analysis of bias at baseline in RCTs of attention bias modification: no evidence for dot-probe bias towards threat in clinical anxiety and PTSD. Journal of abnormal psychology. 2019 Aug;128(6):563. pmid:31368735
  36. 36. Ononaiye MS, Turpin G, Reidy JG. Attentional bias in social anxiety: Manipulation of stimulus duration and social-evaluative anxiety. Cognitive Therapy and Research. 2007 Dec 1;31(6):727–40.
  37. 37. Roberts KE, Hart TA, Eastwood JD. Attentional biases to social and health threat words in individuals with and without high social anxiety or depression. Cognitive Therapy and Research. 2010 Aug 1;34(4):388–99.
  38. 38. Bantin T, Stevens S, Gerlach AL, Hermann C. What does the facial dot-probe task tell us about attentional processes in social anxiety? A systematic review. Journal of behavior therapy and experimental psychiatry. 2016 Mar 1;50:40–51. pmid:26042381
  39. 39. Pishyar R, Harris LM, Menzies RG. Attentional bias for words and faces in social anxiety. Anxiety, Stress & Coping. 2004 Mar 1;17(1):23–36.
  40. 40. Rodebaugh TL, Scullin RB, Langer JK, Dixon DJ, Huppert JD, Bernstein A, et al. Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of abnormal psychology. 2016 Aug;125(6):840. pmid:27322741
  41. 41. Zvielli A, Bernstein A, Koster EH. Temporal dynamics of attentional bias. Clinical Psychological Science. 2015 Sep;3(5):772–88.
  42. 42. Amir N, Elias J, Klumpp H, Przeworski A. Attentional bias to threat in social phobia: Facilitated processing of threat or difficulty disengaging attention from threat?. Behaviour research and therapy. 2003 Nov 1;41(11):1325–35. pmid:14527531
  43. 43. Cisler JM, Koster EH. Mechanisms of attentional biases towards threat in anxiety disorders: An integrative review. Clinical psychology review. 2010 Mar 1;30(2):203–16. pmid:20005616
  44. 44. Clarke PJ, MacLeod C, Guastella AJ. Assessing the role of spatial engagement and disengagement of attention in anxiety-linked attentional bias: a critique of current paradigms and suggestions for future research directions. Anxiety, Stress & Coping. 2013 Jan 1;26(1):1–9. pmid:22136158
  45. 45. Garner M, Mogg K, Bradley BP. Orienting and maintenance of gaze to facial expressions in social anxiety. Journal of abnormal psychology. 2006 Nov;115(4):760. pmid:17100533
  46. 46. Mansell W, Clark DM, Ehlers A, Chen YP. Social anxiety and attention away from emotional faces. Cognition & Emotion. 1999 Oct 1;13(6):673–90.
  47. 47. Mühlberger A, Wieser MJ, Pauli P. Visual attention during virtual social situations depends on social anxiety. CyberPsychology & Behavior. 2008 Aug 1;11(4):425–30. pmid:18721090
  48. 48. Mogg K, Bradley B, Miles F, Dixon R. Brief report time course of attentional bias for threat scenes: testing the vigilance‐avoidance hypothesis. Cognition and emotion. 2004 Aug 1;18(5):689–700.
  49. 49. Mogg K, Bradley BP. Anxiety and attention to threat: Cognitive mechanisms and treatment with attention bias modification. Behaviour research and therapy. 2016 Dec 1;87:76–108. pmid:27616718
  50. 50. Amir N, Taylor CT, Donohue MC. Predictors of response to an attention modification program in generalized social phobia. Journal of consulting and clinical psychology. 2011 Aug;79(4):533. pmid:21707134
  51. 51. Kuckertz JM, Gildebrant E, Liliequist B, Karlström P, Väppling C, Bodlund O, et al. Moderation and mediation of the effect of attention training in social anxiety disorder. Behaviour Research and Therapy. 2014 Feb 1;53:30–40. pmid:24373984
  52. 52. MacLeod C, Mathews A. Anxiety and the allocation of attention to threat. The Quarterly journal of experimental psychology. 1988 Nov 1;40(4):653–70. pmid:3212208
  53. 53. Bradley BP, Mogg K, Falla SJ, Hamilton LR. Attentional bias for threatening facial expressions in anxiety: Manipulation of stimulus duration. Cognition & emotion. 1998 Nov 1;12(6):737–53.
  54. 54. Roberson-Nay R, Strong DR, Nay WT, Beidel DC, Turner SM. Development of an abbreviated Social Phobia and Anxiety Inventory (SPAI) using item response theory: The SPAI-23. Psychological Assessment. 2007 Mar;19(1):133. pmid:17371128
  55. 55. Brown TA, Barlow DH, DiNardo PA. Anxiety disorders interview schedule adult version: Client interview schedule. Graywind Publications Incorporated; 1994.
  56. 56. Di Nardo PA, Moras K, Barlow DH, Rapee RM, Brown TA. Reliability of DSM-III-R anxiety disorder categories: using the Anxiety Disorders Interview Schedule—Revised (ADIS-R). Archives of General Psychiatry. 1993 Apr 1;50(4):251–6. pmid:8466385
  57. 57. Schry AR, Roberson-Nay R, White SW. Measuring social anxiety in college students: A comprehensive evaluation of the psychometric properties of the SPAI-23. Psychological assessment. 2012 Dec;24(4):846. pmid:22369649
  58. 58. Turner SM, Beidel DC, Dancu CV, Stanley MA. An empirically derived inventory to measure social fears and anxiety: the Social Phobia and Anxiety Inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology. 1989 Mar;1(1):35.
  59. 59. Theodorou M, Panayiotou G. Psychometric properties of the Social Phobia and Anxiety Inventory-23 in Greek-Cypriot college population. Poster presented at the CAN annual conference, Nicosia, Cyprus, 2015.
  60. 60. Liebowitz MR. Social phobia. Modern problems of pharmacopsychiatry. 1987.
  61. 61. Baker SL, Heinrichs N, Kim HJ, Hofmann SG. The Liebowitz social anxiety scale as a self-report instrument: a preliminary psychometric analysis. Behaviour research and therapy. 2002 Jun 1;40(6):701–15. pmid:12051488
  62. 62. Tottenham N, Tanaka JW, Leon AC, McCarry T, Nurse M, Hare TA, et al. The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry research. 2009 Aug 15;168(3):242–9. pmid:19564050
  63. 63. Matsumoto D, Ekman P. American-Japanese cultural differences in intensity ratings of facial expressions of emotion. Motivation and emotion. 1989 Jun;13(2):143–57.
  64. 64. Öhman A. Face the beast and fear the face: Animal and social fears as prototypes for evolutionary analyses of emotion. Psychophysiology. 1986 Mar;23(2):123–45. pmid:3704069
  65. 65. Arad G, Abend R, Pine DS, Bar‐Haim Y. A neuromarker of clinical outcome in attention bias modification therapy for social anxiety disorder. Depression and anxiety. 2019 Mar;36(3):269–77. pmid:30408271
  66. 66. Cisler JM, Bacon AK, Williams NL. Phenomenological characteristics of attentional biases towards threat: A critical review. Cognitive therapy and research. 2009 Apr;33(2):221–34. pmid:20622985
  67. 67. Schmukle SC. Unreliability of the dot probe task. European Journal of Personality. 2005 Dec;19(7):595–605.
  68. 68. Staugaard SR. Reliability of two versions of the dot-probe task using photographic faces. Psychology Science Quarterly. 2009 Jul 1;51(3):339–50.
  69. 69. Waechter S, Nelson AL, Wright C, Hyatt A, Oakman J. Measuring attentional bias to threat: Reliability of dot probe and eye movement indices. Cognitive therapy and research. 2014 Jun 1;38(3):313–33.
  70. 70. Bar-Haim Y, Holoshitz Y, Eldar S, Frenkel TI, Muller D, Charney DS, et al. Life-threatening danger and suppression of attention bias to threat. American Journal of Psychiatry. 2010 Jun 1;167(6):694–8. pmid:20395400
  71. 71. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986 Mar 1:121–30. pmid:3719049
  72. 72. Zeger SL, Liang KY, Albert PS. Models for longitudinal data: a generalized estimating equation approach. Biometrics. 1988 Dec 1:1049–60. pmid:3233245
  73. 73. Alon Y, Arad G, Pine DS, Bar-Haim Y. Statistical learning as a predictor of attention bias modification outcome: A preliminary study among socially anxious patients. Behaviour research and therapy. 2019 Jan 1;112:36–41. pmid:30496885
  74. 74. Amir N, Foa EB, Coles ME. Automatic activation and strategic avoidance of threat-relevant information in social phobia. Journal of abnormal psychology. 1998 May;107(2):285. pmid:9604557
  75. 75. Heeren A, Lievens L, Philippot P. How does attention training work in social phobia: disengagement from threat or re-engagement to non-threat?. Journal of anxiety disorders. 2011 Dec 1;25(8):1108–15. pmid:21907539
  76. 76. Price RB, Wallace M, Kuckertz JM, Amir N, Graur S, Cummings L, et al. Pooled patient-level meta-analysis of children and adults completing a computer-based anxiety intervention targeting attentional bias. Clinical psychology review. 2016 Dec 1;50:37–49. pmid:27693664
  77. 77. Price RB, Kuckertz JM, Siegle GJ, Ladouceur CD, Silk JS, Ryan ND, et al. Empirical recommendations for improving the stability of the dot-probe task in clinical research. Psychological assessment. 2015 Jun;27(2):365. pmid:25419646
  78. 78. Steinborn MB, Langner R, Flehmig HC, Huestegge L. Methodology of performance scoring in the d2 sustained-attention test: Cumulative-reliability functions and practical guidelines. Psychological Assessment. 2018 Mar;30(3):339. pmid:28406669
  79. 79. Kuckertz JM, Schofield CA, Clerkin EM, Primack J, Boettcher H, Weisberg RB, et al. Attentional bias modification for social anxiety disorder: what do patients think and why does it matter?. Behavioural and cognitive psychotherapy. 2019 Jan;47(1):16–38. pmid:29729676
  80. 80. Langner R, Steinborn MB, Chatterjee A, Sturm W, Willmes K. Mental fatigue and temporal preparation in simple reaction-time performance. Acta psychologica. 2010 Jan 1;133(1):64–72. pmid:19878913