Introduction

Over the last decade, crowdsourcing marketplaces such as Amazon Mechanical Turk (MTurk) and Prolific have provided participants for up to half of published studies in psychology and other social sciences [1]. MTurk is by far the most popular recruitment platform to offer rapid data collection from large samples at a relatively low cost and increased diversity of samples beyond the typical college undergraduate population. Prolific offers many of the same benefits as MTurk, but is geared toward academic researchers. Of course, these and other crowdsourcing platforms are not entirely distinct, given that there is likely some overlap in participant populations. Moreover, the characteristics of each of these platforms is likely to vary significantly over time. For example, Difallah and colleagues reported that the MTurk participant pool is largely comprised of new individuals every few years, with the half-life of participants ranging between 12 and 18 months [2]. Nevertheless, some crowdsourcing platforms have distinct features that might make recruitment more convenient for psychological research. For example, Prolific uniquely offers participant prescreening based on multiple factors, including self-reported clinical diagnoses (e.g., autism, depression).

Despite the many benefits of using crowdsourcing in psychological research, one general limitation of such online research is attrition. Attrition can occur for many reasons–participants might drop out of a study due to an experimental condition (e.g., punishment versus no punishment [3]), exogenous factors such as interruptions (e.g., attending to a child), or technical difficulties. Anonymity in online research also minimizes the social cost of dropping out of a study. Attrition in online experiments is significant not only because of its potential to increase the time and cost of conducting online research, but also because it can jeopardize internal validity when it occurs disproportionally across experimental conditions.

Research suggests that attrition rates are similar across studies conducted using both Prolific and MTurk. For example, Palan and Schitter reported ~25% attrition after two weeks for inexperienced Prolific participants–i.e., participants who had completed only one study on that platform [4]. Kothe & Ling also reported 25% attrition in a Prolific study after one year and ~40% attrition after 19 months [5]. Attrition rates are similarly high in MTurk studies spanning several weeks to months (30%) or up to one year (55%) [6]. However, attrition often is not reported in shorter-term studies with MTurk [7, 8] or Prolific.

A few studies, however, examined possible remedies for attrition in online research. For example, Zhou and Fishbach examined the effects of additional instructions with MTurk participants, defined as (1) a prewarning–i.e., informing participants about the task, (2) personalization–i.e., asking for personal information (MTurk ID) on the consent form, and (3) appealing to conscience–i.e., explaining the negative effects of attrition on research quality [8]. The combination of these instructions reduced attrition by more than half in a 5-min survey task compared to no additional instructions (from ~50% to ~20%) [9–11].

Increasing pay is a second empirically validated method for reducing attrition in MTurk studies [7, 12]. For example, Auer et al. evaluated attrition in a two-part survey study, with each part lasting approximately 30 min [12]. Across groups, Auer et al. manipulated both initial pay for completing part 1 –ranging from $0.50 to $2.00 –and pay increase from part 1 to part 2, ranging from 0% to 400%. Both initial pay and pay increase significantly predicted attrition, with attrition ranging from 30% to 82% across the highest- and lowest-paid groups, respectively. In line with these findings, Crump et al. reported in a concept-learning task that attrition was higher in a low-incentive condition (total pay of $0.75; 25% attrition) compared to a high-incentive condition (total pay of up to $4.50; 11% attrition) [7]. This limited research suggests that advertised pay likely is an important factor in minimizing attrition among MTurk participants.

The purpose of the present study was to extend findings on effects of (1) higher pay and (2) additional instructions on attrition to Prolific participants. Given the benefits of some of this platform’s features for psychological researchers, we sought only to test these manipulations in novel ways within the Prolific platform. For example, we evaluated effects of pay and instruction manipulations alone and in combination, a novel contribution to this research area. We hypothesized that both higher pay and additional instructions would reduce attrition, with greater reductions in attrition when combining these manipulations.

Methods

Results

Table 1 provides a summary of attrition rates. The Low Pay + Minimal Instructions group demonstrated the highest attrition rate (43.8%). Attrition was slightly lower when combining low pay with additional instructions (36.2%). Finally, attrition was lowest in the high-pay groups with or without additional instructions (~23%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Attrition in each group.

https://doi.org/10.1371/journal.pone.0292372.t001

Actual completion times for both parts of the study were relatively consistent among groups, ranging from 27 to 30 min (Group High Pay + Added Instructions: M = 29.2 min, SD = 9.2; Group High Pay + Minimal Instructions: M = 26.8 min, SD = 5.2; Group Low Pay + Added Instructions: M = 30.2 min, SD = 9.2; Group Low Pay + Minimal Instructions: M = 28.8 min, SD = 6.1). Actual pay rates in all groups were higher than advertised pay rates but similar among High Pay groups (Group High Pay + Added Instructions: M = $15.72, SD = 2.8; Group High Pay + Minimal Instructions: M = $16.74, SD = 2.6) and Low Pay groups (Group Low Pay + Added Instructions: M = $7.52, SD = 1.7; Group Low Pay + Minimal Instructions: M = $7.88, SD = 1.5). Results of Spearman’s rank-order correlation analysis indicated a weak (albeit statistically significant) positive correlation between actual pay rate for Part 1 and performance accuracy, defined as total points earned during the two tasks, ρ(158) = .16, p = .046; see Fig 1. This finding suggests that performance accuracy increases in simple reversal learning tasks with increases in pay rate. There were no significant correlations between these variables within individual groups (data not shown).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Correlation between pay rate and total points.

Note. Perfectly accurate performance would result in ~324 points.

https://doi.org/10.1371/journal.pone.0292372.g001

Finally, Table 2 shows the results of the logistic regression analysis. Only one factor (pay) significantly affected the odds of task completion. More specifically, an advertised base pay of $14.50/hr increased odds of task completion by a factor of 2.53 (95% CI [1.11, 5.77] compared to advertised base pay of $7.25/hr.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results of logistic regression.

https://doi.org/10.1371/journal.pone.0292372.t002

Discussion

We examined approaches to address both attrition and data quality when recruiting participants via crowdsourcing. Although there are many possible measures of data quality (e.g., number of characters on survey responses), performance accuracy on a simple operant task served as the measure of data quality in the present study. Instead of using MTurk, we recruited participants using Prolific, a crowdsourcing platform specifically catering to researchers. Our primary findings were that higher pay corresponded with reduced attrition and greater performance accuracy. Additional instructions did not affect these outcomes.

Additional instructions–that is, informing participants about the task, asking for personal information on the consent form, and describing the negative impact of attrition on research quality–did not impact attrition relative to minimal instructions (cf. [8]). There are several possible reasons for this null effect. Whereas prior research evaluated whether additional instructions could mitigate attrition during a 5-min survey at a pay rate of $6/hr [8], we arranged an operant behavioral task and survey that took approximately 30 min to complete, which is likely more representative of the type of task and duration typically arranged in experimental psychological research. We paid participants a minimum of $7.25/hr and up to double this rate. As a result, it is possible that additional instructions effectively reduce attrition only (1) when tasks are brief and/or (2) within a lower range of pay rates. For example, we saw a slight (but not statistically significant) reduction in attrition across low-pay groups earning $7.25/hr and experiencing the presence or absence of additional instructions (from 44% to 36%). However, attrition was similar in high-pay groups earning $14.50/hr regardless of instructions (~23%). Future research could examine effects of additional instructions across a range of task types, durations, and payment amounts.

Consistent with findings with participants recruited from MTurk [7, 12], results of the present study also suggested that increasing advertised pay rates from $7.25 to $14.50/hr could reduce attrition with Prolific participants. We extended prior research by using what might be considered a relatively high minimum-pay rate consistent with the US minimum wage (cf. [12]). In contrast, Auer et al. evaluated effects of hourly pay ranging between $1.11 and $10.27 on attrition, with most participants (89%) earning $5.75/hr or less [12]. These low pay rates reflect an important difference between MTurk and Prolific; that is, MTurk does not regulate participant payment. When recruiting participants via Prolific, it was not possible to examine effects of the lower pay rates used by Auer et al.; researchers were required to pay participants no less than $6.50/hr at the time that we collected our data in 2021. This has since been increased to $8/hr, as of June 2022. Nevertheless, our findings suggest that the mitigating effects of increased pay on attrition generalize to higher overall pay rates.

A related finding was that higher actual pay rates were associated with more accurate performance on common tasks used to assess inhibitory control and behavioral flexibility [22; cf. 23]. As with studies examining effects of pay on attrition, MTurk studies examining effects of pay on data quality have paid participants at low rates–i.e., below the US minimum wage of $7.25. For example, Buhrmester et al. examined effects of hourly pay ranging between $0.04 and $6 on alpha reliabilities in personality questionnaires as a measure of data quality, with nearly three-quarters of participants earning $1.20/hr or less [23]. While Buhrmester et al. demonstrated no effect of pay on data quality among US and non-US participants, our findings could suggest that this null effect was due to (1) the lower overall range of pay rates examined (but see [24]) or (2) differences among US and non-US participants [22, 25]. Related to the second point, previous research has shown that increasing hourly pay to $10 from a minimum rate of $4 [22] or $0.20 [25] results in higher-quality data among participants based in India but not in the US. The present study is the first to demonstrate that pay could also impact data quality among US-based participants when increasing hourly pay from $7.25 to double that rate.

One limitation of the present study is that demographic variables (e.g., age, sex, income) were not controlled for or assessed when examining performance accuracy. There is some evidence for age- and sex-related differences in performance on tasks assessing inhibitory control. For example, studies have shown that, among healthy participants completing a go/no-go task, older adults make fewer no-go errors compared to younger adults [26] and that women demonstrate greater inhibition on no-go trials compared to men [27]. In the present study, the demographic survey comprised second part of the study, and approximately one-third of participants dropped out before providing demographic information. We presented the demographic survey after the experimental tasks to mirror the presentation of tasks in previous crowdsourcing research [28, 29]. Nevertheless, it is possible that demographic variables and performance accuracy interact in ways that are not accounted for in the present study. Future researchers should evaluate how demographic variables influence the relation between pay rate and performance accuracy.

A second limitation of the present study is that we used a broad definition of attrition. That is, attrition included both failure to (1) complete an ongoing experimental task, and (2) return to the Prolific website to complete a second task (survey). Some researchers might be interested in addressing within-task attrition specifically, given that is rarely tracked or reported [7] and has raised significant validity concerns [8]. Thus, future researchers should consider parsing out the effect of independent variables such as pay rate on each form of attrition. Distinguishing between these forms of attrition would further improve our understanding of how manipulations such as increased pay could improve the quality of crowdsourcing research.

A third limitation is that participation in the present study required installing the Inquisit Player (https://www.millisecond.com/products/inquisitplayer). This software likely does not present issues with compatibility: it can be quickly and easily installed on Windows and Mac devices, as well as Android devices and Chromebooks. However, it is still possible that some participants did not complete the study because either they could not or did not want to install the software. Therefore, future studies might consider experimental tasks that are hosted on the web and do not require software installation [28].

Altogether, our findings demonstrated that paying participants at a rate well beyond the US minimum wage, and closer to a living wage (see livingwage.mit.edu/), not only reduces attrition but also is correlated with higher-quality performance. Low pay rates are commonplace on MTurk, with the average US-based MTurk worker earning a median wage of $3.01/hr, significantly below the minimum wage [30]. This is despite research showing that monetary compensation is the highest-rated motivation for study completion among US-based MTurk participants [25]. Findings from the present study build on previous research and suggest that fair compensation mutually benefits participants and researchers. For example, other research has shown that fair compensation could facilitate recruitment and reduce attrition in groups traditionally underrepresented in research, such as individuals of lower socio-economic status, who cannot afford to dedicate time to research in the absence of adequate payment [31]. The present findings further suggest that paying participants closer to a living wage not only mitigates threats to internal validity via reduced attrition but also could facilitate collection of higher-quality data. Beyond these benefits, researchers should strongly consider their ethical obligation to compensate crowdsourcing participants fairly, even when minimum payment requirements are not imposed (e.g., MTurk).

Supporting information