Dear editor

We appreciate the constructive reviewer comments as well as your clear and explicit guidance for how to integrate the provided suggestions. Also, we would like to thank PLOS ONE for kindly granting us additional time, in order for us to collect more data. In the following we first address your editorial comments, where after we address each reviewers’ comments one by one.

We agree that increasing the sample size greatly benefits our ability to provide stronger evidence for the inferences we intend to draw. Following your recommendation, our plan was to double the sample size. Due to Covid-19 our lab was closed for about 3 months in the Spring of 2020. After reopening, we managed to increase our sample size from 81 to 149, before our university had to close down the lab again, forcing us to cancel the last session we had planned. However, the results remain insignificant. Furthermore, given the current correlation we would need to further increase the sample size substantially, in order to document a significant positive effect of higher stakes; to about 247, as explained in the paper. In other words, the increased total sample size we now rely upon allows us to say that the sample should have shown a correlation of r = 0.16 in order for the results to be statistically significant with a simple linear regression. In other words, we find no significant result, and if there is an effect, it is relatively small – and certainly economically inefficient.

We’d also like to emphasize that our sample size is at the high end, compared to the 9 studies we refer to in our theoretical framework. As in other related theoretical fields, effect sizes of the impact of high (vs. low rewards) seem to turn out smaller than original studies indicate. We should also add that the performance in each condition in the newly collected data is almost identical to our former results, indicating that we should not worry that behavior was different in the Summer of 2020, compared to pre-pandemic times (see S4 Appendix for the relevant table). Our country was functioning quite normally in the relevant period, btw.

All findings, tables and figures have been updated, integrating the new data and points we make in the above section. In accordance with your suggestion, we have maintained the general outline of the paper, but also adjusted some phrasings to ensure that we write, for example; “we do not find a significant difference between the two conditions”, rather than, for example: “our results show that a higher stake does not lead to increased performance”. In essence, we report a null-finding, yet we also document that given our setup, one requires fairly large sample sizes, if one wants to showcase robust significant results.

Response to Reviewer 1

Reviewer comment: Congratulations on a paper that you wrote in a remarkably crispy, clear and concise way. Everything that should be there is there - and nothing more (which is also an art).

Reply: Thank you very much for your gracious remarks on our paper!

Reviewer comment: I like all about the paper with one exception - the sample size!

Reply: We agree on your concern about the sample size and appreciate the opportunity to enhance the strength of our inferences. We therefore planned to double the sample (following the recommendation by the editor). However, due to Covid-19 our lab was closed for about 3 months in the Spring of 2020. After reopening, we managed to increase our sample size from 81 to 149, before our university had to close down the lab again, forcing us to cancel the last session we had planned.

However, the results remain insignificant. Furthermore, given the current correlation we would need to further increase the sample size substantially, in order to document a significant positive effect of higher stakes; to about 247, as explained in the paper. In other words, the increased total sample size we now rely upon allows us to say that the sample should have shown a correlation of r = 0.16 in order for the results to be statistically significant with a simple linear regression. In other words, we find no significant result, and if there is an effect, it is relatively small – and certainly economically inefficient.

We’d also like to emphasize that our sample size is at the high end, compared to the 9 studies we refer to in our theoretical framework. As in other related theoretical fields, effect sizes of the impact of high (vs. low rewards) seem to turn out smaller than original studies indicate. We should also add that the performance in each condition in the newly collected data is almost identical to our former results, indicating that we should not worry that behavior was different in the Summer of 2020, compared to pre-pandemic times (see S4 Appendix for the relevant table). Our country was functioning quite normally in the relevant period, btw.

All findings, tables and figures have been updated, integrating the new data and points we make in the above section. In accordance with your suggestion, we have maintained the general outline of the paper, but also adjusted some phrasings to ensure that we write, for example; “we do not find a significant difference between the two conditions”, rather than, for example: “our results show that a higher stake does not lead to increased performance”. In essence, we report a null-finding, yet we also document that given our setup, one requires fairly large sample sizes, if one wants to showcase robust significant results.

Again, thank you for your encouraging comments.

Response to Reviewer 2

We would like to thank you for engaging with our paper and challenging our choices and findings. In the following we respond to each of your questions and comments.

Reviewer comment: This paper considers the issue of whether higher stakes lead to better performance. It is a somewhat controversial topic. But to me the main issue is at the end of the range, with very high or very low incentives. This study has payoffs in one case multiplied by five and I am not sure this captures the issue. But it appears the experiments were conducted properly, so it would appear to be rigorous. I just don't see what we learn from this study.

Reply: Thank you for challenging our approach with this comment. Former studies investigating the impact of higher (vs. lower) stakes demonstrate varying results, even across a range of differences in the size of the stakes. We worry that publication biases might shape what (significant) results are available, and, it is certainly not clear what effect sizes one should expect. We decided on a factor of five (medium difference compared to former studies), in order to balance the likelihood of maximizing the difference (cf. the Maxmincon principle, Kerlinger 2006), and improve external validity. In other words, a very high factor is less likely to be useful to implement in an actual organizational context. Therefore, we think it would be relevant to know, if it is difficult to establish an effect when multiplying the stakes with five, which already can be argued to be at the high end of what would be realistic in an organizational context. However, we agree that investigating the challenge at the end of the range (very high or very low) would be highly interesting as well, for theoretical reasons.

Reviewer comment: Abstract, 2nd sentence: Seriously? Then why do teams pay pro athletes so much? Can't really be true as stated.

Reply: We agree that the original phrase was unclear and did not convey our intention meaning. We aim to communicate that observational data indicate that increasing the stakes does not seem to enhance the performance, possibly due to choking effects (as Ariely et al. 2009 speculates). The sentence has been rephrased and we appreciate your attentive eye.

Reviewer comment: Introduction, 2nd paragraph: I really don't agree with this. Higher pay clearly typically leads to higher effort. The issue of interest is really just at the extremes of the pay range.

Reply: In the paragraph in question we do not stipulate that higher pay might lead to higher effort. Rather, we discuss the impact on performance. As mentioned above, empirical studies on performance demonstrate ambiguous effects, some emphasizing choking effects (and thus a negative effect of higher stakes) while others find a positive effect on performance of higher stakes. Incidentally, we could add that our original motivation to engage in this study was Ariely et al.’s (2009) highly cited finding that performance would be lower, if you increase the rewards for completing cognitively mundane tasks. Our results do not corroborate Ariely et al.’s (2009) findings.

Reviewer comment: p. 9, middle: These are examples of extreme pressure, not just pressure. And the golf study is just one study. One wonders how many other studies found no effect.

Reply: Thank you for making this relevant point; sports are usually about extreme pressure situations. We have rephrased the section to make explicit, that such observational sports studies are looking at more extreme situations than a regular employee typically meets. We also acknowledge that this field could also be influenced by publication bias, hiding no effect findings.

Reviewer comment: p. 9, lower: It's really quite silly to say that payment per se leads to worse performance.

Reply: We assume the reviewer refers to this part: “[…] some studies find a positive effect on performance of higher stakes compared to lower (Castellan Jr., 1969; Achtziger, et al., 2015; Gneezy & Rustichini, 2000), others find a negative effect (Ariely, et al., 2009; Pokorny, 2008) […]” We consider our choice of words to be in line with the statements made by the original authors who write: “Second, and more importantly, the performance of participants was always lowest in the high payment condition when compared with the low- and mid-payment conditions together.” (Ariely, et al., 2009, p. 458). Furthermore, we also refer to Pokorny (2008, p. 255) “Indeed, subjects who are offered very low incentives perform significantly better than those in the NI and the HI treatments. This provides evidence for a positive effect of very low piece rates on work effort in this context.” Overall, we report the diverging empirical results as well as contrasting theoretical perspectives (micro-economics vs. psychology). We think the fact that empirical results point in different directions and that some imply results that from a micro-economic perspectives appear counter-intuitive, provides further rationale for the relevance of the study.

Reviewer comment: p. 9, penultimate full paragraph: I'm not sure this is an entirely fair review of the literature.

Reply: We have tried to engage in an extensive search of the literature, including looking at all studies that cite older studies (e.g. Castellan Jr. 1969) or the highly cited studies such as Ariely et al. (2009; 949 citations as of August 2020). We would of course appreciate to be made aware of other, relevant studies.

Reviewer comment: First line of paragraph before Hypothesis 1: But this may well depend on where in the range you are.

Reply: We agree that it would be interesting to focus on (for example) the very high range, but consider this outside the scope of our pre-registered approach. We have adapted the paragraph right before the hypothesis, to make our intent even clearer.

Reviewer comment: Hypotheses 1 and 2: Over the entire range?

Reply: As outlined in the comment above, we can only test our pre-registered hypothesis, i.e. comparing high vs. low stakes involving a difference of a factor of five. We have adapted the paragraph right before the hypothesis, to make our intent even clearer.

Reviewer comment: p. 12, lower: The between- versus within-subject could easily have mattered. Good that you mention this later.

Reply: Thank you for this comment. We agree that this difference is meaningful.

Reviewer comment: p. 13, upper: I don't know that a factor of five will matter in this range for this type of task.

Reply: We also did not know in advance what impact a factor of five would have for this type of task at this level of reward. The difference in reward is 8 DKK (more than 1 Euro) per solved task, leading to some of the highest performing participants to earn more than 60 Euros for their effort. If we had asked students about the difference – after finishing the session – we guess that they would find the difference between (for example) 15 and 75 Euros important. We hope future research can examine different levels of rewards, in order to clarify if the effect is different for different levels of rewards and different factors between the groups.

Reviewer comment: p. 14, Performance: This is still dubious. I'd also like to see separate analyses.

Reply: We assume that ‘separate analyses’ mean that there should be two regressions; one with solved adding tasks as a dependent variable and one with solved anagram tasks as a dependent variable. We appreciate the skepticism and interest in seeing a more fine-grained analysis. However, we have prioritized to follow the pre-registered analysis plan, rather than risking finding spurious results by running too many statistical tests. We have made the data available (due to privacy reasons, age has been removed from the dataset) and other researchers can therefore make their own statistical tests on the data.

Reviewer comment: p. 14, Intrinsic motivation: I'm not really keen on any of these psych measures, but presumably this was stated in the pre-analysis plan.

Reply: These measures are listed in the pre-registration, and hence we have decided to keep them in the article.

Reviewer comment: Before Figure 4: But it does in fact look like there is a positive difference. I wouldn't necessarily be claiming a big victory here.

Reply: We agree that it is important to transparently document the (in)significance of findings, rather than claiming victories that the data can’t support. We have updated our results following our additional data collection. As you indicate, we did showcase an insignificant difference in the first submission already. The difference is almost identical for the second round of data collection, leading to a smaller yet still insignificant p-value. Given the current correlation between stakes and performance, we would need a sample size of about 247 to identify a significant effect (if relying on a simple linear regression). The effect size is thus relatively small at best, if not random fluctuations.

All findings, tables and figures have been updated. We have adjusted some phrasings to ensure that we write, for example; “we do not find a significant difference between the two conditions”, rather than, for example: “our results show that a higher stake does not lead to increased performance”. In essence, we report a null-finding, yet we also document that given our research design setup, one requires fairly large sample sizes, if one wants to showcase robust significant results. In any case, the data are transparently provided and a future meta-analysis can integrate them.

Reviewer comment: Secondary analysis: 1) I'm not sure how much I trust these self-determination measures. 2) With the Cronbach number so low, I would just ignore this.

Reply: 1) We have updated the Cronbach numbers for our measures, following our additional data collection, and also added a paragraph on the reliability of these scales. 2) The lowest number (focus) increased to 0.55 (from 0.45), while the other remained above 0.70. We agree that the number for focus is very low, yet have chosen to keep them in the article. First, they are described in the pre-registration. Second, Ariely et al. (2009) indicated that the construct could be important, and while our measurement approach might have been ineffective, it can help provide guidance for future researchers’ attempt to measure the construct.

Reviewer comment: End of first paragraph of Discussion: I think the results are unclear here. Yes, they are cautionary.

Reply: We have deliberately phrased our summary in a cautionary manner (as also exemplified in of our former comments), since demonstrating a null effect is by definition difficult. We do find it relevant to provide evidence for that a positive impact of higher stakes would – if it exists – be relatively small (following reflective guidelines in Funder & Ozer 2018, and Gignac & Szodorai 2016), and would thus appear to be ineffective to implement.

Reviewer comment: First line of penultimate paragraph: I agree with this. It's interesting.

Reply: We appreciate your encouraging comment.

References

Kerlinger, F. (2006). Ch4: Research design as variance control, In D. de Vaus (Ed.), Research Design Volume I, (pp. 57-66). London: Sage.

Attachments

Attachment

Submitted filename: Response to reviewers, final.docx