Reader Comments

Post a new comment on this article

false matches for "all but significant"

Posted by xgregg on 20 Feb 2022 at 13:46 GMT

Looking at the data (thanks!), I was surprised to see so many low p values for the phrase "all but significant". I spot-checked four papers with available full-text, and in all of them, the actual phrase was "small but significant". So I think a broader review is warranted for this and similar prefix losses.
PMIDs checked: 1283590, 31996246, 32503649, 32497061

No competing interests declared.
report a concern respond to this posting

RE: false matches for "all but significant"

jtijdink483 replied to xgregg on 25 Feb 2022 at 00:33 GMT

Dear Xan Gregg,

Thank you for your post on our article and your Twitter thread about this topic, which we read with great interest. We applaud your enthusiasm for the data and your additional analysis with the non-linear model fit. We are grateful that you spotted this omission in our analysis. It is a problem that inherently comes with automated analyses of texts. It also gives a better explanation why most of the reported p values that were linked to the phrase ‘all but significant’ were actually below the threshold of p<0.05. We have discussed this with the other authors and concluded that this is indeed an important limitation, although it will not affect the manuscript’s conclusions. We think that it makes our findings even stronger. Finally, we believe that for most of the phrases, this does not apply. Thank you for spotting this very interesting limitation of our analysis.

Warm regards, also on behalf of the other authors
Joeri Tijdink

No competing interests declared.
report a concern respond to this posting

RE: RE: false matches for "all but significant"

xgregg replied to jtijdink483 on 03 Mar 2022 at 20:43 GMT

Thank you. Glad this finding explains the low p values for that phrase and strengthens the main finding.

No competing interests declared.
report a concern respond to this posting