Reader Comments

Post a new comment on this article

correlation, causation and using a millions of potential assessors

Posted by georgealozano on 20 Nov 2013 at 21:09 GMT

The paper does not define “merit”. How does it differ from “impact”. Is it just what “experts” think is “good”. Then how do these experts define “good”. They probably were thinking "expected future impact”.

In any case, it should be “easy enough” to separate the effects of IF on impact or merit. Remove the authors, affiliations, and any other identifying features from a paper. Then give the same paper to pairs of assessors and tell one that the paper was from a high IF journal, and the other that the paper came from a low IF journal. Then sit and wait for 5-10 years while the papers accumulate citations. Or give them redacted versions of old papers from the same journal, seal them in rooms without internet access, and ask them to assess the papers. Of course, exclude assessors who have already read the papers.

"by definition, papers in high IF journals are more highly cited."

Not necessarily true. The IF has a specific definition. It considers only a 1-2 year window for citations and it is the MEAN of a distribution that is known to be skewed. If anything, the IF says something about the journal, not the papers therein, and definitely not the authors. Even the use of the IF to assess journal quality is highly problematic (http://occamstypewriter.o...)

“The IF might potentially be a better measure of merit than either a post-publication assessment or the number of citations, since several individuals are typically involved in a decision to publish, so the error variance associated with their combined assessment should be lower than that associated with the number of citations”

In contrast to the IF that might depend on “several individuals”, a paper’s citation rate depends on the ENTIRE world-wide scientific community.

Yes, the citation rate of papers is affected by many factors (authors affiliations, journals, affiliations, etc), but at least the citation rate is a measure of the specific paper, not one of the journal, which is just attached on the papers therein just for convenience. It should be easy enough to correct for all these factors. For instance, given 2 papers (same field, same publication date) with similar citations records, the one that appeared in Plos would be of higher quality than the one that appeared in Nature.

No competing interests declared.