Reader Comments

Post a new comment on this article

Referee Comments: Referee 1 (Matthew Symonds)

Posted by PLOS_ONE_Group on 05 Oct 2007 at 21:20 GMT

Reviewer 1's Review (Matthew Symonds)

-----
This paper carries out a statistical analysis to confirm what we probably already know, that the impact of evolutionary papers in the leading journals (Nature, Science etc.) is overestimated by those journals' impact factors (although the papers still have the highest impact of all evolutionary biology papers). Further, evolutionary papers in specialized (but non-evolutionary journals) actually 'punch above their weight' in terms of impact. A formalised system of measuring impact factors based on specific fields is suggested in the conclusion of the paper.

The results are not highly surprising, but it is good to see this effect analysed and quantified.

Some points I think worth considering:

You need to be a little careful when you talk about impact as to whether you are referring generally (across science as a whole), or specifically to the field of evolutionary biology. For example, in the end of the introduction (last paragraph) you say that publishing in Nature and Science may be overrated for an evolutionary biologist. But overrated by who? Not by other evolutionary biologists, since these are still the best journals to publish in, even within the field. Again, first para of discussion 'Although the impact of evolutionary articles published in Nature, Science and PNAS, is relatively low...'. Low relative to what?

You show that journals exhibit differences in relative impact (and impact ranking) right down to specific topics within evolutionary biology. This may not be so surprising. It probably reflects that even within evolutionary biology the journals have certain focuses. So, I would probably not submit a good paper on speciation to Animal Behaviour (I would probably think it more appropriate, even if it concerned a behaviour, to aim for Evolution, say). This might reflect why speciation papers in Anim Behav have lower impact etc.

In terms of the suggestion for a more detailed and sophisticated journal classification system. Isn't it that we need a better way of classifying the papers instead of/as well as the journal - So that we have a separate impact factor for 'ecology papers in Nature'? I think this is what you are trying to say with the 'journals can be in different categories' idea, but it isn't well expressed - some expansion is needed.

There is also the question of 'how far do you go?' in classification - if I worked on the systematics of rotifers, my papers might be the most important papers in the field of rotifer phylogeny, but presumably that would be too specific a category. As with choice of variables to enter a model, at some point you have to draw a line to be able to compare different scientists meaningfully (if that is your aim). How would you decide this?

Given that the rankings of the journals in terms of impact is pretty much the same comparing overall papers and evolutionary papers, one could argue that the impact factor system in qualitative terms is actually working quite well, from the point of view of comparing within evolutionary biologists at least, and that this should be the way to proceed. Perhaps you should emphasise that the problem really lies when comparing scientists as a whole - something that research assessment exercises (such as the upcoming RQF, here in Australia) do need to take into account.

There have been other papers (I can't remember their references) that have suggested modifications to the impact factor system based on field of study. Some reference/discussion to them might be warranted.

-----

N.B. These are the general comments made by the reviewer when reviewing this paper in light of which the manuscript was revised. Specific points addressed during revision of the paper are not shown.