Introduction

Despite the fact that most scientists, funding organisations, promotion committees and journal editors are very much aware of the numerous problems associated with the use of journal impact factors as a measure of scientific quality or even impact [1]–[3], impact factors continue to be amongst the most commonly used measures of journal quality, and thereby of individual papers and their authors. Consequently, researchers take the impact factor of a journal into account when deciding where to submit their work for publication, while journal editors may have the impact factor of their journal in mind when deciding which manuscripts to accept for publication [3].

One of the main problems associated with the use of impact factors as an objective measure of either quality or impact are the large and systematic differences that exist among different scientific disciplines, with impact factors of evolutionary and ecological journals being on the low end of the spectrum [4], [5]. It is these systematic differences among scientific disciplines, which are unrelated to the quality or the size of the field, that make it impossible to directly compare impact factors of journals from different fields [4]–[6].

Indeed, for its annual Journal Citation Reports (JCR), ISI Scientific categorises journals by subject, and journals typically advertise not only their impact factor, but also their ranking within their subject category. Unfortunately however, in a diverse and multidisciplinary field like evolutionary biology, these categories, or any similarly non-hierarchical classification system in which journals can only be a member of a few categories at most for that matter, is far from satisfactory. For example, of the about 5000 articles published between 1996 and 2006 with the term “sexual selection” in either the title, keywords or abstract, only one-fifth was published in journals that are classified by ISI Scientific under ‘Evolutionary Biology’ (542 of which were published in Journal of Evolutionary Biology or Evolution), while 423 were published in Proceedings of the Royal Society B (classified under Biology), and 392 in Animal Behaviour (classified under Zoology and Behavioural Sciences). Consequently, we simply cannot avoid making comparisons among journals from a wide range of subject categories, including Evolutionary Biology, but also, for example, Zoology, Ornithology and Genetics & Heredity.

The problem is particularly obvious when comparing impact factors of biological journals on the one hand, and those journals that are classified under Multidisciplinary Sciences, which by their very nature publish articles from a wide range of scientific fields. Indeed, there is a general feeling that the extremely high impact factors of multidisciplinary journals like Nature and Science are largely driven by the non-evolutionary and/or non-biological papers they publish [also see e.g. 7]. If this was indeed true, this would imply that, at least for an evolutionary biologist, publishing in these journals is currently overvalued. On the other hand, however, the impact factor of journals that are narrower in focus, and in particular of those limited to a particular taxon, may well underestimate the impact of the evolutionary papers they publish. In other words, what is the true impact of evolutionary papers in non-evolutionary journals?

Results

Not surprisingly, there are highly significant differences among journals in the impact of the evolutionary papers they publish, as well as in their overall impact (Figure 1A). More interestingly, however, the size and/or direction of the difference between these two measures of impact varies among journals, as is indicated by the significant interaction between interaction between Evolutionary vs. Overall impact and Journal in Table 1. In other words, the overall impact of a journal does not necessarily provide a good predictor of the impact of the evolutionary papers it publishes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. The impact of evolutionary papers in evolutionary and non-evolutionary journals.

A) The mean absolute impact of evolutionary papers for 39 journals from a range of different categories. The dotted line gives their overall impact. B) The impact of evolutionary articles relative to the impact of the average article for each of these journals. Relative impacts marked with an asterisk are significantly different from zero at the 5% level.

https://doi.org/10.1371/journal.pone.0000999.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Differences between the average impact of the evolutionary papers a journal publishes and its overall impact factor.

https://doi.org/10.1371/journal.pone.0000999.t001

Indeed, Figure 1B shows clearly that for many journals there are large and often significant differences between the impact of the evolutionary articles a journal publishes and a journal's overall impact. While the impact of evolutionary articles published in journals classified under Multidisciplinary Sciences are typically between 15 to 30% lower than the impact of the average paper that is published in these journals, evolutionary articles in many of the more specialized, but not explicitly evolutionary, journals are up to 40% percent higher.

Interestingly, also the relative impact of ‘evolutionary’ articles (i.e. an article with one of the ten evolutionary keywords in title, abstract or keywords) published in Evolution and Journal of Evolutionary Biology is significantly greater than zero (although less than 10%). This implies that the ten keywords used to classify an article as ‘evolutionary’ (see Materials and Methods for more details) may not be representative of the whole field of evolutionary biology. To further test for differences among fields within evolutionary biology, we can compare the impact of papers on, for example, sexual selection, natural selection or speciation. Indeed, we find that, at least for papers on sexual selection and natural selection, there are significant differences among journals over and above the differences among journals in the impact of evolutionary papers in general (Table 2). In other words, those journals in which papers on sexual selection have a relatively high impact are not necessarily the same journals in which papers on natural selection have a relatively high impact.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Differences in the impact of articles on A) sexual selection, B) natural selection, and C) speciation, relative to the impact of evolutionary papers as a whole.

https://doi.org/10.1371/journal.pone.0000999.t002

Nevertheless, while the above analyses unequivocally show that the overall impact of a journal can substantially under- or overestimate the impact of the evolutionary papers that it publishes, their overall ranking remains remarkably unaffected. A brief look at Figure 1A shows that, compared to the average Evolution paper, evolutionary papers published in Nature or Science do still attract three to four times as many citations in the first two years after their publication. Also if we look at their ranking in more detail, we find that limiting ourselves to the impact of evolutionary papers has in fact only very little effect on a journal's relative ranking, and both rankings are strongly correlated across journals (r = 0.96, P<0.001), particularly for the journals with a relatively high overall impact (Figure 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. The ranking of journals across (sub-) disciplines.

To improve clarity, journals with a very low number of publications between 1996 and 2005 in any of the three sub-disciplines were excluded. As in Figure 1, journals with black dots are classified as Multidsciplinary Sciences, journals with grey dots are classified under Evolutionary Biology, and journals classified under any other subject category are indicated with an open dot.

https://doi.org/10.1371/journal.pone.0000999.g002

Discussion

Despite their many inherent problems and limitations, impact factors continue to be used by funding agencies and employers to assess scientists, and this is unlikely to change any time soon. This is at least to some extend justified as impact factors do eliminate some of the bias of other citations statistics, which are more sensitive to the number of papers a journal publishes each year, or the age of a journal. Furthermore, although it may by no means provide a complete measure of impact or quality, a high impact factor, and thus a high average citation rate (in the first years after publication) is by all means a good thing, and a-priori it thus seems reasonable to value a high impact journal over a journal with a lower impact. Although the impact of evolutionary articles published in journals like Nature, Science and PNAS is relatively low compared to the impact of other papers that are published in those journals, which implies that evolutionary papers in any of these journals should maybe not receive as much weight as they do currently, they do indeed have by far the highest impact. Based on this, these journals can indeed be considered to be the most important journals within the field of evolutionary biology.

Evolutionary papers are published in a wide range of journals, most of which are not classified under Evolutionary biology. Here I showed that, within these non-evolutionary journals, there are systematic differences in impact among evolutionary and non-evolutionary papers, and to some extent even among different subjects within evolutionary biology. Consequently, quantitative comparisons of journals on the basis of a single impact statistic are fraught with problems for a field like evolutionary biology. If we want to be able to make meaningful comparisons among journals, we will need a more detailed and sophisticated system in which individual papers, rather than journals, are classified into subject categories. Furthermore, these categories should have a hierarchical structure in which papers can be a member of many different categories. For example, a paper on bird behaviour would be a member of Ornithology and Behavioural Sciences, but automatically also of Zoology and Biology. Provided the number of papers is sufficiently large, this allows us to calculate separate impact statistics for each of these categories, resulting in multiple, subject-specific impact statistics per journal. This would not only allow for fairer comparisons among journals and researchers, it would also aid authors in maximising the impact of their research, while it would not ‘punish’ editors of high-impact journals for publishing evolutionary or ecological papers.

While I have here focussed on the impact of evolutionary papers in non-evolutionary journals, the approach outlined here can easily be applied to other areas of research and to other journals. This will provide more insight into the generality of the patterns described here, and the extent to which journal impact factors provide biased estimates of impact, either up- or downward. This will help us to interpret variation in journal impact factors among journals and fields, which will hopefully contribute to a fairer assessment of the quality of publications, individual researchers and departments.

Materials and Methods

Acknowledgments

I thank all members of the Evolutionary Ecology of Sexual Reproduction Research Group journal club for many stimulating discussions that provided the inspiration for this analysis, and Barbara Tschirren, Rob Brooks and Matthew Symonds for their comments on previous versions of this manuscript.