^{1}

^{3}

^{2}

^{3}

^{2}

^{3}

^{*}

Conceived and designed the experiments: LA MS-P MS. Performed the experiments: MS-P MS. Analyzed the data: LA MS-P MS. Contributed reagents/materials/analysis tools: LA MS-P MS. Wrote the paper: LA MS-P MS.

The authors have declared that no competing interests exist.

The rise of electronic publishing

Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.

Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

As de Solla Price observed

This burden does not only affect researchers. Funding agencies, university administrators, and reviewers are called on to evaluate the productivity of researchers and institutions, as well as the impact of their work. Typically, these agents have neither the time nor the financial resources to obtain an in-depth evaluation of the actual research and must instead use indirect indicators of quality such as number of publications,

Despite the oversimplification of using just a few numbers to quantify the scientific merit of a body of research, the entire science and technology community is relying more and more on citation-based statistics as a tool for evaluating the research quality of individuals and institutions

This act of “judging a book by its cover” has caused researchers to note that we should judge a paper not by the number of citations that the journal in which it is published receives, but by the number of citations the paper itself receives

The need for an estimate of the ultimate impact of recently published articles is the reason that the JIF is often used as a proxy for quality of the research. Indeed, the premise of the peer-reviewing process is that reviewers are in fact able to assess the quality of a paper. Thus, the heuristic that the journal in which a paper is published is a good proxy for the ultimate impact of a paper is likely to be an adaptive one

Like any heuristic, the evaluation of research using citation analysis has weaknesses. These weaknesses have been extensively explored in the literature

In this article, we investigate two fundamental aspects concerning the prediction of the ultimate impact of a published research paper: (i) the time scale

We obtained the number of citations accrued by December 31, 2006 for 22,951,535 papers tracked in Thomson Scientific's Web of Science® (WoS) database. This database comprises information on papers published in ∼5,800 science and engineering journals, ∼1,700 social science journals, and ∼1,100 arts and humanities journals. Journals are typically covered from their inception or from the beginning of the WoS coverage for the research area (whichever is later) until the present date or until their demise (whichever is earlier). The beginning of WoS coverage for science and engineering, social science, and arts and humanities is 1955, 1956, and 1975 respectively. In this study, we restrict our analysis to journals publishing at least 50 articles per year for at least 15 years. This condition restricts our analysis to 19,372,228 articles published in 2,267 journals, and enables us to ensure good statistics on the journals that we include in the analysis. More information about the data is included in

Because the citation history of a paper may be field- and even journal-dependent, we first investigate

(A) Probability density function _{10}(

These results are not restricted to the

The existence of a steady state for

In order to explain our empirical findings, we develop a model for the asymptotic number of citations a paper published in journal ^{q}^{q}

One can interpret ^{q+ε}_{ε}

(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean _{10}(1+

Our finding that the distribution of number of citations is log-normal is in agreement with recent generative models of the citation network

The findings reported in

Our findings thus suggest the possibility of ranking journals according to

We rank journals in different fields according to both

We present results for 13 journals that the ISI classifies primarily in experimental psychology, and 36 journals that the ISI classifies primarily in ecology (see _{i}_{j}_{ij}_{i}_{j}^{AUC}, that is, the ranking that maximizes _{ij}_{ij}

Our analysis also demonstrates that the

(A) Comparison of the rankings of journals obtained using the JIF and the AUC statistic. Though there are clear correlations between the two rankings, deviations can be extremely large. (B) Probability density function of ΔR(^{JIF}(^{AUC}(

Rank | n | Steady state | ||||||

AUC | JIF | Journal abbreviation | q̅ | n̅ | Q2 | JIF | period | |

1 | 1 | ECOLOGY | 1.75 | 0.33 | 71.1 | 52 | 4.782 | 1974–1994 |

2 | 2 | AM NAT | 1.72 | 0.40 | 80.4 | 48 | 4.660 | 1967–1992 |

3 | 4 | EVOLUTION | 1.67 | 0.35 | 69.8 | 43 | 4.292 | 1973–1993 |

4 | 14 | BEHAV ECOL SOCIOBIOL | 1.60 | 0.31 | 44.4 | 36 | 2.316 | 1978–1990 |

5 | 8 | J ANIM ECOL | 1.57 | 0.34 | 47.5 | 33 | 3.390 | 1954–1996 |

6 | 5 | J ECOL | 1.55 | 0.35 | 45.1 | 32 | 4.239 | 1973–1996 |

7 | 15 | MAR ECOL-PROG SER | 1.47 | 0.31 | 33.6 | 26 | 2.286 | 1991–1995 |

8 | 6 | CONSERV BIOL | 1.42 | 0.42 | 37.5 | 23 | 3.762 | 1988–1998 |

9 | 7 | FUNCT ECOL | 1.42 | 0.32 | 29.6 | 23 | 3.417 | 1989–1996 |

10 | 9 | OIKOS | 1.41 | 0.35 | 34.2 | 22 | 3.381 | 1974–1995 |

11 | 10 | OECOLOGIA | 1.40 | 0.29 | 27.5 | 22 | 3.333 | 1994–1997 |

12 | 17 | J EXP MAR BIOL ECOL | 1.31 | 0.30 | 23.5 | 18 | 1.919 | 1988–1995 |

13 | 3 | J APPL ECOL | 1.31 | 0.36 | 25.6 | 17 | 4.527 | 1965–2000 |

14 | 23 | BIOTROPICA | 1.30 | 0.38 | 25.5 | 17 | 1.391 | 1975–1994 |

15 | 13 | J VEG SCI | 1.28 | 0.36 | 22.4 | 16 | 2.382 | 1989–1999 |

16 | 22 | POLAR BIOL | 1.27 | 0.33 | 20.8 | 16 | 1.502 | 1981–1994 |

17 | 28 | ENVIRON BIOL FISH | 1.24 | 0.42 | 21.0 | 15 | 0.934 | 1981–1990 |

18 | 12 | BIOL CONSERV | 1.21 | 0.38 | 22.1 | 14 | 2.854 | 1988–1996 |

19 | 11 | J BIOGEOGR | 1.21 | 0.36 | 20.4 | 13 | 2.878 | 1976–1998 |

20 | 21 | J WILDLIFE MANAGE | 1.19 | 0.34 | 18.3 | 13 | 1.538 | 1984–1995 |

21 | 18 | J CHEM ECOL | 1.11 | 0.31 | 14.4 | 10 | 1.896 | 1995–1998 |

22 | 32 | AM MIDL NAT | 1.07 | 0.36 | 14.5 | 9 | 0.667 | 1964–1995 |

23 | 26 | WILDLIFE RES | 1.05 | 0.32 | 11.6 | 9 | 1.032 | 1990–1997 |

24 | 24 | PEDOBIOLOGIA | 0.99 | 0.41 | 12.8 | 8 | 1.347 | 1965–1997 |

25 | 20 | AGR ECOSYST ENVIRON | 0.97 | 0.44 | 11.4 | 7 | 1.832 | 1982–2001 |

26 | 19 | ECOL MODEL | 0.91 | 0.40 | 11.3 | 6 | 1.888 | 1977–1998 |

27 | 30 | J RANGE MANAGE | 0.90 | 0.37 | 9.9 | 6 | 0.859 | 1966–1995 |

28 | 29 | BIOCHEM SYST ECOL | 0.89 | 0.36 | 9.2 | 6 | 0.906 | 1980–1994 |

29 | 31 | WILDLIFE SOC B | 0.87 | 0.40 | 8.7 | 6 | 0.843 | 1983–1998 |

30 | 25 | J ARID ENVIRON | 0.83 | 0.38 | 7.8 | 5 | 1.238 | 1989–2000 |

31 | 34 | SOUTHWEST NAT | 0.72 | 0.38 | 5.9 | 4 | 0.309 | 1980–1994 |

32 | 33 | J NAT HIST | 0.72 | 0.38 | 6.4 | 4 | 0.631 | 1966–2000 |

33 | 35 | CAN FIELD NAT | 0.69 | 0.40 | 5.8 | 3 | 0.073 | 1983–1993 |

34 | 16 | LANDSCAPE URBAN PLAN | 0.67 | 0.41 | 5.3 | 3 | 2.029 | 1985–2004 |

35 | 27 | J SOIL WATER CONSERV | 0.65 | 0.49 | 7.0 | 3 | 0.949 | 1966–2002 |

36 | 36 | NAT HIST | -0.32 | 0.44 | 0.3 | 0 | 0.059 | 1989–2005 |

We consider the 36 journals that are primarily classified in the field of ecology according to the ISI. We rank journals according to: (i) the maximization of the multi-class AUC statistic for the steady-state distributions

Rank | n | Steady state | ||||||

AUC | JIF | Journal abbreviation | q̅ | n̅ | Q2 | JIF | period | |

1 | 4 | J EXP PSYCHOL LEARN | 1.55 | 0.35 | 47.5 | 34 | 2.601 | 1992–1995 |

2 | 6 | J EXP PSYCHOL HUMAN | 1.56 | 0.38 | 52.1 | 32 | 2.261 | 1974–1995 |

3 | 2 | PSYCHOPHYSIOLOGY | 1.47 | 0.36 | 41.8 | 27 | 3.159 | 1985–1995 |

4 | 1 | NEUROPSYCHOLOGIA | 1.48 | 0.41 | 48.6 | 27 | 3.924 | 1964–1995 |

5 | 10 | MEM COGNITION | 1.38 | 0.40 | 34.0 | 21 | 1.512 | 1977–1997 |

6 | 5 | BRAIN LANG | 1.25 | 0.33 | 22.9 | 16 | 2.317 | 1992–1997 |

7 | 12 | J EXP ANAL BEHAV | 1.22 | 0.38 | 23.8 | 14 | 1.221 | 1970–1991 |

8 | 11 | PERCEPT PSYCHOPHYS | 1.20 | 0.41 | 23.5 | 13 | 1.482 | 1965–1996 |

9 | 8 | J EXP CHILD PSYCHOL | 1.15 | 0.39 | 20.0 | 12 | 2.062 | 1963–1999 |

10 | 9 | PERCEPTION | 1.07 | 0.45 | 17.7 | 9 | 1.585 | 1973–1995 |

11 | 7 | ACTA PSYCHOL | 0.84 | 0.55 | 13.2 | 5 | 2.094 | 1955–2001 |

12 | 3 | BRAIN COGNITION | 0.73 | 0.61 | 9.1 | 3 | 2.858 | 1995–1999 |

13 | 13 | PERCEPT MOTOR SKILL | 0.54 | 0.42 | 4.5 | 2 | 0.333 | 1970–1995 |

We consider the 13 journals that are primarily classified in the field of experimental psychology according to the ISI. We rank journals according to: (i) the maximization of the multi-class AUC statistic for the steady-state distributions

The bias introduced by the JIF arises directly from the major methodological problems raised against using citation analysis to evaluate journals. First, the mean number of citations to papers published in a journal is not representative of the number of citations to each individual paper

Our findings provide a quantitative measure of the efficacy of academic journals, through the work of editors and reviewers, at organizing research based on their prediction of the ultimate impact of that research. Even though far from perfect, the journal system and the ranking of journals provides a powerful heuristic with which to locate the research that will ultimately have the largest impact.

We use the time evolution of _{0}, where _{0} is largest value of

We also determine the periods during which the citation distribution is stable. To this end, we compare the citation distribution for all pairs of years using the Kolmogorov-Smirnov test and fit a box-diagonal model to the matrix of _{0}.

For each steady-state citation distribution, our model (Eq. 3) has three parameters that must be estimated: ^{2} statistic (see _{n}

In practice, we bin the empirical data so that we have at least ten data points in each bin. This is especially important for the tails of the distribution. Then, the contribution to ^{2} is

The fitting parameters suggest that _{0} = 0.91±0.02 and _{1} = 1.03±0.02. For simplicity, when comparing properties of the empirical distributions to model predictions (

As shown in ^{2}. The estimate we obtain for

We define the best ordering as the one that maximizes the value of the multi-class AUC statistic. For a set of journals

We denote as _{AB}_{A}_{B}_{AB}_{A}_{B}_{A}_{B}

In principle, one could calculate the multi-class AUC statistic for every permutation of the ordering of journal citation distributions, and choose the ordering that gives the highest value. However, the number of permutations of a sequence of even modest size is unwieldy. Fortunately, in almost all cases, the distributions obey the property of transitivity, that is, if

Supporting information text, and description of other supporting information files.

(0.06 MB PDF)

Citation history for the 2,266 journals included in our analysis in alphabetical order. For a detailed description of the plots see the caption of panel C in

(19.10 MB PDF)

Comparison of ranking schemes for all the fields listed in the WoS.

(12.95 MB PDF)

Fit to the steady-state citation distribution for the 2,266 journals included in our analysis in alphabetical order.

(21.06 MB PDF)

Median change of rank from JIF to optimal ranking for all fields with at least two journals with more than 50 articles published during the steady-state period.

(0.00 MB TXT)

We thank D. Malmgren, D. Stouffer, R. Guimerà, J. Duch, E. Sawardecker, S. Seaver, P. Mcmullen, I. Sirer, S. Ray Majumder, A. Salazar, and B. Uzzi for comments and discussions.