Journal Impact Factor: Do the Numerator and Denominator Need Correction?

To correct the incongruence of document types between the numerator and denominator in the traditional impact factor (IF), we make a corresponding adjustment to its formula and present five corrective IFs: IFTotal/Total, IFTotal/AREL, IFAR/AR, IFAREL/AR, and IFAREL/AREL. Based on a survey of researchers in the fields of ophthalmology and mathematics, we obtained the real impact ranking of sample journals in the minds of peer experts. The correlations between various IFs and questionnaire score were analyzed to verify their journal evaluation effects. The results show that it is scientific and reasonable to use five corrective IFs for journal evaluation for both ophthalmology and mathematics. For ophthalmology, the journal evaluation effects of the five corrective IFs are superior than those of traditional IF: the corrective effect of IFAR/AR is the best, IFAREL/AR is better than IFTotal/Total, followed by IFTotal/AREL, and IFAREL/AREL. For mathematics, the journal evaluation effect of traditional IF is superior than those of the five corrective IFs: the corrective effect of IFTotal/Total is best, IFAREL/AR is better than IFTotal/AREL and IFAREL/AREL, and the corrective effect of IFAR/AR is the worst. In conclusion, not all disciplinary journal IF need correction. The results in the current paper show that to correct the IF of ophthalmologic journals may be valuable, but it seems to be meaningless for mathematic journals.


Introduction
One of the most famous researchers in bibliometrics, Garfield [1] first proposed the term of impact factor (IF) in his paper published in Science in 1955. At first, IF meant citations to articles, not the current IF used for evaluating journals [2]. Journal IF is the natural result of the establishment of the Science Citation Index (SCI) database. In the 1960s, supported by the US National Institutes of Health, the Genetics Citation Index was established successfully, which led to the direct generation of the SCI database. Garfield and Sher then proposed IF as an indicator for evaluating the academic impact of journals in 1963, and IF was applied to assist the selection of source journals in the SCI database [3][4][5]. In 1972, Garfield [6] formally established the concept and formula of journal IF. However, IF really became a journal evaluation indicator in 1975, when the Journal Citation Reports (JCR) was established, and ever since has been one of the most authoritative scientometric indicators for assessing the international status and academic impact of journals [7].
To be exact, IF is defined as the number of citations within a given year of items published by a journal in the preceding two years divided by the number of citable items published by the journal in these two years [8]. Obviously, the numerator of IF is the number of citations of all types of items, whereas the denominator just includes citable items. Thus, it is undeniable that the design formula of IF is not perfect, and this fact has attracted much attention and aroused much controversy in academia.
Thomson Reuters divides source items into citable items and non-citable items. Citable items include only original research articles and reviews. Meeting abstracts, editorials, letters, news items, corrections, book reviews, biographical items, reprints, and so on are all rated as non-citable items [9][10][11]. In fact, non-citable items are not uncitable, some that are published by many journals, especially several prestigious international journals, are cited frequently and do contribute to a journal's citations [12][13]. When these non-citable items are cited, their citations are counted in the numerator of IF, but are excluded in the denominator. Therefore, citations to non-citable items are usually known as a "free lunch" [14][15][16][17].
Many scholars have noticed this defect of the IF and questioned its fairness and reasonability for journal evaluation. Bensman [18] indicated that distinguishing citable and non-citable items was a fatal flaw that has existed since IF was first presented. Some scholars like Campanario [19][20][21] believe IF is a biased indicator and have pointed out that non-citable items have an important influence on it, and as a consequence the IF of a considerable number of journals including some of the most prestigious ones might be inflated by 30%-40%. Some researchers have pointed out that the incongruence of source items between the numerator and denominator seriously distort the true IF value of journals, and cause its manipulation and abuse [22][23]. More incredibly, one has pointed out that the number of the citable items in the denominator could be negotiated with Thomson Reuter, which might result in an IF variation of more than 300% [24][25].
Some scholars have presented several suggestions and solutions based on the inconsistency of source items in IF, and these fall into three main approaches, as follows: 1. Releasing new journal evaluation indicators. The Scopus database launched the SJR and SNIP indicators in 2007 and 2010. Similar to IF, these two new indicators are average citations to journals. However, they break some of the limitations of IF, such as prolonging the citation time window to three years and considering the citation peaks of different subjects. Among them, one of the most obvious advantages is their consistency in the source items between numerator and denominator, which counts reviews and articles, including proceedings papers [26][27]. Based on the principle and algorithm of weights based on order relation and experts' suggestions, Xu and Fang [28] constructed a weighted IF that changed the order relation of different document types, including non-citable items.
reviews in the numerator only. Some have suggested redefining citable items such as articles, notes, letters, and reviews and just count citations to these four source items [33]. Lozano et al. [34] corrected the inconsistency of document types between the numerator and denominator when analyzing the correlation between IF and citations in the digital age.
Even so, the current studies on the inconsistency between the numerator and denominator of the IF mainly focus on academic criticisms, proposals for modifications, and selecting sample journals to compute the corrected IF and compare it with traditional IF. These studies mainly remain at theory stage, and there is no empirical research on the journal evaluation effect of corrective IFs.
In the current study, based on the quantitative and cited characteristics of non-citable items, we have selected ophthalmologic journals and mathematical journals as research objects to correct the design formula of traditional IF. Considering the fact that scholars in a country might be not familiar with some journals from other countries, we have investigated only American researchers in the fields of ophthalmology and mathematics to obtain the real impact ranking of American ophthalmologic and mathematical journals in the minds of peer experts as the "gold standard" of journal impact. The correlations between the corrective IFs and questionnaire score were analyzed to verify the corrective effect of IF in different subjects to help us determine if there is a need to correct the numerator and denominator of traditional IF. These results have important theoretical significance and practical worth to the whole research community.

Determining research objects
There are significant differences in citation behavior between different subjects. For example, mathematics usually has a later citation peak and is known as a slow moving discipline, but papers in some disciplines in medicine often reach their citation peaks quickly [35][36][37]. Considering that research classification and discipline boundary in ophthalmology are relatively definite and clear, we selected ophthalmologic and mathematical journals as our research objects. As mentioned before, scholars in a country might be not familiar with some journals from other countries, considering America had a large number of journals included in the JCR database in 2014, we meant to investigate American researchers in the fields of ophthalmology and mathematics to obtain the real impact ranking of American ophthalmologic and mathematical journals in the minds of peer experts. There were 30 ophthalmologic journals and 82 mathematical journals published in America. Considering a large sample size may influence the questionnaire results, we just selected comprehensive mathematical journals for our study, 27 journals.

Obtaining the quantity and citations of different types of document
We retrieved all types of document published by each ophthalmologic and mathematical journal from 2012 to 2013, obtained the quantity per type of document through the "Refine" function in the SCI database, and obtained their citations in 2014 using the "Create Citation Report" function.

Corrective methods of traditional IF
Based on the quantitative and citation characteristics of different type of documents in the SCI database, we propose five corrective definitions for IF as follows: In formulae (1) and (2), C t-1 and C t-2 are citations given in year t to all source items published in years (t-1) and (t-2), respectively, N t-1 and N t-2 are the numbers of all source items published in years (t-1) and (t-2), respectively, N AREL(t-1) and N AREL(t-2) are the numbers of items including articles, reviews, editorials, and letters published in years (t-1) and (t-2), respectively. In formula (3), C AR(t-1) and C AR(t-2) are citations in year t given to articles and reviews published in years (t-1) and (t-2), respectively, N AR(t-1) and N AR(t-2) are the numbers of articles and reviews published in years (t-1) and (t-2), respectively. In formulae (4) and (5), C AREL(t-1) and C AREL(t-2) are citations in year t given to items including articles, reviews, editorials, and letters published in years (t-1) and (t-2), respectively, and the variables N AR(t-1) , N AR(t-2) , N AREL(t-1) , and N AREL(t-2) are defined similarly to those in formulae (2) and (3).

Questionnaire survey
First, we selected American researchers as corresponding authors who have published ophthalmologic papers in the last 10 years or mathematical papers in the last five years in the SCI database as our respondents, and we obtained their email addresses by the field of 'E-mail Addresses' provided by WoS database. We obtained 9145 email addresses of ophthalmologic researchers and 7810 email addresses of mathematical researchers. We then designed a questionnaire survey about journal academic impact through the AskForm platform. Finally, we sent emails to those researchers, explained our research object, and requested them to answer the questionnaire via the provided web link. We sent successfully 7077 e-mails to American ophthalmologic researchers and 5136 e-mails to American mathematical researchers. In the questionnaire, researchers gave a score from 1.0-10.0 (the score is accurate to 1 decimal place) to each journal according to its academic impact. A score of 10 is the highest and 1 is the lowest. We calculated the total score of each journal as its real impact standard. We received 124 responses for the ophthalmologic journals and 123 responses for the mathematical journals. We excluded the questionnaires that scored less than three journals, gave the same score to all journals, or gave a score to each journal according to the sequence in which they were presented. Finally, we obtained 112 valid questionnaires for the ophthalmologic journals and 117 valid questionnaires for mathematical journals. The survey was conducted from August 4, 2015 to September 15, 2015. Statistical analysis method SPSS 22.0 was used to analyze the relationships between the corrective IFs and questionnaire score using the Spearman rank correlation test.

Results and Discussion Ophthalmologic Journal Results
Quantity and citations per type of document published by 30 ophthalmologic journals. There are significant differences in the quantity of and number of citations to each type of document for different journals. The quantity and number of citations of each type of document published by 30 ophthalmologic journals from 2012 to 2013 are shown in Table 1.
Overall, these ophthalmologic journals mainly published articles, followed by letters, editorials, and reviews. Articles received the highest number of citations, followed by reviews, letters, and editorials. Although the total number of letters is far higher than that of the editorials, the number of citations to both are nearly identical. Therefore, for ophthalmologic journals, editorials are more often cited than letters. As non-citable items, some editorials and letters can be cited, but biographical items, corrections, and news items are hardly ever cited. Surv Ophthalmol and Curr Opin Ophthalmol published many reviews and few editorials and letters, Surv Ophthalmol only published one article, and Curr Opin Ophthalmol did not publish any articles.
Ophthalmologic journal questionnaire scores and corrective IFs. We analyzed 112 valid questionnaires and obtained total expert score for the ophthalmologic journals. In addition, we calculated the five corrective IF values for each journal and retrieved its traditional IF through the JCR database. The results can be seen in Table 2.
Correlations between the questionnaire scores and various IFs of 30 ophthalmologic journals. The evaluation of an expert in the field is considered to be the most important criterion for verifying the validity of citation indicators, and the true impact of different journals in researchers' true estimation can be reflected by the expert scores in the questionnaire surveys [38,39]. Hence, we can verify the effect of journal evaluation for each corrective IF by analyzing the correlations between them and the corresponding expert score.
First, we illustrate whether it is reasonable to use corrective IFs for journal valuation by looking at the correlations between questionnaire score and the five corrective IFs defined above. The scatter diagrams of these relationships are presented in Fig 1. In addition, the K-S normal distribution test was conducted using the variables, and the results show that IF AREL/AR is not subject to a normal distribution. Thus, we analyzed the relationships between the questionnaire score and various IFs using the Spearman rank correlation test. The results are shown in Table 3. Table 3 shows that the five corrective IFs and traditional IF are significantly correlated with the questionnaire score for the 30 ophthalmologic journals. The correlation coefficients between traditional IF and the five corrective IFs are above 0.9. The traditional IF is most relevant to IF AREL/AR and IF AR/AR , for which the correlation coefficients are 0.967 and 0.966, respectively. There are high correlations between the five corrective IFs. The correlation coefficients between IF Total/Total and both IF Total/AREL and IF AREL/AREL reach 0.999. IF AR/AR is the most relevant to IF AREL/AR , and IF Total/AREL is most relevant to IF AREL/AREL . The questionnaire score is more relevant to the five corrective IFs than to traditional IF. IF AR/AR is the most relevant to the questionnaire score, with a correlation coefficient of 0.715, followed by IF AREL/AR , for which the correlation coefficient is 0.713. The correlation coefficient between IF Total/AREL and the questionnaire score is the same as that between IF AREL/AREL and the questionnaire score, and the correlation coefficients for both are 0.692.

Mathematical Journal Results
Quantity and citations for each type of document published by 27 mathematical journals. To determine the distribution of non-citable items in mathematics, we analyzed the quantity and number of citations for each type of document published by 27 mathematical journals. The results are shown in Table 4. Table 4 shows that these mathematical journals mainly published articles, and only Commun Pur Appl Math published one review from 2012 to 2013. There were 153 editorials and 123 letters published by the mathematical journals, but only 78 other non-citable items. Compared with the number of articles, the number of non-citable items is very small. The citations to articles are far higher in number than those to non-citable items. The review published by Commun Pur Appl Math was not cited in 2014. The citations to the 153 editorials in 2014 numbered only 13, the 123 letters were just cited once in 2014, the citations to other non-citable items numbered only three. Optometry Vision Sci Questionnaire score and various IFs of 27 mathematical journals. We analyzed 117 valid questionnaires to obtain the total expert score of 27 mathematical journals. In addition, we calculated the five corrective IF values for each journal and retrieved its traditional IF through the JCR database. The results are shown in Table 5.
Correlations between questionnaire score and various IFs of 27 mathematical journals. Fig 2 shows the scatter diagrams of the relationships between the questionnaire score and various IFs of the mathematical journals. In addition, the K-S normal distribution test results show that the five corrective IFs and traditional IF are not subject to a normal distribution. Thus, we analyzed the relationships between the questionnaire score and corrective IFs using the Spearman rank correlation test. The results are shown in Table 6. Table 6 shows that the five corrective IFs and traditional IF are significantly correlated with questionnaire score for 27 mathematical journals. The correlation coefficients between the traditional IF and corrective IFs vary between 0.8 and 0.9. The traditional IF is most relevant to IF Total/AREL and IF AREL/AREL , for which the correlation coefficients are both 0.865. Similar to ophthalmology, there are high correlations between the five corrective IFs. IF Total/Total is most corrective IFs. The correlations between corrective IFs and the questionnaire score of the ophthalmologic journals are higher than those of the mathematical journals. For ophthalmologic journals, the effect of journal evaluation shows that the corrective IFs are superior to traditional IF; however, there is little value to using the corrective IFs for mathematical journals. The reason for this could be that ophthalmologic journals publish more non-citable items that attract more citations, while mathematical journals publish fewer non-citable items and these non-citable items have fewer citations. Therefore, the corrective effect of traditional IF might be related to the quantitative and citation characteristics of non-citable items published by journals in different disciplines.

Five Corrective IFs and traditional IF are highly correlated
For both ophthalmologic and mathematical journals, the five corrective IFs are significantly positively and highly correlated with traditional IF. Although both ophthalmologic journals and mathematical journals publish a certain number of non-citable items, their quantities and number of citations are very low compared with those of articles. The numerator and/or denominator in the traditional IF are correspondingly improved. Thus, although there are some differences in the quantity of source items used for the corrective IFs and traditional IF, these differences are not significant. For both ophthalmology and mathematics, the journal ranks based on the corrective IFs and traditional IF are largely consistent, especially for the mathematical journals. In general, the correlations between the five corrective IFs and traditional IF for mathematical journals are lower than those for ophthalmologic journals. For ophthalmologic journals, the source items in the denominator for IF AR/AR and IF AREL/AR , which correct the numerator of traditional IF, are the same as traditional IF. Among the non-citable items published by ophthalmologic journals, letters and editorials attract more citations; other non-citable items are rarely cited. Nevertheless, the number of citations to letters and editorials are much lower than those to articles and reviews. Hence, IF AR/AR and IF AREL/AR are highly correlated with traditional IF. The research results for mathematical journals are totally different from those of ophthalmologic journals. For mathematical journals, IF Total/AREL and IF AREL/AREL are the most correlated with traditional IF. IF Total/AREL is constructed by re-defining the denominator in traditional IF, and IF AREL/AREL is constructed by simultaneously correcting both numerator and denominator in the traditional IF. Therefore, because of the difference in subject nature, the correlations between corrective IFs and the traditional IF of journals are clearly different. The correlations between corrective IFs and traditional IF in other disciplines have yet to be studied.

Five corrective IFs are highly correlated
The statistical analysis shows that there are high correlations between the five corrective IFs for both ophthalmologic and mathematical journals. For ophthalmologic journals, the only difference between IF Total/Total and IF Total/AREL is the denominator. Because document types  published by ophthalmologic journals are relatively concentrated into articles, reviews, letters, and editorials, IF Total/Total is highly correlated with IF Total/AREL . Articles, reviews, letters, and editorials have more citations than other non-citable items, so IF AREL/AREL is highly correlated with IF Total/Total . In addition, citations to letters and editorials are far fewer than to articles and reviews, thus IF AREL/AR is highly correlated with IF AR/AR . The research results for mathematical journals are the same as those for ophthalmologic journals.

It is scientific and reasonable to use five corrective IFs for journal evaluation
The scatter plots show that there are significant linear correlations between the five corrective IFs and questionnaire score for both ophthalmologic and mathematical journals. Further statistical analysis shows that for both ophthalmologic journals and mathematical journals, the five corrective IFs are significantly correlated with the questionnaire score. For ophthalmologic journals, the correlation coefficients between the various corrective IFs and questionnaire score vary from 0.6 to 0.8. For mathematical journals, the correlation coefficients vary from 0.4 to 0.6. Therefore, for both ophthalmologic and mathematical journals, it is scientific and feasible to use all the corrective IFs for journal evaluation. Corrective effect of journal IF in different subjects is obviously different The results of the study show that the correlations between five corrective IFs and questionnaire score for ophthalmologic journals are obviously higher than those for mathematical journals. For ophthalmologic journals, five corrective IFs are more relevant to the questionnaire score than to traditional IF. Therefore, the journal evaluation using the five corrective IFs is better than an evaluation using traditional IF. IF AR/AR is the most relevant to the questionnaire score, followed by IF AREL/AR . Thus, we conclude that the corrective effect of IF AR/AR is best for ophthalmologic journals, IF AREL/AR is better than IF Total/Total , and the effects on journal evaluation from using IF Total/AREL and IF AREL/AREL are the worst. For mathematical journals, journal evaluation using the traditional IF is superior than that using corrective IFs. Among the five corrective IFs, the effect on journal evaluation of IF Total/Total is the best, IF AREL/AR is better than IF Total/AREL and IF AREL/AREL , and the effect in journal evaluation of IF AR/AR is the worst.
In conclusion, although the assumptions behind correcting traditional IF are reasonable in theory, our empirical study finds that not all subjects need to correct their journal IF. In our study, it is valuable to correct the traditional IF of ophthalmologic journals; however, it may not make much sense to correct the traditional IF of mathematical journals. Overall, the quantity and citations of non-citable items are lower than those of citable items. Hence, the corrective effect of IF might be not obvious for most disciplines, but it might be valuable to correct the traditional IF of journals publishing more editorials and letters.
Supporting Information S1 Dataset. Data of questionnaire survey from American ophthalmologic researchers giving each American ophthalmology journal a score.