Research Fields, Journals and Articles

Given the large number of research fields related to computational science, we decided to focus on three fields in the area of our expertise and research interests: bioinformatics (BI), computational statistics (CS) and machine learning (ML). We chose seven representative high-ranking journals (Table 1 lists them), and restricted the survey to articles published in these journals between 2010 and 2012. We also decided to focus on a particular topic, namely on supervised classification as an important topic within computational science.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Overview: Published articles dealing with supervised classification problems in the period from 2010 to 2012.

https://doi.org/10.1371/journal.pone.0061562.t001

Obviously, these choices are not random, and consequently neither the journals nor the papers are random samples. These limitations of our study design were necessary (i) to keep the study practically manageable and to allow us to spend the required time on each of the papers, (ii) to ensure that the selected papers are homogeneous enough to be adequately compared, and (iii) to avoid misinterpretations that would inevitably occur for papers outside our field of expertise. However, we want to stress that at any rate, we neither “tuned” the set of journals during the study, nor did we choose them because we expected them to yield a particular results pattern.

The defined survey setting results in a total set of 5100 articles. We then conducted a manual screening of the articles based on their titles to select those dealing with supervised classification, i.e., from a statistical point of view, regression for categorical dependent variables. A manual screening was preferred to a systematic database search based on keywords, because we realized in a small pilot study that many articles do not mention supervised learning or classification as keywords or in the title but, for example, rather the name of the specific learning method which is considered.

Inclusion Criteria

The goal of the study is to explore common scientific practice in comparing supervised classification methods. Therefore, for each article related to supervised classification we first report.

1. whether the article presents a comparison study comparing two or more methods.

Comparison studies can be executed on simulated/artificial data or so-called “real world” data. This could make a difference for the inference of “standard practice rules”, since real world data are known to often behave differently from simulated data; therefore, we report.

2. whether the article includes an evaluation on real data.

Since one of our main claims in the paper is that comparison studies are important to infer “standard practice rules” we focus the survey on articles fulfilling both Condition 1 and Condition 2; therefore, we report for each article.

3. whether the article includes an evaluation of two or more methods on real data.

Comparison Study Features

We designed the following eight observable comparison study features to capture information on common practice in the context of comparison studies based on real data. To address the issue of neutrality (which is essentially a latent unobservable variable), we particularly focused on the differentiation between comparison studies published as part of a paper introducing a new method and comparison studies on existing methods whose contribution is the comparison itself.

Data Analysis

The collected features were examined using descriptive statistics and exploratory data analysis methods. The goal of the data analysis was to quantitatively describe the common practice in the context of comparison studies based on real data. Note that no inferential statistical methods (e.g., statistical significance tests) were used. The limited sample size and the potential dependence between papers (published in the same journal or, e.g. by the same team) make the validity of standard statistical inference methods questionable.

Articles

The initial situation was a population of articles published in the three computational science fields bioinformatics (BI), computational statistics (CS) and machine learning (ML).

The manual screening identified articles based on their titles. The word “classification” or “classifier” was found in 40 of these titles, while 24 titles included the name of a specific method. The words “comparison” and “comparative” were found in four titles. The word “prediction” also appeared in four titles, while the words “learning” and “discrimination” were found in five titles and two titles, respectively. Three papers were excluded after further consideration because they did not fit our definition of supervised classification in spite of their title, finally yielding a total of papers. Their references can be found in File S1.

Out of these articles, articles presented a comparison of two or more methods as displayed in Figure 2A and Figure 2E. Three out of articles did not present any comparison study based on real data, as displayed in Figure 2B and Figure 2F. This resulted in a total of articles satisfying Condition 3, on which we focused in the rest of our survey; see Figure 2C and Figure 2G.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Overview of selected papers and selection criteria.

The top panel shows the global results for all papers relevant to the respective question (either all 59 papers or the 55 papers satisfying Condition 3), while the bottom panel shows the results stratified by research field (bioinformatics: BI, computational statistics: CS and machine learning: ML). (A) and (E): Does the paper include a comparison of two or more methods?; (B) and (F): Does the paper include an evaluation on real data?; (C) and (G): Does the paper fulfill Condition 3?; (D) and (H): Does the paper present a new method?

https://doi.org/10.1371/journal.pone.0061562.g002

Out of these 55 articles, 43 articles presented new methods and 12 articles presented comparison studies of existing methods only, as shown in Figure 2D and Figure 2H.

Basic Comparison Study Features

Each of the articles fulfilling Condition 3 presented a comparison study of at least two methods based on real data. We recorded the number of data sets, number of included methods and number of accuracy measurements considered in each paper.

As can be seen from Figure 3A, the median number of data sets considered in the comparison studies was , while the mean was . When looking at the boxplots separately per research field, see Figure 3D, the highest mean was found in the machine learning articles with , while the medians varied from in computational statistics to in machine learning. The maximal number of data sets was found in a bioinformatics journal.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Number of data sets, methods and accuracy measurements.

The top panel depicts the results for all papers, the results in the middle panel are stratified by research field (bioinformatics: BI, computational statistics: CS and machine learning: ML), and the bottom panel differentiates between “comparison” and “new method” papers. (A), (D) and (G) show the number of data sets, (B), (E) and (H) the number of methods, (C), (F) and (I) the number of accuracy measurements.

https://doi.org/10.1371/journal.pone.0061562.g003

Figure 3B depicts the number of methods included in the comparisons. The median was , while the mean was 6.36. A separate examination (Figure 3E) shows the highest median and mean for the machine learning papers with a median of and a mean of included methods. The maximum of included methods was again found in a bioinformatics journal.

Finally, the mean number of accuracy measurements was and the median was as depicted in Figure 3C, while the maximum of accuracy measurements was found in the machine learning articles (Figure 3F).

For illustrative purposes these basic features are also separately shown for the papers presenting a new method and for papers focusing on the comparison itself. These boxplots (Figure 3G, Figure 3H, Figure 3I) show that the number of data sets, number of methods and number of accuracy measures were not substantially different for comparison studies and articles presenting new methods.

New Methods and Clear Winners

A more delicate question is whether in comparison studies which introduce a new method clear winner(s) among the considered methods are reported in the conclusion. Having the so called “no-free lunch theorem” [4] in mind, we do not expect papers claiming that one of the methods, for instance the new method, outperforms all competitors for all classification problems.

We identified three main profiles: (W1) comparison studies claiming that the new method outperforms the other methods in general with respect to the chosen accuracy measure; (W2) comparison studies arguing that the new method shows performances similar to existing methods in terms of accuracy but has either other important advantages (for instance computational efficiency) or better performance in some specific cases that makes it a serious competitor; and (W3) comparison studies just saying that the new method performs similarly to other methods without pointing to a specific advantage.

Note that the concept of winner and the classification of the papers into one of the categories W1, W2, W3 implies a high level of subjectivity, since the definitions are intentionally kept vague. In contrast to biomedical research where, say, the -value of the association between outcome and prognosis factor is either significant or non-significant, there are to date no well-established or/and natural criteria to classify the results of methodological computational papers. To make our classification more transparent, we give examples of typical statements for each of the three categories in Table 2. This definition problem underlines the importance of reporting issues in computational literature–a topic that has barely focused any attention so far but might be further investigated in future research.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Examples for the three winner categories.

https://doi.org/10.1371/journal.pone.0061562.t002

Five out of 12 comparison studies did not identify any clear winner(s) (see Figure 4A). In contrast, all papers presenting new methods belonged to one of these three categories W1, W2 or W3. Within these papers, all three categories (W1, W2, W3) had similar frequencies, with a slight advantage for W1 (see Figure 4C). The frequencies of the three categories were not substantially different in bioinformatics, computational statistics and machine learning journals (see Figure 4 G). The winner(s) was (were) often mentioned in the abstract, especially in bioinformatics (see Figure 4B and 4F). In most papers on new methods, several variants of the new method were considered (see Figure 4D and 4H).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Reporting of winners.

The top panel shows the global results for all papers relevant to the respective question (either all 55 papers satisfying Condition 3 or the 43 papers on new methods), while the bottom panel shows the results stratified by research field (bioinformatics: BI, computational statistics: CS and machine learning: ML). (A) and (E): Are there clear winner(s)?; (B) and (F): Are the winner(s) mentioned in the abstract?; (C) and (G): Which type of winner(s) (W1, W2, or W3) is identified? (for the 43 papers on new methods only); (D) and (H): Is the new method presented in several variants? (for the 43 paper on new methods only).

https://doi.org/10.1371/journal.pone.0061562.g004

As expected, no negative results were published, i.e., no paper suggested a new method that finally turned out to perform worse than existing methods.

Limitations

Our study is limited to a set of seven selected journals and to a specific data analysis topic (supervised classification). This possibly implies a selection bias, in the sense that considering other journals and/or other research topics may lead to different results.

Such a selection bias cannot be ruled out–even though the chosen journals do not seem to have an unusual publication policy regarding the investigated issues and the chosen topic is not known to follow an untypical pattern as far as the considered questions are concerned. Our selection of papers was not random since we focused on specific journals, but it was random with respect to the main outcome in the sense that we did not consider any information or “prejudice” concerning the investigated aspects when choosing the journals. The selected papers were also not random with respect to the topic since the topic was fixed from the beginning. Our results and conclusions are thus limited to this topic, although we expect that many aspects are generalizable to other topics. This could be investigated in future research.

Most importantly, we stress that a study involving all computational journals and all topics within computational science would pose numerous methodological problems as well. Beyond the fact that such a large study would hardly be manageable from a practical point of view, the definition of common criteria for the evaluation of papers on completely different research problems or the coordination of efforts among cooperation partners from different expertise areas would be arduous challenges, to cite only a few. In the field of computational sciences literature surveys are still in their infancy, with no well-established procedures and methodologies as it is the case in biomedicine, hence the need to focus on a specific manageable issue in our study. In the long term, however, more effort could be spent on the methodological aspects of surveys in computational sciences. We believe that such efforts are needed and could be highly beneficial to the computational science community.

In the rest of this section, we further reflect on these three striking results by focusing on over-optimism issues and the need for neutral comparison studies, by defining what we consider as a tidy neutral comparison study, and by discussing the issue of negative findings.

Over-optimism and the Need for Neutral Comparison Studies

Comparison studies included in original research articles presenting new methods might often be over-optimistic with respect to the superiority of the new method, as confirmed by our survey that shows that new methods are often reported to be better than existing methods in some sense, but never worse. Some reasons for over-optimism are empirically assessed in detail and discussed elsewhere [5], [6] in the context of supervised classification using high-dimensional molecular data. The first and perhaps most obvious reason for over-optimism is that researchers sometimes “randomly search” for a specific data set such that their new method works better than existing approaches, yielding a so-called data set bias already investigated in the literature [7]. A second source of over-optimism, which is related to the optimal choice of the data set mentioned above, is the optimal choice of a particular setting in which the superiority of the new algorithm is more pronounced. For example, researchers could report the results obtained after a particular feature filtering which favors the new algorithm compared to existing approaches. The third and probably most subtle problem is that researchers often tend to optimize their new algorithms to the data sets they consider during the development phase [5], [6]. This mechanism essentially affects all research fields related to data analysis such as statistics, machine learning, or bioinformatics. Indeed, the trial-and-error process constitutes an important component of data analysis research. As most inventive ideas have to be improved sequentially before reaching an acceptable maturity, the development of a new method is per se an unpredictable search process. The problem is that this search process leads to an artificial optimization of the method’s characteristics to the considered data sets, as empirically demonstrated in a study on supervised classification [5]. This optimization process over different variants of the new method probably often partly takes place before publication and thus remains confidential. It is common practice, however, to report the performance of several variants in the final paper, as clearly shown by our survey. In both cases, the superiority of the novel method over an existing method (for instance as measured through the difference between the cross-validation error rates) might be considerably overestimated through this optimization process.

Other reasons are of technical nature and related to the ability of the researchers to use the compared methods properly. For example, if an implementation problem occurs with the competing approaches and slightly worsens their results, researchers often tend to spontaneously accept these inferior results. Conversely, they would probably obstinately look for the programming error if such problems occur with their new algorithm. In the same vein, they may unintentionally set the parameters of competing methods to sub-optimal values, or choose a variant of the method that is known by experts to be sub-optimal. They may also select competing methods in a sub-optimal way, i.e. consciously or subconsciously exclude the best methods from the comparison for any reason. Beyond the problems of technical expertise and optimization bias, interpretation and representation issues might also affect the final conclusions of a comparison study. Given the same quantitative outputs, the impression of the reader can be affected by the choice of the vocabulary in the results section, by graphical representation, or by the choice of the main quantitative criterion used to compare the methods. For all these reasons, many comparison studies published in the literature as part of an original paper are substantially biased. In an ideal world, scientific practice might be improved to reduce the impact of the optimization problems discussed above. It is questionable, however, whether these problems can all be fully ruled out in the medium term–even if most researchers show good will to address these issues.

Based on these thoughts, we stress the importance of neutral comparison studies that we define as follows:

A. The main focus of the article is the comparison itself. It implies that the primary goal of the article is not to introduce a new promising method.

B. The authors should be reasonably neutral. For example, an author who has published an article on a new method six months before is likely to be less neutral than an author who has often used several of the considered methods for statistical consulting and, say, previously investigated three of them more precisely in methodological articles. Although an informal check based on the authors’ publication lists found on their homepage suggested that it was not a problematic issue for the comparison studies included in our survey, non-neutrality of the authors may induce a bias in general.

C. The evaluation criteria, methods, and data sets should be chosen in a rational way, see Section 4 for a more extensive discussion of this problem.

In this perspective, comparison studies presented as part of papers on new methods should be considered as valuable illustrations but, strictly speaking, not as comparison studies. Going one step further, one could say that the authors of a new method should present a convincing illustration that would typically suggest that a method is promising, but leave the actual neutral comparison to other teams in a collaborative context.

Note that in papers on new methods the comparison between the competing methods is essentially not affected by the bias discussed in the previous section. Hence, an idea could be to extract neutral comparisons from comparison studies included in original articles presenting new methods–by considering the competing methods only. However, one should keep in mind that these methods probably have not been given as much attention as in the case of a real neutral comparison study that does not involve any new method. This relative lack of attention possibly leads the underestimation of their performance. In our survey, the number of methods considered in the comparisons was not substantially lower for the papers focusing on new methods than for the comparison studies. This suggests that in the former papers less attention is devoted to each compared method, since the authors spend a lot of time on the development of the new method.

To come to the point, in an original article on a new method, the focus is on the new method, and that is where the authors generally spend most of their energy. Consequently, comparisons between competing methods should not be over-interpreted because they may be of sub-optimal quality. On this account we make a (passionate) plea for neutral comparison studies in computational sciences.

Tidy Neutral Comparison Studies

In the same way clinical research and clinical studies have to be well planned and executed (following strict guidelines), comparison studies should also follow a well-defined study design. They should be based on a sound theoretical framework, appropriate analysis methods, and carefully selected components. There is a variety of literature on the design and analysis of comparison studies available. In the context of supervised learning we propagate, for example, Hothorn et al [8] as a theoretical framework and Eugster et al [9] as its practical implementation. However, regardless of the concrete framework, general considerations on the individual components–evaluation criteria, methods and method parameters, and data sets–can be made.

Negative Results and Pitfalls

In our survey none of the 43 papers on new methods presented negative conclusions, suggesting that computational literature is affected by a substantial publication bias. In this context, neutral comparison studies can be a good vehicle for negative research findings. Publication biases and the necessity to “accentuate the negative” [11] are well-documented in the context of medical and pharmaceutical research. In applied statistics and data analysis research, however, this issue receives very poor attention [6], even if the publication of negative results may be extremely useful in many cases.

The systematic exclusion of negative results from publication might in some cases be misleading. For example, imagine that ten teams around the world working on the same specific research question have a similar promising idea that in fact does not work properly for any reason. Eight of the ten teams obtain disappointing results. The ninth team sees a false positive in the sense that they observe significant superiority of the new promising method over existing approaches although it is in fact not better. The tenth team optimizes the method’s characteristics [5] and thus also observes significant superiority. The two latter teams report the superiority of the promising idea in their papers, while the eight other studies with negative results remain unpublished: a typical case of publication bias. This scenario is certainly caricatural, but similar things are likely to happen in practice although in a milder form. Note that it is very difficult to give concrete examples at this stage, since such stories essentially remain unpublished.

Nevertheless, the publication of negative results might entail substantial problems. Most researchers (including ourselves!) probably have more ideas that turn out to be disappointing than ideas that work fine. Try-and-error is an essential component of research. It would thus be impossible (and uninteresting anyway) to publish all negative results. But then, what was promising and what was not promising? What is likely to interest readers and what was just a bad idea that nobody else would have thought of? Obviously this decision that would have to be taken by reviewers and editors is a subjective one. Assessing whether a new method with negative results deserves publication in a separate paper is anything but trivial. With this in mind, we believe that the publication of negative findings within large well-designed comparison studies would be a sensible compromise in order to diffuse negative findings without congesting the literature with negative papers.

Journals would not have to fear for their impact, since good comparison studies are usually highly accessed and cited. Authors would not be urged to make something out of their promising idea on which they have spent a lot of time: a large comparison study would be an alternative to publish important results and share their vast experience on the topic without fishing for significance. And “fishing for significance” would lose part of its attractiveness. Most importantly, readers would be informed about important research activities they would not have heard of otherwise.

Note that “standard practice rules” in computational sciences (for example regarding the choice of method parameters) are often implicitly the result of comparison studies. For instance, a standard parameter value becomes standard because it yields better results than another value. In other words, negative results are often hidden behind standard practice rules - most of them remaining unpublished. Our point is that this process could be made more transparent and more informative for the readers if these negative results were published within extensive comparison studies.

Drawing the comparison with clinical research from the introduction even further, we also think that it may be interesting to publish articles on pitfalls. By “pitfall” we mean the inconveniences of a data analysis method such as, e.g., a non-negligible bias, a particularly high variability, or non-convergence of an algorithm in specific cases that may lead to misleading results. In computational literature such research results are often hidden in the middle of articles that are actually devoted to something else. This is in contrast to clinical research, where pitfalls of existing methods (for example an adverse effect of a drug) may be the main object of an article, even if no alternative solution is proposed (for example in form of an alternative drug).

Conclusion

Neutral comparison studies are in our opinion crucial to make the establishment of standards more objective and to give a chance to methods that are at first view unspectacular and would otherwise be pigeonholed. They are probably not devoted enough attention in the literature, as suggested by our survey that identified only 12 comparison studies out of a total of 55 articles on supervised classification. However, comparison studies and their impact should not be over-interpreted. Firstly, one should not forget that no method is expected to work well with all data sets (the well-known “no free lunch theorem” [4]). Hence, a method that scores well in many comparison studies may do poorly in a specific data set. Comparison studies are not expected to yield an absolute truth applicable to all situations. They are solely useful to determine general trends that may be useful to the community to select a set of standard methods that often perform well.

Secondly, comparison studies are essentially limited because they rely on the specific and sometimes arbitrary choices regarding the study design: the choice of simplifying evaluation criteria that probably do not reflect the complexity of concrete data analysis situations, the choice of method parameters that may substantially impact the relative performance of the considered methods, and last but not least the choice of specific example data sets.

Thirdly, comparison studies are often underpowered in the sense that the number of included data sets is insufficient considering the high variability of performance across data sets. With a few exceptions (see the comparison of machine learning algorithms based on 65 gene expression data sets [12]), comparison studies most often include up to 10 data sets. The median number of considered data sets was only 5 in the comparison studies considered in our survey. This is probably not enough. This issue may be further investigated in terms of sample size in a statistical testing framework [13].

Fourthly, comparison studies essentially ignore the substantive context of the data sets they consider. Data sets are sometimes preprocessed without much knowledge of the signification of the variables. All methods are applied in the same standardized way to all data sets. The analysis is thus intentionally over-simplified. An important aspect of the data analysis approach is neglected, which does not reflect the complexity and subtleties of the data analyst’s work [14]. A method that does not work well if applied in a standard way without knowledge of the substantive context might perform better in concrete situations, hence reducing the relevance of comparison studies.

To conclude, neutral comparison studies are often considered as less exciting than projects on new methods by both researchers and journal editors–but not by readers. They can neither be expected to always give the best answer to the question “which method should I use to analyze my data set” nor reflect a real data analysis approach that takes the substantive context into account. However, we believe that they may play a crucial role to make the evaluation of existing methods more rational and to establish standards on a scientific basis. They certainly deserve more consideration than is currently the case in the literature.