We appreciate the comment by Chicco and Jurman [1] about our recent paper [2]. The goal of our paper was to make it easier for students and researchers to get started with machine learning as applied to genomic data to predict drug resistance.

In our paper, there is a section on how to evaluate a machine learning model where we discuss the metrics most commonly encountered in the literature and in other tutorials: accuracy, precision and recall.

When teaching model evaluation, accuracy is the most common metric to start with [3] but it should be made clear (as we, in fact, did in our paper) that accuracy can be misleading if the dataset is imbalanced. In such an imbalanced dataset where the vast majority of samples belong to one label, the model could just predict that label for all samples and get it right most of the time, leading to high accuracy while never correctly predicting the minority label. Nevertheless, in our experience, accuracy is a good starting point for a learner because after making predictions for all of the cases in the test dataset, a logical next step is to tally the number of times that the model got it right. For these reasons accuracy is a great teaching tool, but usually not the right choice for reporting in a research paper. We thus advocate for the usage of other metrics as well.

In our paper, we discussed precision and recall as important evaluation metrics. Precision is the ratio of true positives to all predicted positives and recall is the ratio of true positives to all actual positives. There are many benefits of using precision and recall, including their interpretability. Other options include the F1 score, F1-macro and the Matthews Correlation Coefficient (MCC).

However, instead of prescribing a “single best metric”, our preference is to have the students think about what they want a model to achieve. “Imagine you are a doctor” is what we often say, or “Imagine you run a ward in a hospital”. When resistant strains are rare, but failure to detect such a strain can be deadly, the most important task of a model is to identify the resistant samples correctly, whereas identifying the susceptible samples is much less crucial. In this scenario, recall for resistance is the most important metric.

Here we will give a brief overview of the pros and cons of the following metrics: Accuracy, Recall, Precision, F1 scores, the F1-macro score and MCC, which were touched on in the original article and the subsequent comment.

We use accuracy for a dataset with balanced target labels, for example, a similar number of resistant and susceptible isolates to any specific antibiotic drug. As mentioned before, leaning on accuracy only can be deceiving when the dataset is not balanced. Accuracy also gives the same weight to different kinds of errors, false positives and false negatives, whereas the person using the model may weight false positives and false negatives differently.

When a model user is particularly interested in preventing false positives or false negatives, they should use precision or recall respectively. For example, if our model is used to classify emails into spam (positive class) and legitimate emails (negative class), then we want to prioritize precision because it minimizes false positives (a false positive here means that a legitimate email ends up in the spam folder). On the other hand if we use a machine learning model to detect a serious disease, we want to make sure we err on the safe side and not miss any potential cases (those missed cases are false negatives) so we would prioritize recall for disease.

If a user is interested in balancing the performance of both metrics, precision and recall, F1 is the obvious choice. The F1 score is the harmonic mean between precision and recall [4,5]. We use it in cases where we want to avoid false positives and false negatives equally. Unfortunately, F1, just like precision and recall, is sensitive to a problem called “class swapping”, which means that the metric varies depending on which class is considered the positive class in a binary classification. To avoid the problem of class-swapping, one can calculate the F1 for the resistant class and the F1 for the susceptible class and average the two values to get the F1-macro score. Because F1-macro is a useful single metric that tells us how well a model performs, we used it for cross-validation in our paper [2].

Finally, the MCC has many characteristics in common with the F1-macro score. It considers all 4 values in the confusion matrix and class swapping does not affect its value. However, it may not be easy for non-technical users to interpret a metric that ranges from -1 to 1. Neither F1-macro or MCC scores are perfect for capturing model performances and like all metrics they have their particular drawbacks, such as being sensitive to prevalence changes, extreme imbalance, etc [3]. In Table 1, we summarize pros and cons of the metrics we discussed.

The authors of the comment argue that the F1 score can be overly optimistic about model performance [1]. This can be true in some cases when only the F1 for one class is used. In most cases using F1-macro leads to more accurate model evaluation. The confusion matrix in Fig 1 of the comment (reproduced here as Fig 1) is a great way to illustrate how F1-macro can be used to evaluate a model when the data set is unbalanced.

The example from the comment is a case of an unbalanced dataset with only one negative sample. The confusion matrix shows 3 true positives, 1 false negative, 1 false positive, and 0 true negatives. The confusion matrix shows that the underlying model doesn’t perform well because it labels the only negative sample as a positive. Despite this, accuracy and F1 for the resistant class are fairly high, which the commenters pointed out is misleading. However when we compute the F1 score for the susceptible class, we see that it is low, as it should. When there is a class imbalance the minority class usually gets a low F1 score [6] which offsets the high score from the majority class. MCC scores should be compared to F1-macro rather than individual F1 scores. As we can see, both the MCC score of -0.25 and the F1 macro score of 0.375 shows the poor performance of the model. In addition, the negative MCC score shows that the model performs worse than random guessing.