Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale

Cumulative receiver operator characteristic (ROC) curve analysis extends classic ROC curve analysis to discriminate three or more ordinal outcome levels on a shared continuous scale. The procedure combines cumulative logit regression with a cumulative extension to the ROC curve and performs as expected with ternary (three-level) ordinal outcomes under a variety of simulated conditions (unbalanced data, proportional and non-proportional odds, areas under the ROC curve [AUCs] from 0.70 to 0.95). Simulations also compared several criteria for selecting cutpoints to discriminate outcome levels: the Youden Index, Matthews Correlation Coefficient, Total Accuracy, and Markedness. Total Accuracy demonstrated the least absolute percent-bias. Cutpoints computed from maximum likelihood regression parameters demonstrated bias that was often negligible. The procedure was also applied to publicly available data related to computer imaging and biomarker exposure science, yielding good to excellent AUCs, as well as cutpoints with sensitivities and specificities of commensurate quality. Implementation of cumulative ROC curve analysis and extension to more than three outcome levels are straightforward. The author’s programs for ternary ordinal outcomes are publicly available.

I am pleased to submit to PLoS ONE my revised manuscript, "Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale," for re-consideration as an original Research Article in the area of statistical methods. As requested, I have uploaded both marked-up and unmarked versions of the revised manuscript. The uploaded files are PDFs compiled from a LaTeX source file based on the PLoS ONE LaTex template. The LaTeX source file complies with PLoS ONE editorial specifications and is available upon request.
Today's manuscript has been revised in response to comments received 4 July 2019 from two PLoS ONE reviewers assigned by Academic Editor, Alessandro Parolari, MD, PhD. I am grateful for the reviewers' time in considering this manuscript and their constructive comments, which have substantially improved the manuscript. As requested, I have uploaded a point-by-point response to the reviewers' comments.
In addition to the manuscript, this work includes items uploaded during the original submission: (a) two datasets as ZIP files for inclusion in the Supporting Information; (b) two grayscale figures as TIF files with 300 dpi resolution, 2250 pixel width, two-pixel white border, and labels corresponding to references in their respective captions (color versions available upon request). Forthcoming items are: (c) SAS computer programs implementing the described method, to be available in the Supporting Information and on GitHub and released under a suitable open source license.
PONE-D-19-14500 "Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale" 10 July 2019 2 Reviewer #1

Reviewer #1 Comments to the Author Author's Rebuttal
The paper is well written, even if the subject is rather complex. The methodology is sound and the conclusions are in line with the results obtained. I have a few minor comments and modifications to suggest: The author is grateful for the reviewer's constructive comments and the consequent improvements to this manuscript.
1. Line 192: it would be helpful to specify on which base (arbitrary or objective?) the three quality levels for the cork stoppers were established.
Agreed. Added: Fifty cork stoppers were quantified in each of three quality levels ≪subjectively assigned by human experts≫ 2. The discussion of the simulations is not deep enough (lines 268-274). For instance, the author should discuss the contrasting results of NPO1 and NPO2 (tables 4), where a substantial under-or over-estimation of the parameters occur.
Agreed. Added: ≪The previously noted divergence of the cumulative logit parameters in the NPO2 simulation condition also suggests that caution may be warranted in some nonproportional odds situations, particularly when AUCs of the cumulative ROC curves are widely separated, as in NPO2. If, however, the primary aim is cutpoint estimation, the NPO2 condition indicates that estimated cutpoints were robust against divergence in the logit parameters. Moreover, qualitative results from the NPO2 condition regarding cutpoint selection criteria and parametric cutpoints were consistent with those from the proportional odds and NPO1 simulations.≫ See also original lines 146 -150, 164 -167.