Statistical Analysis of Receiver Operating Characteristic (ROC) Curves for the Ratings of the A-Not A and the Same-Different Methods

Ratings are widely and frequently used in sensory analysis. However, ratings from different methods cannot be treated equivalently because they represent different types of data generated from different cognitive mechanisms. Ignoring this fact may lead to misleading conclusions about true sensory difference or treatment effects obtained from the ratings data analysis. The receiver operating characteristic (ROC) curve in signal detection theory is used in this paper to model the ratings of the A-Not A and the same-different methods. This paper comprehensively discusses statistical analysis of the ROC curves for the methods, including presentations of the ROC curve functions, estimations of d′s and their variances, goodness-of-fit tests, difference and similarity tests, testing powers and sample sizes, etc. The paper also discusses alternative indices of sensory difference derived from the ROC curve, including the R-index and Gini index. It is appealing to rethink the significance of the R-index, not only as a powerful test statistic but also as a useful sensory measure. R functions are provided for implementing all the analyses discussed in the paper.