Statistics Department

Topic:

Population and Empirical PR Curves for Assessment of Ranking Algorithms

Abstract:

The ROC curve is widely used to assess the quality of prediction/classification/ranking algorithms, and its properties have been extensively studied. The precision-recall (PR) curve has become the de facto replacement for the ROC curve in the presence of imbalance, namely where one class is far more likely than the other class. While the PR and ROC curves tend to be used interchangeably, they have some very different properties. Properties of the PR curve are the focus of this paper. We consider: (1) population PR curves, where complete distributional assumptions are specified for scores from both classes; and (2) empirical estimators of the PR curve, where we observe scores and no distributional assumptions are made. The properties have direct consequence on how the PR curve should, and should not, be used. For example, the empirical PR curve is not consistent when scores in the class of primary interest come from discrete distributions. On the other hand, a normal approximation can fit quite well for points on the empirical PR curve from continuously-defined scores, but convergence can be heavily influenced by the distributional setting, the amount of imbalance, and the point of interest on the PR curve.

Biography:

Jacqueline M. Hughes-Oliver is Professor of Statistics at North Carolina State University. She earned her PhD in Statistics from NC State in 1991, following a BA in Mathematics from the University of Cincinnati in 1986. After one year at the University of Wisconsin—Madison, Dr. Hughes-Oliver returned to NC State where she transitioned through the usual academic ranks. From 2005 to 2009, Dr. Hughes-Oliver was Director of the Exploratory Center for Cheminformatics Research at NC State. Her methodological research focuses on prediction and classification, analysis of high-dimensional data, variable and model selection with dimension reduction, design and analysis of pooling or mixture experiments, optimal design, and spatial modeling. Dr. Hughes-Oliver is a Fellow of the American Statistical Association and was awarded the 2014 Blackwell-Tapia Prize for contributions to the mathematical sciences.

This week's seminar speaker is sponsored by our Korvis Professsor, Javier Rojo.