Comments on: A simple example of how to stabilize classification errors in uncertain environments by sacrificing accuracy http://www.corrada.com/blog/2007/10/03/a-simple-example-of-how-to-stabilize-classification-errors-in-uncertain-environments-by-sacrificing-accuracy/ Randomness, entropy, pattern matching, maps, geometry, knots, and scientific readings Mon, 26 Apr 2010 20:47:41 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Andres http://www.corrada.com/blog/2007/10/03/a-simple-example-of-how-to-stabilize-classification-errors-in-uncertain-environments-by-sacrificing-accuracy/comment-page-1/#comment-40 Andres Mon, 08 Oct 2007 12:42:05 +0000 http://www.corrada.com/blog/?p=13#comment-40 I agree, Will. As you say, the question really is what metric is most relevant to the task at hand. My rhetorical excesses are meant to counteract the obsession with highest accuracy that I encounter everyday at work. Since I am using random detectors, the area under the ROC curve (AUC) is fixed for all of my examples. As you say, clients want us to reduce the AUC. But not always. I have a funny story about this from when I worked at Dragon Systems. Dragon participated in a series of speaker identification competitions sponsored by NSA in the 90’s. I was lead developer for a system that won 6 out of 9 testing conditions. By won, I mean we got the lowest AUC of any of the other entries by research powerhouses like IBM and BBN. George Doddington had stated he would give 25 dollars for the ‘best’ system in each testing condition. When he got up to award his prizes during the conference with the competition participants, I started licking my chops over the 150 dollars I was going to get. George then proceeded to award all the prizes to other systems because his criterion was not lowest AUC! He chose the systems that set their operating point at their lowest cost! That is, even though we had fielded a system that could attain the lowest cost for 6 of the 9 conditions, we had not set it to the lowest cost it could get. The lesson to me: users sometimes want to minimize costs with what they have. I agree, Will. As you say, the question really is what metric is most relevant to the task at hand. My rhetorical excesses are meant to counteract the obsession with highest accuracy that I encounter everyday at work.

Since I am using random detectors, the area under the ROC curve (AUC) is fixed for all of my examples. As you say, clients want us to reduce the AUC. But not always. I have a funny story about this from when I worked at Dragon Systems.

Dragon participated in a series of speaker identification competitions sponsored by NSA in the 90’s. I was lead developer for a system that won 6 out of 9 testing conditions. By won, I mean we got the lowest AUC of any of the other entries by research powerhouses like IBM and BBN. George Doddington had stated he would give 25 dollars for the ‘best’ system in each testing condition. When he got up to award his prizes during the conference with the competition participants, I started licking my chops over the 150 dollars I was going to get. George then proceeded to award all the prizes to other systems because his criterion was not lowest AUC! He chose the systems that set their operating point at their lowest cost! That is, even though we had fielded a system that could attain the lowest cost for 6 of the 9 conditions, we had not set it to the lowest cost it could get.

The lesson to me: users sometimes want to minimize costs with what they have.

]]> By: Will Dwinnell http://www.corrada.com/blog/2007/10/03/a-simple-example-of-how-to-stabilize-classification-errors-in-uncertain-environments-by-sacrificing-accuracy/comment-page-1/#comment-39 Will Dwinnell Wed, 03 Oct 2007 19:50:51 +0000 http://www.corrada.com/blog/?p=13#comment-39 I think the question at hand is: “Which performance metric is most relevant?” Observation-level accuracy is known to suffer from exactly the peculiarity which you describe. My experience in practice, though, is that many clients want probability outputs (as opposed to class label outputs) and need to maximize class separation (measured by, for instance, AUC). I think the question at hand is: “Which performance metric is most relevant?” Observation-level accuracy is known to suffer from exactly the peculiarity which you describe. My experience in practice, though, is that many clients want probability outputs (as opposed to class label outputs) and need to maximize class separation (measured by, for instance, AUC).

]]>