Last night I did some calculations to see what random detector has the lowest error cost variation. That is, given that the detector will be deployed in an uncertain environment, what random detector should you use to minimize its error cost variation. The math is simple so I will show it all in this post.

As in previous posts, I am restricting myself to the simplest case: a binary classification problem. We have two classes, call them A and B. The variability of the testing environment is codified by the single parameter $f_A$, the frequency of class A. The binary random detector is described by a single parameter also, $d_A$ — the classification rate for class A.

The cost of misclassifying class A as class B will be denoted by $C_{A \rightarrow B}$. The cost of classifying class B as class A by $C_{B \rightarrow A}$. The error cost or loss function in this simple example is then equal to
$$L = C_{A \rightarrow B} f_A (1 – d_A) + C_{B \rightarrow A} (1-f_A) d_A.$$

What random detector setting $d_A$ minimizes $\frac{\partial L}{\partial f_A}$, the loss variation under a changing environment? Calculating the derivative one quickly obtains that zero variation in the cost is attainable when
$$ d_A = \frac{C_{A \rightarrow B}}{C_{A \rightarrow B} + C_{B \rightarrow A}}.$$
Note that this is not the most accurate detector setting for a particular environment, $f_A$.