Impact of label flipping on ROC-AUC score

Question

I was exploring a problem of label flipping in the target column:

Problem: I have a data frame with just two columns: Predicted probability | Target. Target column is binary (contains only 0 and 1) --- the actual target values. Predicted probability column contains outputs from my model. I calculate the ROC-AUC metric based on these two columns and, let's say, I have ROC-AUC = $a_1$.

Next, I flip some zeros in the Target column to ones, and I measure the ROC-AUC again, and I get value $a_2$ now. And $a_2 < a_1$.

---

Question: What should be the distribution of positions of flipped labels, with which the second ROC-AUC (after flipping 0 to 1) is lower than the initial ROC-AUC?

I know that it is just a particular case, but in my problem, it would be sufficient to understand only the case when the ROC curve after flips lies completely under the ROC curve before flips. So, make assumptions, under which the ROC curve after flipping lies below the initial ROC curve.

For example, I am looking for something like (dummy statement):

• For each threshold (which corresponds to a point on the ROC curve) I want to have the number of flips to the right (here and further I assume that data frame is sorted by predicted probabilities and leftmost is the lowest one) of this threshold to be greater than the number of initial ones to the right of the threshold

---

Now, what I tried to do is:

1. Let's say I have a point on the initial curve $(p_0, p_1) = (\frac{x}{X}, \frac{y}{Y})$. This point corresponds to the threshold $t$ and $x$ is the number of zeros to the right of $t$, $X$ -- the total number of zeros in Target before flipping ($y, Y$ are the same but for ones). Then after all the flips this point, obviously, changes its coordinates. New coordinates will be $(\frac{x-c^r}{X - C}, \frac{y+c^r}{Y+c^r+c^l})$, where $C$ -- is the total number of flips, $c^r$ -- number of flips to the right of threshold $t$, $c^l$ -- to the left.

2. I want this new point to be under the initial ROC curve. So for the new first coordinate, second coordinate should be less than coordinate of the point on a blue curve. Namely, $\frac{y+c^r}{Y+c^r+c^l} < \frac{?}{Y}$. Where $\frac{?}{Y}$ is the point on a blue curve with first coordinate equal to $\frac{x-c^r}{X - C}$.

3. I have no idea how to find $?$ in the numerator.

Any help will be really appreciated.

Answer 1

I don't think this should be a surprise.

You fit your model on the original labels, and while you (probably) did not optimize the ROCAUC in the model training, many measures of model performance are at least roughly associated with ROCAUC.

This is to say that the predictions are as good as the model can get for the true labels.

Then you go and change the rules by switching some of the categories. "Well if I knew that was a $1$ then I would have made a different regression equation with different predictions," the model must be thinking. When you change the rules, I find it predictable that the model does not do as well.