$X_m$ and $Y_m$ are not shaded, because each of them is potentially the median of $X \cup Y$.
Consider two examples (Below I take the $\lfloor (n+1)/2 \rfloor$-th element as the median of an array of size $n$):
(1) $X = \{ 2, 4, 6 \}$ and $Y = \{ 3, 5, 7 \}$.
(2) $X = \{ 2, 4, 6 , 8 \}$ and $Y = \{ 3, 5, 7, 9 \}$.