Proof of NP-completeness via extra information

Question

I have a set of multisets $S = \{ X_1, \dots, X_K\}$ where $X_i \subset \mathbb{R}$. I need to find an optimal partition $L^*, R^*$ such that this $E(L) + E(R)$ is minimized. Denote $K(X) = \cup_{I \in X} I$, then $E(X) := \sum_{i \in K(X)} |i - \text{median}(K(X))|$, where $|.|$ is the absolute value. $X_i$ might contains duplicated elements and all operations are on multisets; in $K(X)$, $\cup$ is a union of multisets, and in $E(X)$, $\sum$ adds "with repetition" (repeated elements are summed multiple times).

I want to prove this problem is hard, but I don't have a very straightforward way to prove it's NP-complete. What I did instead was assuming given extra information, suppose I know the median of both $K(L^*)$ and $K(R^*)$, and then I can show find the optimal partition is an integer linear programming problem, which is NP-complete. Can I conclude the original problem is at least NP-complete?

---

For the sake of sharing, a wrong attempt I did on proving NP-completeness of with extra information problem.

I was converting this "given $m_L = \textbf{median}(L), m_R = \textbf{median}(R)$ and $S$, find $L$ and $R$" problem into a integer programming feasibility problem. Thanks to D.W helped me, I should do the opposite. Maybe doing the opposite is also possible. So sharing my effort in case it helps.

We can collect two vectors $A, B$ with length $K$, for every $X_i \in S$, get the imbalance with respect to $m_L$ and $m_R$. Namely $A[i] = \sum_{j \in X_i} sign(m_L - j) $, $B[i] = \sum_{j \in X_i} sign(m_R - j) $, where $$\text{sign}(x) = \begin{cases} 1 &\text{if }x>0\\ 0 &\text{if }x=0\\ -1 &\text{if }x<0 \end{cases}$$

Now we can introduce another boolean indicator vector $I = \{0, 1\}^K$. When $I[i] = 0$, we assign $X_i$ to $L$, if $I[i] = 1$, we assigns $X_i$ to $R$.

$$ \langle I, A \rangle = 0 \\ \langle 1-I, B \rangle = 0 $$

Since $m_l$ is the median of $L$, we would definitely have the sum of imbalance equals to 0. The same applies to $m_R$. And here $A, B$ are coefficients and $I$ is the variable. If such a problem has any solutions, then we found $L$, $R$.

And clearly, the newly formed problem is an integer programming problem. Which is well-known NP-complete. Whether if it is possible to convert any such type of problem into our set partition problem in polynomial time, I still need some help

Answer 1

No. Your argument is not valid; it doesn't prove NP-completeness. You have shown that an ILP solver can be used to solve your problem. But an ILP solver can be used to solve easy problems, too, so it doesn't rule out the possibility that your problem might be easy (solvable in polynomial time). The reduction needs to go the other way. I suggest studying standard material on NP-completeness and reductions, e.g., What is the definition of P, NP, NP-complete and NP-hard? and What are common techniques for reducing problems to each other? and How do I construct reductions between problems to prove a problem is NP-complete?.

As it happens, I believe your problem can be solved in polynomial time; in other words, I believe it is not hard. I believe the following algorithm solves your problem:

• Enumerate over all possibilities $m_L,m_R$ for $\text{median}(K(L))$ and $\text{median}(K(R))$. For each such possibility:

  • Find the partition $L,R$ that minimizes $E(L)+E(R)$, subject to the requirement that $\text{median}(L)=\mu_L$ and $\text{median}(R)=\mu_R$.

• Output the best solution found at any of these stages.

I show below how to do the innermost step in polynomial time. This gives a polynomial-time algorithm for your problem, as there are only $O(n^2)$ possibilities for $m_L,m_R$ (i.e., each $m_L,m_R$ must be an element of $\cup_i X_i$).

We'll use dynamic programming for this. Let $\text{imb}(T,m)$ be defined by

$$\text{imb}(T,m) = (\sum_{t \in T} \text{sgn}(t-m), \sum_{t \in T} 𝟙(t=m))$$

where the sum is "with repetition", and where

$$\text{sgn}(x) = \begin{cases} 1 &\text{if }x>0\\ 0 &\text{if }x=0\\ -1 &\text{if }x<0 \end{cases}$$

$$𝟙(x=y) = \begin{cases} 1 &\text{if }x=y\\ 0 &\text{otherwise} \end{cases}$$

Define the imbalance of a partition $L,R$ with respect to $m_L,m_R$ to be the pair $(\vec{b_L},\vec{b_R})$ where $\vec{b_L}=\text{imb}(L,m_L)$ and $\vec{b_R}=\text{imb}(R,m_R)$. Then we can reformulate the problem as follows:

Given $m_L,m_R$, find $L,R$ that minimizes $E(L)+E(R)$ subject to the requirement that the imbalance of $L,R$ with respect to $m_L,m_R$ is $(0,0)$ (or $+\infty$ if no such partition exists).

Now we can solve that using dynamic programming. Define $A[k,\vec{b_L},\vec{b_R}]$ to be the smallest possible value of $E(L)+E(R)$ subject to the requirements that $L,R$ are a partition of $X_1,\dots,X_k$ with imbalance $(\vec{b_L},\vec{b_R})$ with respect to $m_L,m_R$ (or $+\infty$ if no such partition exists). Then $A$ satisfies the recurrence relation

$$A[k,\vec{b_L},\vec{b_R}] = \min(A[k-1,\vec{b_L}-\text{imb}(X_k,m_L),\vec{b_R}] + \sum_{i \in X_k} |i - m_L|,A[k-1,\vec{b_L},\vec{b_R}-\text{imb}(X_k,m_R)] + \sum_{i \in X_k} |i - m_R|).$$

Using this recurrence, you can fill in the array $A$ in $O(Kn^4)$ time, where $n=|\cup_i X_i|$ is the number of elements (counting repetitions): you just initialize every entry to $+\infty$ except that $A[0,0,0]=0$, then fill it in, in order of increasing $k$. Then, scan all $A[K,(a,b),(c,d)]$ such that (i) $|a|<b$ and $b>0$, or $a=0$, and (ii) $|c|<d$ and $d>0$, or $c=0$; the smallest such value is the answer to the original problem.