2
$\begingroup$

Assume $X$ and $N$ are two sets of vectors (observations) from a normal distribution, where $X$ represents clean data and $N$ represents noise; and $A$ a projection matrix of a filter. The scenario is that our clean data was corrupted by a multiplicative noise via matrix $A$ and an additive noise of $N$:

$$Y=A X + N.$$

How can we learn the projection matrix $A$ and $N$ from the training data $X,Y$? Does the Gaussian assumption of $A$, $N$ and $X$ help to have a better estimation or guide to use a specific solution?

Here is matlab code for the training data, noise and a simple projection:

    dataVariance = .10;
    noiseVariance = .05;  
    mixtureCenters=randn(13,1);
    X=randn(13, 1000)*sqrt(dataVariance ) + repmat(mixtureCenters,1,1000);

    %N and A are unknown and we want to estimate them.
    N=randn(13, 1000)*sqrt(noiseVariance ) + repmat(mixtureCenters,1,1000);
    A=2*eye(13);

    Y=A*X+N;

    for iter=1:1000
        A_hat,N_hat = training(X_hat,X,Y);
    end


Note: if necessary, for each estimation of $A$, an error can be calculated for an estimation of $N$ using a current $A$.

For example:

for iterate=1:1000
  initiate A
  estimate N using current A (N=Y-A*X)
  calculate error of estimation (err=Y-A*X+N)
  update A

But I would prefer not to go for gradient descent approaches.

$\endgroup$
  • 1
    $\begingroup$ Look up multivariate linear regression. One way would be learning each row of $A$ separately using classical linear regression. $\endgroup$ – Yuval Filmus Jan 22 '16 at 19:39
  • $\begingroup$ @YuvalFilmus what about $N$? I need to learn both. $\endgroup$ – PickleRick Jan 22 '16 at 19:40
  • 1
    $\begingroup$ After learning $A$, you have samples of the noise, and then you can estimate its parameters statistically. $\endgroup$ – Yuval Filmus Jan 22 '16 at 19:41
  • $\begingroup$ @YuvalFilmus I see. Is this ok to forget about the additive noise $N$ while estimating via multivariate linear regression and estimate $A$ without considering $N$? $\endgroup$ – PickleRick Jan 22 '16 at 19:44
  • 1
    $\begingroup$ Multivariate linear regression takes the noise into account. If there was no noise, you could find $A$ by solving linear equations. In fact, since your noise is Gaussian, you should use a min least squares method. $\endgroup$ – Yuval Filmus Jan 22 '16 at 19:45
1
$\begingroup$

Your problem is that of multivariate linear regression. Finding each row of $A$ corresponds to a classical linear regression problem, which can be solved using the method of least squares, which is optimal here since the noise is Gaussian.

Once you have an estimate for $A$, you can estimate the parameters of $N$ by isolating $N = Y-AX$. (This is not necessarily the optimal method since you only have an estimate for $A$, but practically it might be good enough.)

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Just to summarize the answers given for this question, here I provide a visualization of different methods carried out to learn the projection matrix $A$ given clean samples $X_i$ and noisy samples $Y_i$:

learning the projection matrix using different methods: first row: original matrix for 5 different cases, second row: initialization of the matrix, third row: using normal equation, directly derive the projection matrix, fourth row: linear regression using gradient descent and finally, fifth row, using kalman filtering

In the first row we can see 5 different projection matrices ($A$) chosen to project $X$ before adding Gaussian noise $N$.

In the second row, you can find the random projections initialized to be learned via different methods.

In the third row, using the normal equation, we directly derive the projection matrix: $$ \hat A = (X^T \times X)^{-1} \times X^T \times Y$$

In the fourth row, a linear regression using gradient descent is used to learn the projections via $X$ and $Y$.

And finally, the last row, using kalman filtering method (find out more about it here).

As you can observe, the GD solution and the normal equation are very similar and since in this example, $(X^T \times X)^{-1}$ is feasible to calculate, it sounds a more reasonable solution and can be done in only one step. If the dimensionality of our features were too high, then we can use the GD since both $X$ and $Y$ are Gaussian.

The KF solution converges faster than GD, but can not achieve a good performance as GD, I believe.

After learning $A$, $N$ can be calculated via: $$\hat N=Y- \hat A \times X$$

and further $\hat {\mu_N}$ and $\hat {\Sigma_N}$ can be computed using $\hat N$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This answer was copy-pasted on on SciComp.SE too. $\endgroup$ – D.W. Jan 28 '16 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.