0
$\begingroup$

I transformed the MNIST dataset as follows:(X (70000 x 784) is the training matrix)

np.random.seed(42)
def transform_X():
    for i in range(len(X[:,1])):
        np.random.shuffle(X[i,:])

I had thought that shuffling the pixels in an image would make the digits unrecognizable by humans,but the machine learning algorithms would still be able to learn from the images with shuffled pixels correctly.

I cannot understand the mathematical reason behind not learning. Do you think this should work and I should recheck my code for any mistakes? I trained on SVC and SGDClassifier of sklearn.Results from both are just randomness.

$\endgroup$
1
$\begingroup$

You are essentially setting each image to a (more or less) random image. And now you wonder why it is not learning? There is (almost) no data to learn from left.

Let me give you a similar example. Here is some (unshuffled) training data:

0 0 0 -> 0
0 0 1 -> 0
0 1 0 -> 0
0 1 1 -> 0
1 0 0 -> 1
1 0 1 -> 1
1 1 0 -> 1
1 1 1 -> 1

The pattern is pretty obvious: The first column is equal to the label. The next two columns are irrelevant. Now let me give you some shuffeled training data:

0 0 0 -> 0
0 1 0 -> 0
1 0 0 -> 0
0 1 1 -> 0
1 0 0 -> 1
1 1 0 -> 1
1 0 1 -> 1
1 1 1 -> 1

Good luck with finding the correct pattern there. And this is only 3 dimensional; things get much worse with more dimensions.

One thing that could work if you stored a random permutation and applied the same permutation to each image.

$\endgroup$
  • $\begingroup$ I thought that setting a seed to np.random will apply the same random permutation to each image.Am I mistaken about the use of seed? $\endgroup$ – Kartik chincholikar Jul 25 '17 at 6:31
  • 1
    $\begingroup$ Thank You.I looked it up and implemented the correct code. $\endgroup$ – Kartik chincholikar Jul 25 '17 at 8:59
  • 1
    $\begingroup$ Yes,It works. I'm getting 94.3 cross validation accuracy using RandomForestClassifier of sklearn. It does recognize digits from what looks as pure randomness to us! $\endgroup$ – Kartik chincholikar Jul 25 '17 at 13:19
  • 1
    $\begingroup$ @Kartikchincholikar Nice! I guess there is not a (significant) differnce to the non-permutated version, right? Now it would be interesting to see if CNNs perform significantly different between permutated vs non-permutated. (I expect that they do) $\endgroup$ – Martin Thoma Jul 25 '17 at 13:24
  • 1
    $\begingroup$ Yup,No significant difference $\endgroup$ – Kartik chincholikar Jul 25 '17 at 14:05
1
$\begingroup$

A good indication, although quite rough, if an algorithm is capable of solving a task (especially pattern recognition tasks) is to check if humans can. We evolved to be quite good at it.

Obviously, that might not always be true. But in your case, by shuffling you loose all the structure of the data. The only possible feature your ML algorithm can use is the number of pixels set vs unset (or the pixel values, for grayscale images), thus basically losing all the structure. The problem is that this is a bad feature since there is little correlation between the number of pixels set and the digit. There is no pattern left, and any attempt to learn a pattern is doomed to fail. Thus, your algorithm may:

  • try to learn a pattern where there is none
  • not have enough features to do the classification

or a combination of the 2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.