1
$\begingroup$

I want to train a Perceptron using stochastic gradient rulefrom the stream data. I have very limited amount of memory and i can store only $N$ examples.

Suppose my population consist of point as show in the following picture: enter image description here

Now suppose my first $N$ examples come in following fashion and i can classify them correctly as show in the next picture:

enter image description here

Now the problem is if the next $N$ examples come in this way:enter image description here

and i have classified them as shown. The problem is that since i can't train perceptron for previous $N$ examples(because i can store only $N$ examples and previous $N$ examples need to be thrown away) and training on next $N$ examples contradicts the hyper-plane for previous $N$ examples.

How to train a perceptron from stream data? Do i need to store all examples or there is an alternative way?

$\endgroup$
1
$\begingroup$

I think you're confusing yourself by worrying about a worst case that likely won't happen. Assuming each element of the stream comes randomly and independently from some distribution, then it is exponentially unlikely that the case you mention happens.

How do you train from a stream? Easy: each time you see an example, you update the weights of the neural network following the rules of gradient descent, then throw away that example. In this way, the weights of the neural network change over time, and you don't need to store any past examples. Over time, as you've seen enough examples, this should converge to a neural network that performs about as well as possible on the data, assuming that all of the data in the stream comes from the same distribution.

$\endgroup$
0
$\begingroup$

After the first $N$ examples, the separator you got is after its rigorous adjustments through shift and angular orientation operations from its initial position. These adjustments are done by the weights of the data of each dimension. So every time a new data arrives and fed to the perceptron, the weights changes accordingly (rate of change in error is calculated) and the separator gets a new position. Thus there is no question of storing old data and state. If your system is too low in memory to hold the neural network and the feature vector, then it will be handled by the OS through the virtual memory system.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.