I have been coding and testing Neural Networks for a while but as of now I have only used IMAGE datasets. (i.e. I have M training images and N testing images).

Some datasets are video datasets. The UCSD Peds dataset for example has the following variants:

Peds 1: 34 training videos (of 200 frames each) and 36 testing videos (of 200 frames each) Peds 2: 16 training videos (of varying number of frames) and 12 testing videos (of varying number of frames)

So basically, in this case, I have M SETS of training images and N SETS of testing images.

I know the inputs to a neural network is arbitrary and anything can be fed, but I do not really understand how to feed a set of videos.

Will we merge all the 34 x 200 = 6800 training frames together and 36 x 200 = 7200 testing frames together and use the resulting set just like we use MNIST, etc?

How do we feed a set of training videos to a neural network?

What if I want to detect anomalies in the test videos? For example a neural network is trained using videos from UCSD dataset involving only pedestrians. And tested on videos which also have cars and bikes. So these cars and bikes are anomalies.

I'd want to classify an entire video on whether it contains just normal elements or some anomalous elements as well.

  • 1
    $\begingroup$ It's not clear what problem you are trying to solve, or what you are trying to train your neural network to do. $\endgroup$
    – D.W.
    Commented Feb 14, 2018 at 0:30
  • $\begingroup$ I don't think this is the best site to ask this on, since it seems to concern rather practical matter of how to implement NNs. Migrate to Computational Science or Artificial Intelligence? (cc @D.W.) $\endgroup$
    – Raphael
    Commented Feb 14, 2018 at 11:50
  • $\begingroup$ @Raphael, probably Data Science.SE or Stats.SE are the two I'd guess as most relevant. I wouldn't recommend AI as they seem to focus more on conceptual/philosophical rather than technical implementation/engineering. Not sure Computational Science sees much of this sort of question. In any case, I'm somewhat reluctant to migrate until I understand better what the question is. $\endgroup$
    – D.W.
    Commented Feb 14, 2018 at 16:30
  • $\begingroup$ Just made it more clear. I hope you understand the question now? $\endgroup$
    – user83728
    Commented Feb 14, 2018 at 17:11
  • $\begingroup$ Thanks for making an edit to elaborate! I guess my confusion is about something different. My confusion is what you're trying to accomplish with your classifier. Suppose you figure out how to train a classifier, and it is great. How will you use it? Are you going to use it to classify something about a single frame of video? Or are you going to use it to classify something about an entire video? You'd train it one way for the former, and a different way for the latter. The method to use will depend on what you are trying to achieve. $\endgroup$
    – D.W.
    Commented Feb 14, 2018 at 17:25

1 Answer 1


If you want to detect anomalies, my suggestion would be to build a classifier to classify individual frames of the video to determine whether that frame contains an anomaly. You'll need to categorize what you count as an "anomaly" and arrange for labelled data in your training set. But then you can extract all 6800 frames from the training videos, label each frame, and train a classifier on that training set of 6800 images. At this point the problem is the same as building a classifier on individual images. Then, given a test video, you apply the classifier to each frame to determine whether any of its frames contain an anomaly, and aggregate those results somehow (e.g., report the video as anomalous if the number of anomalous frames exceeds some threshold).

  • $\begingroup$ +1 ed. So basically feeding a video dataset is the same as merging all the frames of all videos together and feeding them image by image. There seems to be no other sensible technique right? $\endgroup$
    – user83728
    Commented Feb 15, 2018 at 1:49
  • $\begingroup$ @Cosmonavt, there are other techniques but I am suggesting one that is probably most appropriate for the particular situation you asked about. $\endgroup$
    – D.W.
    Commented Feb 15, 2018 at 6:36
  • $\begingroup$ Yes what are the other techniques? Can you elaborate a bit? $\endgroup$
    – user83728
    Commented Feb 15, 2018 at 12:21
  • 1
    $\begingroup$ You could alternatively use Conv3D or ConvLSTM2D. $\endgroup$
    – Lugi
    Commented Feb 15, 2018 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.