I have an image-based processing module that takes photos for some computer vision processing. I have many videos, but I need to sample representative frames as its inputs, preferably those frames with higher attention.

What are some good and effective algorithms or approaches to take K sample frames in a video? Interesting feature is that usually camera doesn't move much in successive frames with higher attention. So basically we have more stable scenes when a person is focusing on something important. Is there any ways to take this into account for our sampling?

  • $\begingroup$ I can't understand your requirements. What makes an algorithm "good and effective"? How do you define "good", exactly? How will you evaluate answers? What makes a frame representative? What's wrong with sampling uniformly at random? That seems like the most representative way to get a sample that reflects the distribution of data in the videos. What do you mean by "higher attention"? $\endgroup$ – D.W. Jul 3 '18 at 16:40

The question misses a lot of detail, so I will try to make an educated guess.

I don't know of any specific algorithm for the task you are trying to achieve, but the first step towards your solution would be perhaps the computation of the average of the magnitude of the optical flow throughout the whole duration of a video. Why: you said that an interesting feature in your videos is the fact that the camera moves less while filming something interesting. Then, the magnitude of optical flow vectors will be expectedly smaller. If you plot the average magnitude of optical flow throughout time of the video, you could start to see parts where there is potentially something interesting here and there.

Edit: look-up at the algorithm used by Youtube for "smart" thumbnail selection. E.g. https://stackoverflow.com/questions/28263511/smart-video-thumbnail-generator-algorithm

  • $\begingroup$ Welcome to the site! This looks like a good attempt to answer a rather under-specified question -- thanks! $\endgroup$ – David Richerby Aug 27 '18 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.