Computer Vision algorithm to tell if camera is moving?

Question

I'm looking for a computer vision algorithm or method that can tell if the camera is moving in a video. Or maybe an alternate way of telling if the background is moving. I have a lot of videos and only want to keep the ones where the camera is stationary.

All videos feature a person talking directly to the camera (the person is mostly stationary throughout the video). I want to keep videos where the camera is stationary, but there are a lot of videos where the person is holding the camera and walking or the camera is moving around too much

I plan to run some algorithm one by one on a collection of videos to determine which video IDs have a moving camera in them.

Answer 1

An approach that is fast and seems to work well for me, and does not involve any fancy stuff, is to use the correlation of pixels over time as an indicator of motion:

0. Make your video greyscale (lets call it video $\in \mathbb R^{T\times H\times W}$)
1. Spatially Highpass filter video to make video_shp
2. Temporally Lowpass the result to make video_shp_tlp
3. Sample some pixels in the video (e.g. 1000). Lets call these indices $\in \mathbb N^{2 \times N}$. Only do this on the first frame and reuse the sample-indices in subsequent frames.
4. Compute the correlation coefficient between the sampled pixels of the spatially-highpassed video_shp and the temporally-lowpassed video_shp_tlp: $\rho \in [0, 1] = corr($video_shp[indices], video_shp_tlp[indices])
5. Take your motion_score $\in [0, 1]$ to be $1-\rho$.

Results look like this:

Python Code:

@dataclass
class MovementDetector:
    seed: int = 1234
    n_samples: int = 1000
    decay_rate: float = 0.2
    spatial_highpass_filter_width: int = 10

    _ixs: Optional[NDArray] = None
    _lowpass_img: Optional[BGRImageArray] = None

    def get_moving_score(self, image: BGRImageArray) -> float:
        """ Takes the latest image, and outputs a score indicating the 
        amount of recent motion.  Score in in [0, 1] interval, 
        with 1 being lots of motion and 0 being None.
        """
        im_grey = image.mean(axis=2)
        im_grey_highpass = im_grey - cv2.boxFilter(im_grey, ddepth=None, ksize=(self.spatial_highpass_filter_width, self.spatial_highpass_filter_width))
        im_flat = im_grey_highpass.reshape(-1)
        if self._ixs is None:
            self._ixs = np.random.RandomState(self.seed).choice(len(im_flat), size=min(len(im_flat), self.n_samples), replace=False)
            if self._lowpass_img is None:
                self._lowpass_img = np.zeros_like(im_flat)
        historical_values = self._lowpass_img[self._ixs]
        current_values = im_flat[self._ixs]
        pixel_correlation = np.corrcoef(historical_values, current_values)
        self._lowpass_img = self.decay_rate * im_flat + (1 - self.decay_rate) * self._lowpass_img
        return 1 - pixel_correlation[0, 1]

Answer 2

Use background detection algorithm, say Gaussian Mixture-based background detection (present in OpenCv) and then make some assumption - say backround takes more than 60% of image.
Then convert frames to color model which keeps luminance as separate component. Compare consecutive frames with small tolerance per color per pixel and bigger tolerance per light (to take into account shadows). If your camera is not really stationary, check by movement algorithm if frame is shifted by some distance (camera may move slightly even if it stands still), in that case compare it with shifted frame after detecting shift.

Now gather frames with some tolerance, say 95% of frames match background in at least 60% of pixels with slight shifts (it depends on distance from camera to background).
If it matches, camera was still, if not then either parameters were too big or it was moving.
Adjust parameters, done.

Answer 3

This is an experimental area, so expect that you will need to try something, see how well it works, and then adjust to address error cases.

Here is a possible approach, that's a variation on what Evil suggests:

You could use optical flow on the video, to find the direction in which the pixels are moving. Use background detection to detect which pixels are part of the background.

Then if all (or most) of the non-background pixels in the image are moving, and all in basically the same direction (though not necessarily the same amount, as some objects may be near and some far), conclude that the camera is moving. Otherwise, conclude that the camera is still.

This is something you could evaluate, to see how often it gets the right answer, and for the cases that it's wrong, if there are commonalities to the failures.

Answer 4

For a simple solution, take the difference between pairs of successive frames. There will be a significant difference where there is motion.

When the person is moving, motion will be detected essentially in the middle of the screen. When the camera is moving, the whole frame will be affected.

Note that if the background is uniform, it will be impossible to differentiate between a moving person and a moving camera.