# Computer Vision algorithm to tell if camera is moving?

I'm looking for a computer vision algorithm or method that can tell if the camera is moving in a video. Or maybe an alternate way of telling if the background is moving. I have a lot of videos and only want to keep the ones where the camera is stationary.

All videos feature a person talking directly to the camera (the person is mostly stationary throughout the video). I want to keep videos where the camera is stationary, but there are a lot of videos where the person is holding the camera and walking or the camera is moving around too much

I plan to run some algorithm one by one on a collection of videos to determine which video IDs have a moving camera in them.

An approach that is fast and seems to work well for me, and does not involve any fancy stuff, is to use the correlation of pixels over time as an indicator of motion:

1. Make your video greyscale (lets call it video $$\in \mathbb R^{T\times H\times W}$$)
2. Spatially Highpass filter video to make video_shp
3. Temporally Lowpass the result to make video_shp_tlp
4. Sample some pixels in the video (e.g. 1000). Lets call these indices $$\in \mathbb N^{2 \times N}$$. Only do this on the first frame and reuse the sample-indices in subsequent frames.
5. Compute the correlation coefficient between the sampled pixels of the spatially-highpassed video_shp and the temporally-lowpassed video_shp_tlp: $$\rho \in [0, 1] = corr($$video_shp[indices], video_shp_tlp[indices])
6. Take your motion_score $$\in [0, 1]$$ to be $$1-\rho$$.

Results look like this:

Python Code:

@dataclass
class MovementDetector:
seed: int = 1234
n_samples: int = 1000
decay_rate: float = 0.2
spatial_highpass_filter_width: int = 10

_ixs: Optional[NDArray] = None
_lowpass_img: Optional[BGRImageArray] = None

def get_moving_score(self, image: BGRImageArray) -> float:
""" Takes the latest image, and outputs a score indicating the
amount of recent motion.  Score in in [0, 1] interval,
with 1 being lots of motion and 0 being None.
"""
im_grey = image.mean(axis=2)
im_grey_highpass = im_grey - cv2.boxFilter(im_grey, ddepth=None, ksize=(self.spatial_highpass_filter_width, self.spatial_highpass_filter_width))
im_flat = im_grey_highpass.reshape(-1)
if self._ixs is None:
self._ixs = np.random.RandomState(self.seed).choice(len(im_flat), size=min(len(im_flat), self.n_samples), replace=False)
if self._lowpass_img is None:
self._lowpass_img = np.zeros_like(im_flat)
historical_values = self._lowpass_img[self._ixs]
current_values = im_flat[self._ixs]
pixel_correlation = np.corrcoef(historical_values, current_values)
self._lowpass_img = self.decay_rate * im_flat + (1 - self.decay_rate) * self._lowpass_img
return 1 - pixel_correlation[0, 1]

• Seems like a reasonable approach. Probably not perfect: If a large bird flies in front of the camera, or if a large tree branch in front of the camera is swaying in the breeze, this might wrongly conclude that the camera is moving.
– D.W.
Commented Sep 25, 2022 at 17:08
• True - there may be a more robust way to do it that looks at, e.g. only the 50% of most correlated pixels, and thus ignores significant components of the image that are moving, so long as they don't form the majority. Commented Sep 26, 2022 at 17:22
• Clever thank you!
– D.W.
Commented Sep 26, 2022 at 22:46

Use background detection algorithm, say Gaussian Mixture-based background detection (present in OpenCv) and then make some assumption - say backround takes more than 60% of image.
Then convert frames to color model which keeps luminance as separate component. Compare consecutive frames with small tolerance per color per pixel and bigger tolerance per light (to take into account shadows). If your camera is not really stationary, check by movement algorithm if frame is shifted by some distance (camera may move slightly even if it stands still), in that case compare it with shifted frame after detecting shift.

Now gather frames with some tolerance, say 95% of frames match background in at least 60% of pixels with slight shifts (it depends on distance from camera to background).
If it matches, camera was still, if not then either parameters were too big or it was moving.

• In case of sudden strong illumination change, the algorithm may result in a movement detection I think. Commented Dec 23, 2020 at 13:52

This is an experimental area, so expect that you will need to try something, see how well it works, and then adjust to address error cases.

Here is a possible approach, that's a variation on what Evil suggests:

You could use optical flow on the video, to find the direction in which the pixels are moving. Use background detection to detect which pixels are part of the background.

Then if all (or most) of the non-background pixels in the image are moving, and all in basically the same direction (though not necessarily the same amount, as some objects may be near and some far), conclude that the camera is moving. Otherwise, conclude that the camera is still.

This is something you could evaluate, to see how often it gets the right answer, and for the cases that it's wrong, if there are commonalities to the failures.

For a simple solution, take the difference between pairs of successive frames. There will be a significant difference where there is motion.

When the person is moving, motion will be detected essentially in the middle of the screen. When the camera is moving, the whole frame will be affected.

Note that if the background is uniform, it will be impossible to differentiate between a moving person and a moving camera.