Trying to figure out the best way to approach this problem with Computer Vision

Question

Newbie to CV here so sorry of this is basic. Here's the deal, I have a program that I run many times. and each run I produce a screenshot. I need to compare screenshots from N-1 and N runs and make sure they aren't different in any dramatic way. Of course there are some minor changes like logos and pictures getting updated, etc.

SO far I've used something as simple as absdiff from opencv to highlight the difference regions and then use some sort of threshold to determine whether something passes or not. But I want to make it slightly intelligent but I'm not 100% sure how to proceed. Google hasn't yielded ghe best answers.

Essentially, I want to train the model on many different pairs of images and have the output be binary, yes or no depending on whether it should pass or not. In theory, I should be able to plug in 2 images and based on previous training, it should be able to tell me whether there is significant difference or not. What are some ways I might approach this, particularly with regards to what kinds of models to use. Thank you!

Answer 1

I am skeptical whether computer vision is going to be the most effective approach, but you can certainly do it. There is lots of work on image similarity metrics using neural networks (Siamese networks, triplet loss, etc.) and you could certainly implement any of those architectures, acquire a few thousand labels, and train on it. I would suggest pre-training on some standard similarity dataset (e.g., start with LPIPS) and then fine-tune on your data. But be prepared that this might or might not be effective.

It might be more effective to extract features and then build a ML model that works with those features rather than with the raw images. For instance, OCR both images and compare them; compute L1 and L2 distance; etc.