# Matching/finding mathematical plot images

I'm trying to come up with a method to match a given mathematical plot against a database of other plots. To make it more specific: plots are generated in R as PNG and might have different dimensions. The latter means that simple pixel-by-pixel matching won't work, even when resized, because text, plotting symbol, margins etc. between two versions of the same old are non-linear transformations of each other.

There's a number of good image matching algorithms available online but those assume any area in the picture is relevant, whereas with mathematical plots it's predominantly about the artifacts representing data that's being plotted.

I'm looking for pointers: is there an algorithm, paper etc. related to this task? Or maybe someone heard of a specialized DNN (which could help in identifying those most important artifacts)?

Appreciate any suggestions.

• I think we need to know more precisely what you mean by "matching". What do you consider to be a match? What kind of match are you looking for? There are many possible ways one might measure that. – D.W. Oct 4 '18 at 19:24
• That's a fair ask. Let me answer by presenting the images I have been trying to match: (1) image to search match against database (user input), (2) first image in the database, (2) second image in the database As you will notice, these images are very similar even though the data presented in the plots are different. – Lukasz Oct 6 '18 at 4:43
• Sorry; presenting a few example images doesn't answer my questions. – D.W. Oct 6 '18 at 5:25
• Yeah, it didn't, sorry I didn't sleep enough that night. What I meant was that the first image (1) should give higher score for p1.png than p2.png. It's not a general metric, I understand, but I'm not sure what is the right metric here. All I want is to: (1) match plots which originated with the same data, (2) match plots that use same/similar aesthetics. So it's about the intent behind the plot, more of a semantic match than pure graphical match. – Lukasz Oct 9 '18 at 7:15

You'll have to choose the distance measure yourself. You haven't answered questions about what notion of similarity you want to use, and we can't tell you what to use. You could, for instance, rescale the data so that the x-axis and y-axis both vary across the range [0,1] (so all plots share the same scale), then use the $$L_2$$ distance (Euclidean norm) between the two time series represented on the plots. That might or might not suit your needs.