I'm trying to come up with a method to match a given mathematical plot against a database of other plots. To make it more specific: plots are generated in R as PNG and might have different dimensions. The latter means that simple pixel-by-pixel matching won't work, even when resized, because text, plotting symbol, margins etc. between two versions of the same old are non-linear transformations of each other.

There's a number of good image matching algorithms available online but those assume any area in the picture is relevant, whereas with mathematical plots it's predominantly about the artifacts representing data that's being plotted.

I'm looking for pointers: is there an algorithm, paper etc. related to this task? Or maybe someone heard of a specialized DNN (which could help in identifying those most important artifacts)?

Appreciate any suggestions.

  • 1
    $\begingroup$ I think we need to know more precisely what you mean by "matching". What do you consider to be a match? What kind of match are you looking for? There are many possible ways one might measure that. $\endgroup$ – D.W. Oct 4 '18 at 19:24
  • $\begingroup$ That's a fair ask. Let me answer by presenting the images I have been trying to match: (1) image to search match against database (user input), (2) first image in the database, (2) second image in the database As you will notice, these images are very similar even though the data presented in the plots are different. $\endgroup$ – Lukasz Oct 6 '18 at 4:43
  • $\begingroup$ Sorry; presenting a few example images doesn't answer my questions. $\endgroup$ – D.W. Oct 6 '18 at 5:25
  • $\begingroup$ Yeah, it didn't, sorry I didn't sleep enough that night. What I meant was that the first image (1) should give higher score for p1.png than p2.png. It's not a general metric, I understand, but I'm not sure what is the right metric here. All I want is to: (1) match plots which originated with the same data, (2) match plots that use same/similar aesthetics. So it's about the intent behind the plot, more of a semantic match than pure graphical match. $\endgroup$ – Lukasz Oct 9 '18 at 7:15

A natural approach is to first scrape the data from each plot; then come up with a distance measure to compare the similarity of the data from two plots; then given a template plot, loop over all plots in the database, compute the distance, and find the most similar one.

There are many tools for scraping plots (i.e., given an image that contains a plot, extract the underlying data). See, e.g., https://academia.stackexchange.com/q/7671/705, https://stats.stackexchange.com/q/14437/2921, etc.

You'll have to choose the distance measure yourself. You haven't answered questions about what notion of similarity you want to use, and we can't tell you what to use. You could, for instance, rescale the data so that the x-axis and y-axis both vary across the range [0,1] (so all plots share the same scale), then use the $L_2$ distance (Euclidean norm) between the two time series represented on the plots. That might or might not suit your needs.

I don't expect deep learning to be particularly useful here. I would try more direct methods first.

  • $\begingroup$ This is pretty great, thanks! I'm going to run some tests and get back to you once I know how these methods perform in my use cases. $\endgroup$ – Lukasz Oct 9 '18 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.