I have two datasets that I am trying to match to one another. One dataset's contents is a subset of the other, but contains typographical errors of varying types and magnitude.

I am applying the following logic to the data linkage process:

Apply regular expressions in an iterative manner with increasing flexibility until a match is found (for example, in one iteration, leave vowels as optional). If two matches are found for one record within one iteration, categorize match as tie and leave unmatched. Apply Python's fuzzy regex to handle scenarios which a rule-based regex can't handle, namely character insertions, deletions, and substitutions within an edit distance of one.

I'm having to discuss this process in detail for specs. However, I'm having trouble deciding how to categorize the process. Would this be considered a deterministic process?

Your help would be appreciated. I do not have a CS background, so I apologize if my question is fairly rudimentary.

  • $\begingroup$ There are plenty of metrics (careful, not all of them are metrics in the mathematical sense!); without context, it is impossible to say which suits your purpose best. See also here and here. $\endgroup$
    – Raphael
    Mar 6 '13 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.