I've been tasked with figuring out the best way to implement record linkage between multiple data sources. My suspicion is that supervised approaches will increase accuracy over unsupervised approaches as long as we are willing to do a clerical review to create a training data set that is large enough to be representative. Admittedly this would be very time consuming but I think it might be worthwhile in the long run. After reviewing many papers/blogs etc I can't seem to find many comparisons of current supervised vs unsupervised algorithms. Has anyone seen any work on this? Any links or guidance is appreciated. Or just general record linkage insight is appreciated also. Thanks!

  • $\begingroup$ What is "best" is subjective, and we typically discourage subjective questions. Instead, we ask for an answerable technical question, with clarity about how you will evaluate proposed answers. What are the requirements or evaluation metrics? The question sounds pretty vague to me. It's not clear what exactly you mean by "record linkage between multiple data sources"; there are many things that could potentially mean (e.g., it could mean "finding an exact match for the SSN" or it could mean "fuzzy matching" based on any number of criteria). $\endgroup$ – D.W. Jun 2 at 17:18
  • $\begingroup$ I don't think anyone is going to be able to tell you which method is more effective (and certainly not without more details) -- in machine learning, the only way to know is to try it and see. If no one else has tried it before, that means you'll have to be the one to try it. $\endgroup$ – D.W. Jun 2 at 17:19
  • $\begingroup$ Ok thanks for the insight. At this preliminary stage I actually don't have the data in hand yet so I can't specify the exact structure. I'm just doing a preliminary lit review before I get the data and I need to make some general recommendations about the best path forward. I am matching people between multiple sources. Some of the sources will have ssn, name, address, dob. Others will be missing ssn which of course makes the task much harder. $\endgroup$ – ALA Jun 2 at 17:44
  • $\begingroup$ So in other words its not exact matching. The binary classification problem will be highly imbalanced so I will most likely use F-Measure to balance precision and recall. I'll likely implement several alternatives and see which yields the best results. $\endgroup$ – ALA Jun 2 at 17:44
  • $\begingroup$ I suggest doing some more research, then. We expect you to do a significant amount of research before asking, and to share with us in the question a summary what you've found so far, to make this useful for others: meta.stackoverflow.com/q/261592/781723. See, e.g., datascience.stackexchange.com/search?q=fuzzy+matching. Also, we want that kind of context in the question itself. Please don't put clarifications in the comments: instead, edit the question so it reads well and will be useful for others who encounter it. $\endgroup$ – D.W. Jun 2 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.