I'm trying to think of a way to do approximate joins between 2 rows in a data table. Like for example lets say I have a row where the person name is "John Doe" but then another data table uses a different person format like "Doe, John." Now suppose I can put them both to a standard format say "John Doe." Sweet so I have a way to reference them and let's say there's other data correlated with "John Doe" is there any algorithms out there to infer whether these 2 John Does are the same given the other data columns? You can see the other data columns as features as well, because I know FB has an algorithm to stop duplicates for the same person and StackOverflow will merge accounts if it thinks they're being used by the same person. Any suggestions?
1
-
$\begingroup$ en.wikipedia.org/wiki/Record_linkage $\endgroup$– D.W. ♦Commented Jan 29, 2020 at 1:25
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
I am not able to clearly understand your use case, but from what I infer I believe if you are trying to de-dupe data on other features, the approach would be to use LSH(Locality Sensitive Hashing) you can see the following links to get an overview. hope this helps.
https://stackoverflow.com/questions/12952729/how-to-understand-locality-sensitive-hashing
-
$\begingroup$ Sort of, this is useful, but I was thinking more like if there's 2 datasets of names, being able to infer if the same person is in both. $\endgroup$ Commented Feb 4, 2020 at 3:25