We have a database with well-formated and structured data. From time to time we get Excel - files from our clients. This Excel Files have to be imported to the database. It it very important that we dont create duplicate data so we first need to check if the data record already exists in the database.

The problem is: The records in the excel files do not contain a unique identifier. Although there are certain properties that are always included, but they are not suitable for easy comparison. In addition, the excel files contain many properties that are missing in other files and that we do not have in our database.

We are talking about tens of thousands of records here. It is not an option that this happens manually by an employee. At the very least, a large part should be handled automatically and only a small part should be manually controlled.

My question is very open. I do not know which solution I'm looking for exactly. Maybe there is a general generic algorithm for merging unstructured data with structured data?

  • 1
    $\begingroup$ Do you have any criterion for when a data is a duplicate? This is a modeling issue that only you can answer, without which it is impossible to address your question. $\endgroup$ – Yuval Filmus Jul 19 '18 at 7:22
  • $\begingroup$ This seems like, with a little more explanation of the record(s) structure, it might be a better fit for StackOverflow. $\endgroup$ – Lance Pollard Jul 20 '18 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.