We have a database with well-formated and structured data. From time to time we get Excel - files from our clients. This Excel Files have to be imported to the database. It it very important that we dont create duplicate data so we first need to check if the data record already exists in the database.
The problem is: The records in the excel files do not contain a unique identifier. Although there are certain properties that are always included, but they are not suitable for easy comparison. In addition, the excel files contain many properties that are missing in other files and that we do not have in our database.
We are talking about tens of thousands of records here. It is not an option that this happens manually by an employee. At the very least, a large part should be handled automatically and only a small part should be manually controlled.
My question is very open. I do not know which solution I'm looking for exactly. Maybe there is a general generic algorithm for merging unstructured data with structured data?