I've more than 10 million strings of length 1-100 characters. This number will be even bigger in the future. I'm interested in clustering this data, but I'm not quite sure what would be effective at this scale.
These are the clustering algorithms I've been looking into:
- Affinity propagation:: seems like a good solution, but the memory usage seems way too high, since the data is dense.
- DBSCAN: could also be an option, but I want all nodes/strings to belong to a cluster and not be considered "noise.
- K-medoids: seems like a good option in terms of memory usage, but the computation time seems worrying. It would also be highly preferred if the number of clusters was not determined before running the algorithm as in affinity propagation.
Do you have any ideas on how this problem can be solved? The computation time is not extremely important, as long as the result is satisfiable and it can be done within a couple of days.
Thank you in advance!