0
$\begingroup$

I have thousands of times series of different length and different time. I want to group them together so that I know the optimal ones to pick as input for a ML algorithm and to document how they are related. Those times series are from stations, some of them inactive and some still active.

My goal would eventually be to use data from active stations in the cluster in order to model the inactive station by assuming that the correlation of the signal will stay the same in time.

I am able to get the euclidean distance and a distance matrix ( used for DTW) for series that overlap but the issue is that most of them will not overlap together. So if I summarize the distance value and put it into a global distance matrix I will have a sparse distance matrix.

I know that hierarchical clustering can deal with data considering only their distance to each other but the distance matrix may contain only finite values (in the scipy version) and adding any sort of fake distance values will greatly affect clustering.

I am aware that DBSCAN might be my only option since I only have sparse distance measure and that any other approach with the raw data will not be possible due to their varying total length and their varying start and end period.

Is DBSCAN the only solution to these types of problems or are there other algorithms or methodology that can cluster using only sparse distance data ?

$\endgroup$
  • $\begingroup$ What do you mean by "optimal ones to pick as input for a ML algorithm"? Why do you think it is best to pick only a subset? What do you mean by "document how they are related"? I wonder if there is a better way to accomplish whatever you are trying to accomplish. $\endgroup$ – D.W. Jul 3 '18 at 6:38
  • 1
    $\begingroup$ What do you mean by "for series that overlap"? You can compute the DTW distance for any pair of time series; there's no restriction that they "overlap" (whatever that means). Are you sure you are using DTW properly? I don't understand your constraints. $\endgroup$ – D.W. Jul 3 '18 at 6:39
  • $\begingroup$ The goal would be to model a station from the observer points of another station. To do so I need to see how they behave in relation to each other for a given time or in a relatively short timeframe. If some of those stations exhibit the same response at the same time or with a handful of days of different then I know that their response signal is correlated and that can be used to estimate the state of the system at the inactive station. Using all 5000+ stations for input for a ML algorithm is too much unnecessary data since only a few correlated input should be sufficient. $\endgroup$ – Al rl Jul 3 '18 at 10:32
  • $\begingroup$ If I were to use any random series with DTW that woulnd't help since I want to do a daily prediction and matching series that range from 1950 (and ending randomly) with some starting in the 2000s that can cover a couple of years to many decade in order to compare their behaviour in time makes no sense. This is why I must join the dataset with an inner join based on the date to obtain the distance of their response for the same time period. $\endgroup$ – Al rl Jul 3 '18 at 10:38
  • $\begingroup$ I still question your premise. I question that putting 5000+ stations into the training set is "too much data"; ML algorithms are perfectly well able to handle extra data, and usually the more data, the better. It seems like you might be trying to solve a problem that you don't need to solve. $\endgroup$ – D.W. Jul 3 '18 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.