Best algorithm for correlation between time series?

Question

I have some biological data (ECG), which are quite chaotic in nature, and and some other data; that are not chaotic but related in some way, like fatigue. I want to find out how the time series, chaotic, data are related to each of these other time series.

My undergrad thesis was on string "emotion" classification and time series is a whole other animal. My goal is to understand how the "other data" trend and at what percentage in correlation the biological data, not just classes [A, B, C].

I want to be able to distinct what is happening throughout the biological's phases in retrospect to the other time series. Like in this phase of the ECG there was 20% increase in fatigue, 7% in attention etc. I have all those data I just don't know how to make the connection.

I tried to find on time series ML but all of them jump from pretty basic algorithms and vague narratives to hardcore mathematical implementations.

If you know of a site, blog, docs, anything that ease into such examples and, very important, have code (python would be nice).

Would Random Forest work for such a task or should I be looking more into KNN (K-Means Nearest Neighbour), DTW (Dynamic Time Wrapping), STMF (Shape-based Template Matching Framework)?

Thank you very match for your time. Would be happy to provide and clarification I can, if anything doesn't make sense.

Answer 1

You haven't given us much information about your problem, so let me give you some hints about how to approach this.

First, look for a model. Ask yourself: Do you have a plausible biological/physical model for how the dependent variables might depend on the independent variables? Why do you think it's plausible that the independent variables might help in predicting the dependent variables? This may help you select an appropriate machine learning tool.

If you don't have a model and don't have any physical/biological reason to expect a causal relationship, then you're doing data mining (and maybe you should reconsider why you are doing this project, or maybe you need to study the underlying application domain better to try to investigate candidate explanations why there might be a causal connection between the two).

Second, classify the data you have. The right algorithm will depend on what data is in the time series. Ask yourself: What is the value at each point in time, for the dependent variables and the independent variables? For each, decide whether you have ordinal data, categorical data, or interval data -- that will play a huge role in narrowing down which machine learning algorithm might be applicable. Also, ask yourself whether you know anything about their distributions.

Third, visualize your data. Before you start diving into complex machine learning algorithms, do some data visualization. Try to find as many ways as you can think of to look at your data. Sometimes when you stare at data in the right way, that suggests hypotheses or relationships that might be relevant.

So, do all of that. That will help you think more clearly about the problem, and may help you narrow down the scope of machine learning problems. Once you've done all of that and can answer all of those questions, you might be in a position to ask a better-crafted, more narrowly scoped question.

---

Last, don't forget to consider some variety of linear regression: for interval data where you hypothesize some kind of linear relationship, it's a natural first thing to try.