Let's say that I have $N$ data sets where I have data points at some fixed frequency, such as "daily".

What would be a good method for finding correlation between any of the data sets, or choosing a single data set, and asking what other data sets are most correlated?

To make it less abstract, imagine that I'm logging how much water I drank a day in ounces, how much exersize i did a day in minutes, my mood level, and how bad of a headache I had that day from 1 (no headache) to 10 (bad headache). I'd want to be able to see if my headaches were correlated (keeping in mind, it might not be causation) to me drinking less water perhaps, or doing less exercise. However, it's possible that there may be a lag between cause and effect (or correlation), etc.

That's my basic question, but in case it's relevant to the answer, I'm also wondering how I might be able to handle missing data (maybe some days didn't get data?).

Also, I was wondering what if the data sets AREN'T at the same frequency? Like one data set may be weekly while the others are daily.

My best guesses at how to approach this could be to use a discrete Fourier transform of each data set, and then do a dot product between the frequency histograms to find ones that had similar frequencies (even if different phases), and then I could use the phase information to figure out the kind of lag time there is.

Since I'm also interested in negative correlation, I'm thinking that when i do the dot product between the histograms and get my 0 to 1 answer, that i would report "correlation" as $abs(dotProduct-0.5)$ so that things close to either 0 or 1 were treated as important.

Is there a better way? Is my best guess a reasonable approach?



Let's start with comparing two time series. To compare them, you probably want to use cross-correlation. The cross-correlation computes the correlation between signal $f$ and a delayed version of signal $g$, for all possible delays. An efficient way to compute the cross-correlation is indeed by using a FFT, taking the product, and then applying an inverse FFT.

Now suppose you have $k$ time series. I think the best you can do is to compute the cross-correlation between each pair of time series. This requires ${k \choose 2}$ cross-correlation computations.

| cite | improve this answer | |
  • $\begingroup$ Thanks DW. If I'm missing a data point, do you think I ought to interpolate it from the other data points given? Perhaps cubic interpolation would be enough? $\endgroup$ – Alan Wolfe Jan 8 '17 at 3:12
  • $\begingroup$ @AlanWolfe, yeah, that would be a fairly standard approach to deal with that. $\endgroup$ – D.W. Jan 8 '17 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.