Avoiding a Common Mistake with Time Series

http://svds.com/post/avoiding-common-mistake-time-series

by davidkellis • 11 years ago

161 41 11 years ago

SixSigma2• 11 years ago

I ordered the book, Quantitative Forecasting Methods by Farnum and Stanton (PWS-KENT, 1989). it was only £2.81, sounds like money well spent.

frozenport• 11 years ago

Unless the book is junk, in which case you lost potential money by reading it, perhaps well in excess of the 3 bucks :-)

awch• 11 years ago

I got the last "cheap" one, at $3.94.

SixSigma2• 11 years ago

hmm, my purchase got cancelled and now they are all 10x the price.

I reckon they had a look at the competition before putting it in the envelope

bbrazil• 11 years ago

Something I've repeatedly found useful is that when debugging and you have a conjecture, not only look for evidence that a correlation/causation is present; but also look for evidence that it isn't.

Doing a very quick A/B test helps too.

jerf3• 11 years ago

"Something I've repeatedly found useful is that when debugging and you have a conjecture, not only look for evidence that a correlation/causation is present; but also look for evidence that it isn't."

In my opinion, the absolute, utter core essence of science can be expressed simply as "Always be trying to prove yourself wrong." The human brain is extremely biased in the other direction, and it's darned good at proving itself correct. It can prove itself correct in absurdly powerful ways. Always be fighting it, always be looking for ways to prove yourself wrong. If you do it seriously, things like "the scientific method" will naturally fall out of your serious attempts and need not be surrounded by near-worship, whereas no amount of worshipfully-following a checklist of the "scientific method" without the true effort to prove yourself wrong will produce truth; the human brain is far more powerful than the "scientific method" or any such static methodology.

I have deliberately left the word 'theory' out of this post. This scientific mindset beyond that into engineering and any number of day-to-day activities.

drcomputer3• 11 years ago

Try to prove yourself wrong about believing it is better to be wrong all the time.

Insanity!

I agree with you for the most part. But there's a line. When it starts to destroy your personal identity and sense of self, you've crossed that line. Sometimes it's just better to be right and trust yourself.

Or not. That's zen, I think. Trying to have so much information about everything that you wind up overloading yourself with it and can't make heads or tails of it, because in every simple truth exists an infinite proof. Analysis paralysis!

NumberCruncher2• 11 years ago

What worked for me is findig someone who is not involved in your analysis but has an analytical background and understanding what you are working on, and asking him to prove that you are wrong. My experience is that the most analyst are happy to have an opportunity to prove that you are wrong and they are right.

cbsmith2• 11 years ago

Doing a very quick A/B test helps too.

But then there are all the ways people screw up A/B tests...

NumberCruncher2• 11 years ago

If you search long enough, especially in times of big data, you will always find some statistically significant correlation. That´s why it is getting even more important to validate your model/hypothesis. In our case over time. A correlation passing an adequate validation is high probably a causation. A not validated correlation is only an opinion.

princeb• 11 years ago

perform a dickey fuller if you are unsure if a time series is nonstationary, perhaps.

plg• 11 years ago

also: statistical tests on correlation coefficients don't test whether the correlation is "significant" or not --- they only test whether the correlation is reliably different than 0.00

So a small correlation (e.g. r=0.10) can still be "statistically significant" at p<0.001 but all this means is that r is reliably different than 0.00 --- it doesn't mean r is big

foobarian• 11 years ago

I'm not sure. The author says that adding a common component to two random time series doesn't make them correlated. But that's not true, by construction, at least using any of the simple correlation tests. It's a complicated subject explained in a confusing way.

veryluckyxyz• 11 years ago

Yeah. A better example would be to add 2 different slightly sloping lines to Y1 and Y2 random variables.

morpher• 11 years ago

I don't think that will actually change anything. Two positively slowing lines (of any slope) have, by definition correlation = 1.0. The "random data" he used is just a small amount of noise on top of the line(s).

Leszek• 11 years ago

I don't get it either. The author is adding a common dependency to two independent series makes them correlated, which to me seems trivially true. He then goes on to say that this form of correlation is uninteresting, because we're only interested in whether variations are correlated, which doesn't seem so trivially true: after all, a linear trend is a first-order variation, and if two things both increase at the same rate, then their relationship is worth at least a second look.

It looks like what the author is really trying to say is that we should pass the data through a high-pass filter, eliminating any 'expected' trends such as inflation, and instead observing if the noise of the two datasets is correlated. This is an observation that has some value, but is certainly not trivial to pick a threshold for the high-pass (certainly it's not always just a linear trend), and the mutually dependent variable can have as much noise (if not more) as the two measured data sources, so you still might get "false" correlation.

arrrg• 11 years ago

It seems to me that in the end it all comes down to your actual hypothesis whether and what is actually interesting …

tel2• 11 years ago

Increasingly I'm finding that any description of causation/correlation which (a) isn't written by a more or less formally trained statistician of some variety or (b) invoking Pearl-like causal diagrams directly has a coin-flip's chance of being actively dangerous to read.

Causation is super tricky business. You probably have an okay model of it intuitively once you've done a little stats work, but to explain it to someone else or tackle it in more exotic or hypothetical domains is an exercise is insanity if you're not armed with the very best tools.

rflrob• 11 years ago

Causation is dead simple---if you can directly manipulate the putative causal variable. Otherwise, you're left with the old saw about correlation, and while you might be able to tell a story about why there should be causation, the best you can do in that case is eliminate other potential confounds.

photon1373• 11 years ago

Adding a common deterministic trend to two random time series does not make them correlated. Intuitively, you still cannot predict the deviation from the trend of one timeseries using the deviation from the trend of the other timeseries.

His statement (after adding the constant trend) is misleading:

"Now let’s repeat the same tests on these new series. We get surprising results: the correlation coefficient is 0.96 — a very strong unmistakable correlation"

What he's calculated is the correlation between a set of points from y_1 and y_2 - and that will be large, of course - their (deterministically increasing) means have correlation 1.

The quantity that qualifies as correlation for predictability purposes is actually the correlation between the deviations from mean.

This is all fairly clear if you use the actual formula (Pearson correlation):

E[(y_1 - mu * t)(y_2 - mu * t)] / (sigma_y1 * sigma_y2)

nemo44x4• 11 years ago

You missed the point. This has to do with "time series data" which means time is the same among all the data sets. Because you know this you should remove it to test for similarity. You could apply this to any automatic variable in your domain but time & change is very obviously a typical common variable for many data sets.

nkurz2• 11 years ago

Please help me understand. Assume I have 100 years of atmospheric CO2 levels and 100 years of temperature data over the same span. These are both "time series data". What are you suggesting that I remove before testing for similarity? Does this help me to determine the degree to which one of the two variables is causing the other?

nkurz2• 11 years ago

Had he stopped there I might have concluded it was just a poor explanation. But luckily he continued:

Put another way, we've introduced a mutual dependency. By introducing a trend, we've made Y1 dependent on X, and Y2 dependent on X as well. In a time series, X is time. Correlating Y1 and Y2 will uncover their mutual dependence — but the correlation is really just the fact that they're both dependent on X. In many cases, as with Jennifer Lawrence’s popularity and the stock market index, what you’re really seeing is that they both increased over time in the period you’re looking at.

What, pray tell, is the X for which the stock market index and Jennifer Lawrence's popularity depend on? Oh, you say they're both dependent on the underlying trend... What?

nemo44x4• 11 years ago

That's right. Sort of like factoring out common variables/values in algebra I'd say. Time series is the "easy case" but I suspect identifying less obvious commons is the real art of it all.

photon1373• 11 years ago

What the author is trying to explain are the concepts of cointegration and stationarity. A useful introduction here: http://www.uta.edu/faculty/crowder/papers/drunk%20and%20dog....

lazzlazzlazz• 11 years ago

Great article, thanks very much for posting it. Nobody else seems to have captured the relevant terminology: "cointegration" and "stationarity". Those perfectly captured it for me.

gengkev• 11 years ago

where are the scrollbars??

esfandia• 11 years ago

Isn't the author exaggerating in the other direction? There is obviously correlation between the two time series. Sure, who's saying there is causation (as mentioned in the article there can be a third random variable that the first two depended on)? But also, who's to say there's no causation? Is it ok to always remove the correlated part of the two time series? What if that's the interesting part and the explanation you're looking for?

photon1373• 11 years ago

Even if you're looking for causation, detrending is usually necessary to obtain consistent estimators (in statistical terms). For example, Granger causality works with two stationary timeseries:

https://en.wikipedia.org/wiki/Granger_causality

eouw0o83hf• 11 years ago

Wow, and the graphs on http://www.tylervigen.com/ are just incredible.

nemo44x4• 11 years ago

Does this mean that if I apply this algorithm and that 2 or more time series data sets are still similar that they are in fact correlated? I find this test fascinating.

omalleyt• 11 years ago

Never use first differencing, that's crazy. It magnifies measurement error. Pass the signal through a high pass filter to remove trend

radarsat1• 11 years ago

Or simply subtract a first-order fit. (What the matlab/numpy `detrend()` function does)

Edit: I'll note that this is the same thing as subtracting a very long-windowed low-pass filter, i.e., performing high-pass filtering.

n00b101• 11 years ago

This is called spurious correlation. It's well known in financial / economic time-series analysis. The lesson is that you never measure the correlation between the PRICE LEVELS of products, instead you measure the correlation between the daily/weekly/etc CHANGE IN PRICE LEVELS.

A famous example of this:

The tale of David Leinweber, which is related in the excellent new book "Quantitative Value," illustrates this point about "stupid data miner tricks." Leinweber sifted through a United Nations CD covering the economic data of 140 countries. He found that butter production in Bangladesh explained 75 percent of the variation of the S&P 500 Index. Not satisfied, he found that if he added a broader category of global dairy products, the correlation would rise to 95 percent. Then he added a third variable, the population of sheep, and found that he had now explained 99 percent of the variation in the S&P 500 for the period 1983-'99.

(http://www.cbsnews.com/news/what-butter-production-means-for...)

Avoiding a Common Mistake with Time Series

Related Stories