Researchers claim Tweets predict The Dow

October 23, 2010 8:05 AM

Invented by Charles Dow in 1896, The Dow Jones Average ("The Dow") is perhaps the most widely known metric of equity market behaviour. Calculated as a price weighted average of thirty stocks, The Dow is generally eschewed by professional investors who prefer the broader S&P 500, a so-called market capitalisation weighted index consisting of 500 stocks. Regardless, proponents of the Dow claim its simplicity, long history and careful design as a reliable proxy of US economic activity as points in its favour. But can they now claim predicability as well?

Bollen, Mao & Zeng from the School of Informatics and Computing, Indiana Unviersity-Bloomington, integrated OpinionFinder (a tool that parses feeds and can identify positive vs negative moods), and a customised version of the Profile of Mood States tests with what they described as "large scale Twitter Feeds" to identity a non casual relationship between Tweets and subsequent performance of the Dow Jones.

In fact their research [ .pdf ] claims that they can predict the daily up or down state of the close with an accuracy of 87.6%.

How long until the Efficient Markets Hypothesis asserts itself, and this opp (if it really exists) is traded away? Or will it persist, much like the long studied Weekend Effect or January Effect?

Of course as far as calendar effects go neither have the staying power of sell in May and go away, which has been documented in England since 1694.

posted by Mutant (19 comments total)

i always keep my eye peeled for Mutant posts. thanks for the morning reading.
posted by TrialByMedia at 8:20 AM on October 23, 2010

- Twitter and the Stock Market
- War & Peace + Inflation + Secular Bull = Dow 38K?
posted by kliuless at 8:36 AM on October 23, 2010 [1 favorite]

Deeply flawed. I've done exactly this before (and hinted as much in this comment) and the level of accuracy they are claiming is severely overstated.

Fundamentally, their test data window is in 2008 and gives a strong correlation for "calm" words. The problem is clearly that the data set shows high 80s% return on the most extreme market in over 70 years. So assuming high volatility, you will get superior returns on looking for calm keywords. The obvious difficulty is with predicting periods of volatility (and the continuation thereof). This model probably wouldn't work today.

What really makes their results suspect, though, is the window date. Why pick 2008 instead of the past year? It makes ms think they are overfitting. Anyone can overfit their models post-hoc and find insanely high correlations.

However, as much as I am suspicious of the results, I do think that a wider audience for this kind of stuff is good. Thus kind of stuff gives you a good picture of what a data saturated society looks like. It's increasingly easy to build bigdata correlations with just an Internet connection and statistical toolboxes like R-lang. It's perfectly feasible to do crazy stuff like crawl geolocation checkins and weather forecasting for something as simple as predicting how many customers a coffee shop will have today.

While I don't trust this paper, it is useful to see it as a window for how a torrent of data can be used.
posted by amuseDetachment at 8:39 AM on October 23, 2010 [2 favorites]

curious what you mean by 'traded away'
posted by lslelel at 8:40 AM on October 23, 2010

Unless I'm missing something, but this seems sort of like junk:

In particular we point to a significant
deviation between the two graphs on October 13th where the
DJIA surges by more than 3 standard deviations trough-topeak.
The Calm curve however remains relatively flat at that
time after which it starts to again track changes in the DJIA
again.

If it can't track the important 2-3 standard deviation + changes, it is not that predictive a model, is it? It seems like if you had a large enough dataset you could get some nice p-values out of anything.
posted by geoff. at 8:48 AM on October 23, 2010

But can they now claim predicability as well?

Is this a cross between predicament and predictability? Very strange.

Pretty sure mods can edit what is otherwise a good summary of an interesting piece from this week. Thanks!
posted by JoeXIII007 at 8:52 AM on October 23, 2010

Whenever someone at my company shows a graph like the one with the "gray areas of significant overlap", I laugh on the inside. Sometimes I also laugh on the outside.

Bivariate fit, then show me R-squared at the very least...
posted by cman at 8:55 AM on October 23, 2010

I'm forgiving of that, as it was during periods of extreme government intervention. One could say that if the market was left to itself they would have been correct.

It's true to say that one should never believe the market is wrong and the model is right -- when trading, the market is never wrong, price is what pays, not the model. However, it's fairly useless to specially train for events that are unpredictably mandated by individuals who are not entirely using market data for decisionmaking.

I hear supposedly rumors that a lot of peoples' models are breaking down in a similar fashion these past two months because of the Fed's POMO injections.
posted by amuseDetachment at 8:58 AM on October 23, 2010

(the above comment was directed at geoff. I'm out typing on a phone so this is too hard and didn't notice)
posted by amuseDetachment at 9:00 AM on October 23, 2010

Ive read the paper. their out of sample testing period was 28 days only; i would have liked to see a little bit more. However the basic finding that calm equity market tends to rise do resonate with anyone who keeps an eye on the VIX..
posted by 3mendo at 9:23 AM on October 23, 2010

(sorry for all the typos, typing on a 9" netbook)
posted by 3mendo at 9:25 AM on October 23, 2010 [1 favorite]

amuseDetachment : What really makes their results suspect, though, is the window date.

Nah... Personally, I find it highly suspect for no more reason than that the authors publicized it before retiring as the richest people on the planet.

Seriously, a 90% correlation? People make fortunes on real trends that only hold true a mere 50.1% of the time. If true, this would make someone very, very rich, and destroy the markets almost overnight.
posted by pla at 9:48 AM on October 23, 2010 [6 favorites]

From the paper: "To predict the DJIA value on day t, the input attributes of our SOFNN include combinations of DJIA values and mood values of the past n days. We choose n = 3 since the results shown in Table II indicate that past n = 4 the Granger causal relation between Calm and DJIA decreases significantly."

This is just to say that
I have picked
the cherries
that were in
the study
and which
you were probably
saving
for future study
Forgive me
they were delicious
so sweet
and so cold

apologies to William Carlos Williams
posted by storybored at 11:22 AM on October 23, 2010 [5 favorites]

So you can predict with 90% accuracy the direction of the Dow in 2-6 days time?
I can predict with 100% accuracy it is going to rain later. Later may be some time, but I will still be accurate.
In a four day window it would be usual to have a mix of up and down days, and if you picked e.g. an upward move, three days of 100 point falls and a single day of a 1 point rise would give you a true result, but you would still be down $$$.
So, effectively untradeable.
posted by bystander at 3:35 PM on October 23, 2010

From Mutants last link to his earlier post: the summer of 2008 as well as the autumn will no doubt prove very interesting.

Interesting indeed.
posted by caddis at 8:29 PM on October 23, 2010 [2 favorites]

pla nails it.
posted by sfts2 at 9:56 PM on October 23, 2010

10 months of data? That is enough to get another small round of funding to experiment on 10 years of data.

Working with a start-up with a novel technique for portfolio optimisation, finance types are more interested in .5% of provable alpha than 50% of possible come-to-jesus alpha. There is so much money involved in these kinds of things, revolutionary gains are near impossible. Constant investment in anything that could potentially give a leg-up ensures there are few surprises.

As mentioned, 10 months of data is nothing to be surprised about. Further, their results should then work on other indices and other tools. Feeling a bit cynical here but the goal should be to match the twitter results to the existing data, not to align one brief set of results with one index and claim nirvana.
posted by nickrussell at 3:39 AM on October 24, 2010

Burton G. Malkiel would like to have a word with you all.

This is my favorite stock picker story of the moment.
posted by bukvich at 8:11 AM on October 24, 2010

My favorite part about 2008 and the Dow was how they pulled (I think) AIG and GM out of it and put in some others and magically "the market" was up 200 points the next day.

OK, my favorite part is how Dow manages to maintain the perception of relevance.

"The Dow was up today amid [make some shit up]."
posted by gjc at 10:24 AM on October 24, 2010