no one could have predicted
March 10, 2013 6:08 PM Subscribe

With sincere regret we must inform you that due to circumstances recently discovered we must immediately cease trading activity on www.intrade.com. InTrade is apparently shutting down completely, at least for a while. Previously.
posted by gerryblog (98 comments total) 13 users marked this as a favorite

And nothing of [real] value was lost.
posted by wenestvedt at 6:12 PM on March 10, 2013

I see what you did with the title.
posted by arcticseal at 6:16 PM on March 10, 2013 [4 favorites]

Those recently discovered circumstances sound pretty ominous. Good thing that the circumstances have been discovered though.
posted by Ad hominem at 6:16 PM on March 10, 2013 [2 favorites]

And nothing of [real] value was lost.

I think we're all flabbergasted that a market for trading holdings with absolutely no underlying value would turn out to have financial irregularities.
posted by gerryblog at 6:17 PM on March 10, 2013 [7 favorites]

I like that they went to the trouble of settling all the bets, and then preventing anyone from getting any of their money out. That seems completely legit.

I'm a little sad that their forums seem to be shut down. I was wondering if any of the regulars know what is going on.
posted by Elementary Penguin at 6:20 PM on March 10, 2013

There was some debate when when they were sued last year and shut down in the US whether or not the Obama administration was overstepping and cracking down on an essentially harmless entity, and stifling a young industry. I liked Matt Yglesias take on that. If you run a futures contract market, you'll have to expect to be regulated like one.
posted by gkhan at 6:22 PM on March 10, 2013 [1 favorite]

they went to the trouble of settling all the bets

I wasn't a user of the site, but I'm surprised this was in the TOS. How do you settle a bet about a future event "fairly"? The only thing you could do (it seems to me) is nullify all existing bets.
posted by gerryblog at 6:22 PM on March 10, 2013

Sounds to me like a big ol' fraud investigation underway. They can't say anything because it's ongoing <-- speculation
posted by unSane at 6:23 PM on March 10, 2013

"I think we're all flabbergasted that a market for trading holdings with absolutely no underlying value "

Isn't that how futures derivatives work?
posted by gingerest at 6:23 PM on March 10, 2013 [4 favorites]

Too powerful. They were the only poll I watched last election.
posted by telstar at 6:24 PM on March 10, 2013

Too powerful. They were the only poll I watched last election.

Did you notice how it tended to follow Nate Silver's prediction almost exactly? HMM, WHY COULD THAT BE?
posted by gkhan at 6:26 PM on March 10, 2013 [14 favorites]

Yeah, sounds like all assets are frozen pending an investigation. My bet (heh) is that the US gov't somehow found a way to snare them in an anti-money laundering (AML) or anti-terrorism funding (CFT) net.
posted by ceribus peribus at 6:30 PM on March 10, 2013

I think they figured out people were gaming the numbers to fit whatever narrative they were pushing. Whether is is gambling or futures trading, blocks of people colluding to manipulate the numbers can't be kosher.
posted by Ad hominem at 6:31 PM on March 10, 2013 [1 favorite]

Except slightly skewed towards Mitt Romney. Lemurrhea made a bunch of money during the election taking advantage of this bias.
posted by Elementary Penguin at 6:31 PM on March 10, 2013 [6 favorites]

%
posted by alms at 6:46 PM on March 10, 2013 [2 favorites]

How do you settle a bet about a future event "fairly"?

Are you sure they didn't say they willan on-settle the bets?
posted by Mr. Bad Example at 7:14 PM on March 10, 2013 [10 favorites]

Damn straight I did! ...and then gave a bunch back to the League of Women's Voters (which it turns out I couldn't get a charitable receipt for because it's American. Whoops!)

I am sad about this - Intrade is both interesting to watch and a good way to take money from right-wing idiots. On the other hand, they absolutely need to be regulated like a futures market, and my understanding is that they didn't really want to have that happen.

Realistically we don't know why they shut down, yet. I hope it ends up being pretty innocuous and that they come back up more legit than ever.
posted by Lemurrhea at 7:18 PM on March 10, 2013 [10 favorites]

There was usually a pretty good arbitrage opportunity available between intrade and the euro sports betting sites (pinnacle, etc) during election season. That appeared to be because people were buying stuff on intrade to move Romney's line favorably, i.e. someone buying up shares to move the line arbitrarily. Sad to see intrade go away.
posted by RustyBrooks at 7:19 PM on March 10, 2013 [2 favorites]

I would bet they were hacked.
posted by Brian B. at 7:26 PM on March 10, 2013 [1 favorite]

They'll be back, denominated in Bitcoin.
posted by hattifattener at 7:27 PM on March 10, 2013 [3 favorites]

RichardP, I'll take the action on that. Is there a site where we can make bets on arbitrary events?

Damnit!
posted by fatbird at 7:40 PM on March 10, 2013 [14 favorites]

Sounds to me like a big ol' fraud investigation underway. They can't say anything because it's ongoing <-- speculation

Yeah? I'll give you five to one on that.
posted by krinklyfig at 8:03 PM on March 10, 2013 [1 favorite]

I like that they went to the trouble of settling all the bets, and then preventing anyone from getting any of their money out. That seems completely legit.

Harmless? Do you know how much money online gambling cost the domestic casinos who paid for this law!? Well, actually I don't, but I assume it was enough for them to pay lobbyists to get it on the legislative radar.

I liked Matt Yglesias take on that. If you run a futures contract market, you'll have to expect to be regulated like one.

Too powerful. They were the only poll I watched last election.

Well, that was pretty stupid. All Intrade did was reflect conventional wisdom, and the market was small enough that single wealthy bidders could move it - which is exactly what happened last time: Wealthy bidders placed big bets on Romney, giving him a false edge according to Intrade. Of course, we know a lot of wallstreet insider types and other rich people like Donald Trump were totally convinced Romney would win, and obviously Intrade reflects the views of the wealthy more then the middle class or poor.

Intrade just an example of the ridiculous "free market omnipotence" ideology, that says the free market is always right, knows everything, bla bla bla. It's completely absurd.

If people want to gamble on elections, that's obviously their choice. But the idea that it could have any more predictive power compared to reading polls is ridiculous.

They'll be back, denominated in Bitcoin.

There are actually a couple of bitcoin betting sites out there, including betsofbitco.in and bitbet.us They actually do a good job of illustrating just how simple bitcoin transactions can be. They'll almost certainly have bets on the next election by the time it rolls around.
posted by delmoi at 8:06 PM on March 10, 2013 [2 favorites]

obviously Intrade reflects the views of the wealthy more then the middle class or poor.

Intrade just an example of the ridiculous "free market omnipotence" ideology, that says the free market is always right, knows everything, bla bla bla. It's completely absurd.

I would respectfully suggest that this is the sort of argument against prediction markets that is best addressed by reading the existing literature on prediction markets. Furthermore, Intrade had Obama to win, so it seems to me that the argument that Intrade yielded a false prediction because it was a tool of capitalist ideology is pretty decisively undercut by the observation that Intrade did not yield a false prediction.
posted by lambdaphage at 8:14 PM on March 10, 2013 [8 favorites]

My guess: they have discovered that the escrow account in which InTrade holds funds on behalf of traders is insufficient to cover the value on deposit due to an insider making unauthorized withdrawals from the account.

If it's anything like the other futures brokers who got caught with their pants down in the last few years, someone on the inside was using deposited funds to make speculative bets on the side and lost a bunch of money. Those brokers were regulated, too (not very well, obviously).
posted by krinklyfig at 8:18 PM on March 10, 2013

I'm assuming these laws go back to the 2006 ban on international money transfers for gambling. This is actually the law that got Nate Silver (who'd been a poker buff) into politics.

This is a law that was passed near the end of the session, right before the election - along with a ban on horse meat. It was just a desperate attempt to appeal to social conservatives, and benefit the local casino industry, doing the same double-play that guys like Jack Abramhoff excelled at: Go after the competition of the casino owners who paid you and use the Christian right as the ground troops in making that happen (because it was "anti-gambling" when actually what it was doing was fucking over)

No, actually not. I don't know why you're obsessed with gambling, but the US investigation had nothing to do with that. It was an investigation by the Commodity Futures Trading Commission for violations of the Commodity Exchange Act, which dates from 1936 (and was modified in 1974), which requires all trade in futures contracts sold to the general public to be regulated (the Commodity Futures Modernization Act of 2000 largely, and foolishly, deregulated much of this market for "sophisticated buyers", i.e. hedge funds and such, but that doesn't apply to InTrade). InTrade was selling bets on the price of oil and gold, otherwise known as "commodity futures contracts". Of course the federal government was going to shut down their US operations, it was acting plainly outside the law!

Thinking the US investigation into InTrade was initiated by some crooked casino people instead of the proper and standard exercise of authority by a federal regulator seems like a rather outlandish conspiracy theory to me.

If you want to run a site that lets people bet on who is going to be president, that's one thing. But if you're going to run a site that lets people bet on the price of commodities in the future, you're now making yourself an active participant in the US financial system, and then different rules apply. InTrade brought this on themselves.
posted by gkhan at 8:30 PM on March 10, 2013 [3 favorites]

"I think we're all flabbergasted that a market for trading holdings with absolutely no underlying value "

Isn't that how futures derivatives work?

Not exactly. The futures contract is usually deliverable at the end of the contract, meaning you get your pork bellies or whatever physically delivered to you, at least if you are still holding it at the deliverable date (in practice this happens very rarely, even by accident, except when futures are traded by direct participants in the market). Derivatives for futures such as options are also contracts which are deliverable (assigned and exercised), but in this case the underlying product is the futures contract.
posted by krinklyfig at 8:32 PM on March 10, 2013 [4 favorites]

Furthermore, Intrade had Obama to win,

Yes, predicted after the election had occurred. People were able to place bets until 7AM PST on the day after the actual voting, long after the contest was decided. AMAZING.

Obama was way, way ahead. The problem was that Intrade estimated Obama's probability of winning lower then it actually was if you just went by the polls, like Nate Silver and other polling aggressors (he's not the only one).

Intrade was showing Obama with a 55-60% probability of winning, and outside of one spike up to 80 in September, it was actually down around 55-60 even in October leading up to the actual election.

So in other words, Intrade was showing that Mitt Romney had a very realistic, almost coin-flip level chance of getting elected. That's not the same thing as saying "Obama will win". That's saying it's slightly more likely that Obama will win but it's basically a coin flip.
posted by delmoi at 8:32 PM on March 10, 2013 [2 favorites]

My guess: they have discovered that the escrow account in which InTrade holds funds on behalf of traders is insufficient to cover the value on deposit due to an insider making unauthorized withdrawals from the account.

Like Full Tilt Poker? That is a fascinating story:

The Lederer Files 2 3 4 5 6 7

Distortion of Truth in the Lederer Files, Final Summary

I wonder if sports books will start adding more proposition betting like this. I don't see why betting on an election, or whatever, needs to be regulated any more than betting on a basketball game. The proposition just needs to have a clear binary result.

Nevada Legalizes Online Gambling

I think they figured out people were gaming the numbers to fit whatever narrative they were pushing. Whether is is gambling or futures trading, blocks of people colluding to manipulate the numbers can't be kosher.

I don't think that matters to them. They probably do very little 'market making' with their own finances. It may be a benefit. The more inefficient the market, the more likely to attract new participants.
posted by Golden Eternity at 8:33 PM on March 10, 2013 [3 favorites]

If you want to run a site that lets people bet on who is going to be president, that's one thing.

Well, it's still illegal under US law even to transfer funds for Americans to let them do that.

Also, Intrade shutting down now is the result of Irish law, not US law. That they stopped taking bets from Americans last year was a separate issue.
posted by delmoi at 8:36 PM on March 10, 2013 [1 favorite]

I liked Matt Yglesias take on that. If you run a futures contract market, you'll have to expect to be regulated like one.
I'm assuming these laws go back to the 2006 ban on international money transfers for gambling. This is actually the law that got Nate Silver (who'd been a poker buff) into politics.

No. Its the Commodity Exchange Act of 1936 which it falls under.
posted by Ironmouth at 8:47 PM on March 10, 2013 [1 favorite]

Well, it's still illegal under US law even to transfer funds for Americans to let them do that.

Also, InTrade shutting down now is the result of Irish law, not US law. That they stopped taking bets from Americans last year was a separate issue.

I was only commenting on the American investigation, but the point is that InTrade was clearly a company which did not value things like "ethics" or "sound judgment". I wouldn't be surprised in the least if people were using their service for some shady shit and they were well aware of it. InTrade was the author of its own misfortune.
posted by gkhan at 8:48 PM on March 10, 2013

Yes, predicted after the election had occurred. People were able to place bets until 7AM PST on the day after the actual voting, long after the contest was decided. AMAZING.

Obama was consistently favored since trading began, excepting a brief dip in Fall of 2011 as the chart I linked shows. If Oracle A asserts that the probability of event X is 70% and Oracle B asserts that it is 60%, what do you learn about the relative predictive power of the oracles when you observe that X happens once?

Intrade estimated Obama's probability of winning lower then it actually was if you just went by the polls, like Nate Silver

Well, here's Nate Silver on the matter:

The probabilistic forecasts issued by FiveThirtyEight have been quite close to Intrade and those at other trading and betting markets over the course of the election. The spread as of Tuesday night – with Intrade implying that Mr. Obama has a 55 percent chance of winning the election, and FiveThirtyEight a 68 percent chance – is much wider than usual...

A 2009 paper by David Rothschild found that FiveThirtyEight’s forecasts were somewhat better than Intrade’s over the course of the 2008 presidential election cycle. Mr. Rothschild wrote, however, that Intrade’s forecasts would have outperformed FiveThirtyEight’s had they been corrected to adjust for the market’s tendency to overrate the chances of the trailing candidate (what is known as a favorite-long shot bias).

It's worth reading in full.

Elsewhere, Silver cited the convergence between 538 and Intrade as a validation of his model. What do you think the significant discrepancy between Intrade and 538 is?

FWIW, Silver has repeatedly (and correctly, IMHO) encouraged the use of prediction markets, especially for pollsters and analysts. I think there are reasonable criticisms of Intrade (e.g. insufficient liquidity, high transaction costs), but I'm afraid I still don't understand what yours is.
posted by lambdaphage at 9:02 PM on March 10, 2013 [3 favorites]

CFTC: Event Markets are not Gambling (pdf)

Okay, I guess it makes sense that betting on things like elections and oil prices, that have a big impact on people's actual lives, should be regulated differently than sports betting.
posted by Golden Eternity at 9:08 PM on March 10, 2013

I really wish the term "conspiracy theory" would go away. It should be the new Godwin's Law. Painting with a broad purjorative brush does not have the magical powers of delegitimizing anything you happen to think is unlikely.
posted by spock at 9:36 PM on March 10, 2013

So in other words, Intrade was showing that Mitt Romney had a very realistic, almost coin-flip level chance of getting elected. That's not the same thing as saying "Obama will win". That's saying it's slightly more likely that Obama will win but it's basically a coin flip.

You can prove anything with numbers.
posted by Blazecock Pileon at 9:52 PM on March 10, 2013 [1 favorite]

What do you think the significant discrepancy between Intrade and 538 is?

Uh, you just said what it was:

The spread as of Tuesday night – with Intrade implying that Mr. Obama has a 55 percent chance of winning the election, and FiveThirtyEight a 68 percent chance – is much wider than usual...

Elsewhere, Silver cited the convergence between 538 and Intrade as a validation of his model. What do you think the significant discrepancy between Intrade and 538 is?

First of all, a lot of betters were probably using Silver's model to place their bets. Secondly Nate Silver wasn't the only one doing predictions, there were others like Sam Wang who gave Obama better odds of winning before the election.

Finally, as the election drew to a close and Nate's "now-cast" converged with his overall estimate (including the possibility of future 'game-changers') Obama's probability of winning. So for example on November 1st, Nate had Obama with an 80% chance of winning, while Intrade had him at about 66%. On November 5th Silver had Obama with a 91% chance of winning. Intrade had him at 65%

You don't think there's a significant difference between 91% and 65%?
posted by delmoi at 10:28 PM on March 10, 2013

It's hard to imagine what they did that Irish financial law made them shut down so precipitously... betting is pretty widely understood and has a long legal history there.
posted by fshgrl at 11:27 PM on March 10, 2013

Wow, I just got my money out of Intrade last week. Intrade was incredible fun, largely because the market was so irrational and swayed by hopeful idiots. Money won is twice as sweet as money earned, and I think it doubles again when you're winning money from Ron Paul, Sarah palin, Herman Cain and Rick Santorum supporters.
posted by skewed at 11:41 PM on March 10, 2013 [5 favorites]

On November 5th Silver had Obama with a 91% chance of winning. Intrade had him at 65%

If one weatherperson said it's 90% likely to rain and another says 70% and then it rains, the former is *not* more correct. To find out which is the better estimate, you have to watch a number of predictions and outcomes. The certainty with which they make a specific correct prediction is basically meaningless statistically.

In other words:
You don't think there's a significant difference between 91% and 65%?

No, not on a single sample dataset.
posted by freebird at 11:43 PM on March 10, 2013 [6 favorites]

First of all, a lot of betters were probably using Silver's model to place their bets

That's precisely the point. If Nate Silver is a credible source of opinion about the electoral results, then it is rational to hold a position conforming to Silver's projections. That's what Silver himself publicly advocated until he was admonished by the NYT public editor for daring to suggest that we use markets to aggregate information about political outcomes.

Unless you think that Nate Silver is the true model, i.e. that Intrade was supposed to capture the state of Nate Silver rather than the state of the election, I'm not too worried. Here is the closest thing I've found to a time series comparison for Intrade and 538. (If that dataset is publicly available in a convenient format, please let me know). I don't really see what the problem is, or even really understand what you believe the problem to be. I've already suggested (in concurrence with Silver) that Intrade was hobbled by low liquidity as a result of the fact that trading on Intrade was very logistically frustrating, especially for Americans. I'm not claiming that Intrade was perfect, but it predicted the 2008 and 2012 presidencies calling 49 and 47.5 states correctly, respectively. (NB: Silver called all 50 correctly in 2012).

Given that we're talking about single-event probabilities, I don't know what more you want. Look, suppose ex ante you are indifferent between Intrade and 538: you think they are equally likely to be "the true model". Suppose, just to make the most generous assumptions possible, that P(Obama|538) = .95, and P(Obama|Intrade) = .61. (Now in reality it strikes me as fishy that we'll characterize the disagreement between the two models by the L1 norm, as it were, but I'm trying to be generous.) By Bayes' rule, P(538|Obama) = .583, and P(Intrade|Obama) = 0.416.

Now, if you think that a probability of 55-60% was "basically a coin flip" in the case of Intrade's prediction about Obama, then our posterior distribution on {Intrade,538} is "basically a coin flip" too. See what I mean about single-event probabilities?

I still don't understand what your criticism of Intrade actually is, apart from the fact that it merely reflected Nate Silver's predictions rather than embodied them wholly. I (and Silver) think that this is partly explained by illiquidity. Do you agree with that, or do you think there's another problem? If you thought Intrade was completely out to lunch on the election, were you busy in November taking money from people who thought otherwise?
posted by lambdaphage at 11:44 PM on March 10, 2013 [1 favorite]

If one weatherperson said it's 90% likely to rain and another says 70% and then it rains, the former is *not* more correct. To find out which is the better estimate, you have to watch a number of predictions and outcomes. The certainty with which they make a specific correct prediction is basically meaningless statistically.

This is only true under the frequentist definition of probability, and most epistemologists would tell you that this is just wrong. Probability as certainty (that is, the Bayesian perspective) is a different animal than probability as frequency. Probability as certainty is subject to contraints of rationality that can make one certainty judgment better than another even with a single sample. It all comes down to how you use the information you're given to update your certainties.

An example: If a new poll comes in showing high support for Obama and there's no obvious problem with it, you're compelled to shift your beliefs in favor of Obama winning. To use the data in the poll to do the opposite would not be reasonable. So if two people have roughly the same starting points, and one uses data to shift their certainties in one direction and the other in the opposite direction, we can most certainly say that of those certainty estimates is better than another, regardless of the eventual outcome. Quality certainty judgments are about rational use of data.

This is all very relevant, considering we're talking about the election in which Republicans didn't seem to be able to rationally evaluate the evidence coming in, and also that markets are assumed by some to be "rational" in the same sense.
posted by Philosopher Dirtbike at 12:44 AM on March 11, 2013 [3 favorites]

If one weatherperson said it's 90% likely to rain and another says 70% and then it rains, the former is *not* more correct. -- freebird

Yes they are.

If you think there is a 90% chance of something occurring, and someone else thinks there's an 70% chance of something occurring, then it would be in their interest to take you up on a 4:1 bet that it won't happen, because in the long run they'll make money if they're correct and you'll make money if you're correct.

(And in fact, if it doesn't rain, the person estimating a 70% chance is actually more correct then the person guessing 90%)

No, not on a single sample dataset. -- freebird

Huh? You mean a data set with one sample?

That doesn't really make any sense. If there's was no difference between 65% and 91% then there isn't any difference between 51% and 49% either. But lambdaphage said that intrade picked Obama to win, indicating he thought that the percentage probability mattered. And it does.

And if you think the difference between 51% and 49% is more significant then the difference between 65% and 91%, then you don't know anything about statistics or probability at all.

That's precisely the point. If Nate Silver is a credible source of opinion about the electoral results, then it is rational to hold a position conforming to Silver's projections. -- lambdaphage

Right, and if a people are irrational then they won't hold a position conforming to Silver's projections, which is what intrade's market, along with lots of republicans did. Most republicans listened to Dick Morris and Karl Rove and convinced themselves they were going to win, when in reality the data showed it wasn't going to happen.

I'm not too worried. Here is the closest thing I've found to a time series comparison for Intrade and 538. (If that dataset is publicly available in a convenient format, please let me know). I don't really see what the problem is -- lambdaphage

Maybe because the scales are different? Intrade is off by several percentage points and diverges in the last few days of the election, when silver's model no longer needs to account for future events.

or even really understand what you believe the problem to be. -- lambdaphage

Yeah dude, Intrade had Romney with a 45% chance of winning the day before the election. That means it sucked. Not sure why this is so confusing. Intrade did a terrible job of predicting the election.

Given that we're talking about single-event probabilities, I don't know what more you want. Look, suppose ex ante you are indifferent between Intrade and 538: you think they are equally likely to be "the true model". -- lambdaphage

Except you would never do that, because Nate Silver's model has a rational basis, and Intrade's does not.

___

Anyway, the original question was "What do you think the significant discrepancy between Intrade and 538 is?"

And the answer is one had 91% probability of Obama winning, and one had a 65% probability of Obama winning. That's a discrepancy.

Given that we're talking about single-event probabilities, I don't know what more you want. Look, suppose ex ante you are indifferent between Intrade and 538: you think they are equally likely to be "the true model". Suppose, just to make the most generous assumptions possible, that P(Obama|538) = .95, and P(Obama|Intrade) = .61. (Now in reality it strikes me as fishy that we'll characterize the disagreement between the two models by the L1 norm, as it were, but I'm trying to be generous.) By Bayes' rule, P(538|Obama) = .583, and P(Intrade|Obama) = 0.416.

P(538|Obama) = .583
P(Intrade|Obama) = 0.416

Your own math shows that there is a less then 50% probability of the random variable you labeled "Intrade" being "true" - I'm not sure what you mean for that variable to represent (and what are you using for P(Intrade) and P(538)?)
posted by delmoi at 12:47 AM on March 11, 2013 [2 favorites]

So if

P(538|Obama) = (P(Obama|538) * P(538))/P(Obama) = .583 = (0.95 * P(538))/1

and

P(Intrade|Obama) = (P(Obama|Intrade) * P(Intrade))/P(Obama) = .416 = (0.61 * P(Intrade))/1

Then the input you used in your calculations was P(538) = 0.61 and P(Intrade) = 0.68. Where the hell did those numbers come from? And why are you giving Intrade higher independent probability?

If you use a 50% probability for both - you get P(538|Obama) = 0.95*0.5/1 = 42.5% and P(Intrade|Obama) 0.61*0.5/1 = 30.1%
posted by delmoi at 1:10 AM on March 11, 2013 [1 favorite]

Oops, actually For P(Intrade) = P(538) = 0.5 P(538|Obama) should be 47.5 not 42.5. And P(Intrade|Obama) is 30.5 not 30.1. (I did those two computations in my head as I was typing)
posted by delmoi at 1:23 AM on March 11, 2013 [1 favorite]

This is all very relevant, considering we're talking about the election in which Republicans didn't seem to be able to rationally evaluate the evidence coming in, and also that markets are assumed by some to be "rational" in the same sense.

That is the part I don't believe. They would have us believe that the republican machine, which can afford the best pollsters, was taken by surprise. We a supposed to believe they thought their guys were right and everyone else was wrong because of mittmentum or something. I think they shaped their polling results and manipulated intrade to show Romney had a much better chance than he actually did. Maybe some thought everyone loves a winner, if people think Romney has a shot they will come out to vote. I think some kept the sham going to keep donations coming in. Remember the spate of stories asserting that billionaire donors were pissed? I think eventually they found out they got took and Romney never had a chance at all.
posted by Ad hominem at 1:36 AM on March 11, 2013 [1 favorite]

lambdaphage, I also don't understand why you would even think, in theory, that a futures market would be a good predictor of election results.

In the stock market, the market is reflecting the sum of the varied information that individual investors have. It's reasonable to suppose the aggregate has more information than any single investor.

But in the election futures market, all the investors have essentially the same information -- the polls. There's no reason to think that Intrade in aggregate has any more information than any individual person who follows the polls.
posted by straight at 1:42 AM on March 11, 2013 [1 favorite]

That is the part I don't believe. They would have us believe that the republican machine, which can afford the best pollsters, was taken by surprise.

I actually do think most of them believed it. There's no way you could really have a conspiracy that large where everyone says they believe they are going to win, but actually they all believe they are going to lose, because it order to do that, they would need to co-ordinate.

Secondly, we know who some of their pollsters were, guys like Dick Morris - who were absolutely ~~cooking~~ 'unskewing' their books. Maybe Dick Morris, personally, knew the republicans were going to lose, but he never communicated that. So anyone who listened to him and took him seriously would have thought that Romney really would win.

Also, reporters who interviewed them all said they seemed genuinely shocked when they lost. It's one thing to fake bravado before an election, but faking crushing disappointment and disillusionment seems after an election seems less likely.
posted by delmoi at 2:13 AM on March 11, 2013

Delmoi, your math is incorrect. The problem with your calculations lies in the assumption that P(Obama) = 1. But it is not 1.

P(Obama) = P(Obama | Intrade) *P(Intrade) + P(Obama | 538) * P(538)

Under the assumption that P(Intrade) = P(538) = 0.5,

P(Obama) = .95 * .5 + .61 * .5

Assuming, of course, that 538 and Intrade exhaust the hypothesis space. Under those assumptions, the posterior distribution is indeed 0.583 for 538, and 0.416 for Intrade as I stated above. My point about the posterior was this:

Now, if you think that a probability of 55-60% was "basically a coin flip" in the case of Intrade's prediction about Obama, then our posterior distribution on {Intrade,538} is "basically a coin flip" too. See what I mean about single-event probabilities?

Under the most generous assumptions we could muster about the difference between 538 and Intrade, we get a posterior distribution which favors 538 by a margin you had previously declared to be negligible.

Your point about the irrationality of participants in a prediction market has been anticipated by folks who study this stuff (see Wolfers and Zitzewitz 2004). The participants in a prediction market don't need to be rational in the sense that they bet in order to maximize their return. They might be partisan enthusiasts who bet out of emotive solidarity with their candidate, or cynical operators attempting to "manipulate" the market. All that is required is the rationality of at at least some participants; their rationality allows them to profit from the irrationality or cynicism of others.

Indeed, irrational participants can increase the efficiency of prediction markets (Hanson and Oprea 2007): if all agents were constrained by some pretty mild assumptions about how informed, rational people act, then the price would quickly equilibrate to reflect the available information (assuming sufficient volume &c.) and no one would have an expectation of positive return. Irrational agents drive prediction markets by baiting rational agents to take their action. What do you do if some naive fat cat offers to buy a $1 Romney ticket for $1? You sell it happily, and the price on Romney goes back down. Attempts to "manipulate" the market merely result in the manipulator losing a bunch of money.

straight,

I wouldn't claim that a prediction market generates new information about the election, just that it efficiently digests it. As this summary from 2008 shows, polling organizations can differ dramatically in their reliability; prediction markets provide an incentive to sift through them and extract signal.
posted by lambdaphage at 2:18 AM on March 11, 2013 [5 favorites]

> Here is the closest thing I've found to a time series comparison for Intrade and 538. (If that dataset is publicly available in a convenient format, please let me know).

Here's a better comparison. (I made this the morning of the election.) The 538 dataset is available in an inconvenient format. The Intrade dataset is a little friendlier.
posted by ecmendenhall at 2:23 AM on March 11, 2013 [3 favorites]

Finally, if prediction markets are so irrational, then seriously: why aintcha rich?
posted by lambdaphage at 2:24 AM on March 11, 2013 [1 favorite]

Delmoi, your math is incorrect. The problem with your calculations lies in the assumption that P(Obama) = 1. But it is not 1.

Are you saying you're not sure if Obama won? Of course the probability of Obama having won the election is 1.

The whole point of Bayes theorem is that you can use new information (like who won the election) to figure out other things (like which model was more accurate)

Under the assumption that P(Intrade) = P(538) = 0.5,

P(Obama) = .95 * .5 + .61 * .5

Assuming, of course, that 538 and Intrade exhaust the hypothesis space. I

Why would you assume that? In fact, we know the true probability of either Intrade or 538 being a perfict model is zero actually zero, since there is an infinite space of possible models. What we actually want to know is which one is better.

Under those assumptions, the posterior distribution is indeed 0.583 for 538, and 0.416 for Intrade as I stated above.

Even with that whack assumption of P(Obama) = 0.78, your computation is just straight up wrong. You actually end up with

P(538|Obama) = (0.95 * 0.5) / 0.78 = 0.60897 And P(Intrade|Obama) (0.61*0.5)/0.78 = 0.39102

___

Under the most generous assumptions we could muster about the difference between 538 and Intrade, we get a posterior distribution which favors 538 by a margin you had previously declared to be negligible.

Uh, no. I said a 60% probability of winning is practically coin flip compared to 91%, which I stand by. Secondly, you you just straight up did the computation wrong and the difference between 538 and intrade, even if we accept that P(Obama) = 0.78 is actually 21.7% twice as great as the difference between 60% and a coin flip (60% - 50% = 10%)

Finally, if prediction markets are so irrational, then seriously: why aintcha rich?

this guy paid off 10% of his student loans betting against Romney on intrade.

Sadly for me, I didn't even realize Americans were even allowed to use it during the election - I just assumed we were banned like other online gambling sites.

Also, while 40% profit over a few months is nice, it's not going to make anyone rich, unless they're already close. I don't know why you think this is such a great argument.
posted by delmoi at 2:54 AM on March 11, 2013 [2 favorites]

If one weatherperson said it's 90% likely to rain and another says 70% and then it rains, the former is *not* more correct. -- freebird
Yes they are.

If you think there is a 90% chance of something occurring, and someone else thinks there's an 70% chance of something occurring, then it would be in their interest to take you up on a 4:1 bet that it won't happen, because in the long run they'll make money if they're correct and you'll make money if you're correct.

(And in fact, if it doesn't rain, the person estimating a 70% chance is actually more correct then the person guessing 90%)

I agree with your overall point that Intrade was biased towards Romney, especially late in the race when most statistical models converge on a clear likely winner. But I think the point freebird was making is that you can't use hindsight to say that there was a 100% chance of whatever the actual outcome was and judge predictions based on that. For example, say I think there's a 1 in 52 percent chance that the Ace of Spades will be randomly drawn from a deck of cards, and someone else says there is a 1 in 5 chance. If the Ace of Spades does end up being drawn, that does not mean that the 1 in 5 prediction was better than the 1 in 52 prediction, or that the best possible prediction before the random drawing would have been that there was a 1 in 1 chance that the Ace of Spades would be drawn. The true chance of a given candidate winning cannot be determined simply by looking at the results from that single case, in the same way that a weather forecasting system can't be evaluated from a single day of weather.
posted by burnmp3s at 5:43 AM on March 11, 2013 [1 favorite]

delmoi,

I accidentally performed the computation with:

P(Obama|Intrade) = .91 and
P(Obama|538) = .65

instead of:

P(Obama|Intrade) = .95 and
P(Obama|538) = .61.

I just transposed the last digits when I typed them on my machine. With the correct conditionals, the posterior is indeed:

P(Intrade|Obama) = .609 and
P(Obama|538) = .391

rather than:

P(Intrade|Obama) = .583 and
P(Obama|538) = .416

Thanks for catching that.

But about Bayes' theorem: P(Obama) is not 1. P(Obama|Obama) is 1, but P(Obama|Obama) is not the required normalization constant. See here for a fully worked example. Notice that in the linked example, P(L) is not 1 either.
posted by lambdaphage at 7:22 AM on March 11, 2013

But I think the point freebird was making is that you can't use hindsight to say that there was a 100% chance of whatever the actual outcome was and judge predictions based on that.

Actually you can, once something has happened you actually do know the probability of it happening was 1.

For example, say I think there's a 1 in 52 percent chance that the Ace of Spades will be randomly drawn from a deck of cards, and someone else says there is a 1 in 5 chance. If the Ace of Spades does end up being drawn, that does not mean that the 1 in 5 prediction was better than the 1 in 52 prediction

Well, it could be that the person who guessed 1 in 5 had some additional knowledge about how the deck was stacked or how it would be drawn. Or they might have known that 10 of the cards in the deck were actually aces of spades.

in the same way that a weather forecasting system can't be evaluated from a single day of weather.

Actually a single data point can be used to evaluate things, just with a large margin of error - but you have much, much more information about it then you would if you had zero data points.

(Also, Nate Silver, and the other poll aggregators also made predictions for all 50 states. Silver got all of them right, and that could be used to verify the model - I don't know if Intrade had bets for each of the states.)

But about Bayes' theorem: P(Obama) is not 1. ... See here for a fully worked example.

No, you're missing the point. In that example, they calculate the probability that any random person, man or woman has long hair as being 0.52 = P(L)

On the other hand - in the example, if you asked if the person had long hair - then you could set P(L) = 1 (or if you wanted to be really precise you could figure it as .999... depending on how accurate they are at measuring hair length)

What you're doing is taking the information we had before the election as and trying to evaluate them without using the actual outcome of the election at all - it doesn't figure into your calculation, the math would be exactly the same after the election then before it. In fact, you would have the same result if Romney had won. Obviously that's not what you want.

You're using the average of the two as the 'true' value, which makes no sense at all - in fact it just favors whichever estimate is more extreme. If Intrade had put Obama's chances at zero, the averaging method would put P(Obama) at 0.455 and thus make Intrade "more accurate", despite the fact that Obama won. Again, that's obviously not a useful calculation.

What I am saying is that given the outcome of the election - that is to say, given P(Obama) = 1 - we are trying to evaluate how good the two models were now that we know Obama won the election.

Anyway, a couple of points: 1) It doesn't really matter what kinds of calculations you can do assuming both methods might be equally valid. We know Nate Silver's method is based on sound statistics, and he also got all 50 states right. There are other statisticians doing the same kind of predictions, and some of them were even more certain about Obama's victory further out - The problem is there's no reason to assume that Intrade is accurate in the first place, other then "free market magic"

2) The original question was just whether or not Intrade and Nate Silver made different predictions. It's obvious that they did. It's obvious that a 0.95 prediction and a 0.61 prediction are significantly different predictions.

Another way to look at is the amount of information entropy between the prediction and the actual result. If you have a (fair) coin flip and the election - which should be totally uncorrelated, those two things give you two bits of information - or the election gives you one more bit given the coin flip and vice versa.

With 538's prediction of 0.95 the election and the prediction combined 1.286 bits, so Nate is giving you 0.714 'bits' of information about the election, assuming Nate is correct.

On the other hand, Intrade's prediction of 0.61 probability of Obama winning takes up 0.964 bits (or 1.964 bits for both the prediction and the election), which means intrade is only "giving" us 0.036 "bits" of information, in this case also assuming Intrade is 'correct' about the outcome. (And a coin flip, of course gives us zero bits of information about the election)

That's what I mean when I say it's not much better then a coin flip. It's not telling you much Nate is giving almost twenty times as much information.

You're comparing things by looking at raw quantities of probabilities. But there is actually a much greater difference in information quantity between 0.9 and 1 then there is between 0.5 and 0.6. (in fact even H(0.98) - H(0.99) > H(0.5) - H(0.6))

And finally, even if you do look at it in terms of raw quantities of probability, Intrade was only 5-10 % points off from a coin flip, while Nate Silver's predictions differed from Intrade's by 34 percentage points.

My original point was simply that Intrade was bad because it differed from Nate silver's prediction by a wide margin. The main reason for thinking 538 is better is simply that it has a sound model and has been tested over multiple cycles. It also predicted state by state results, senate seats, and so on.

I realize this got kind of long to sum up again - There are two separate issues: Whether or not intrade and 538 differed in their predictions (Yes: there was a huge difference in terms of information content, and a big difference in terms of probability quantity) and whether or not we can say that 538 did better given the results of the election (again, yes)
posted by delmoi at 8:53 AM on March 11, 2013

delmoi,

You're still making incorrect claims about Bayes' theorem. P(Obama) is the prior probability of Obama being elected, and P(L) is the prior probability of encountering someone with long hair, both of which we compute via the law of total probability. Do you deny the law of total probability?

Missing this point can lead you to incorrect inferences. For example, you claim:

If Intrade had put Obama's chances at zero, the averaging method would put P(Obama) at 0.455 and thus make Intrade "more accurate", despite the fact that Obama won. Again, that's obviously not a useful calculation.

This is not correct. If P(Obama|Intrade) = 0, then by Bayes' theorem P(Intrade|Obama) = 0, and P(538|Obama) = 1. That does not "make Intrade 'more accurate'". To the contrary, it rejects Intrade as strongly as possible.

This argument has become straight-up Carrollian. If you are so confident that your interpretation of Bayes' theorem is correct, let's bet on it. Let's post this question to the stats stackexchange and let them adjudicate the dispute. Whoever is wrong will pay a sum of your choice to a charity of the winner's choice. My charity of choice is Against Malaria; here is a list of reputable charities if you would like some reputable advice about choosing one.

I am willing to wager any amount, because my confidence that I have the correct interpretation of Bayes' theorem is about equivalent to my confidence that 2+2=4, that there exists an external world, &c.

Looking forward to hearing from you!
posted by lambdaphage at 9:31 AM on March 11, 2013 [5 favorites]

Finally, if prediction markets are so irrational, then seriously: why aintcha rich?

Because not even Nate Silver is right all the time. He could be 10x more accurate than the prediction markets but still not be accurate enough to make me willing to stake significant amounts of my money betting against the Intraders.

I wouldn't claim that a prediction market generates new information about the election, just that it efficiently digests it...prediction markets provide an incentive to sift through them and extract signal.

But there are lots of people with plenty of incentive to do that. I don't know why you would think the Intrade aggregate would be more likely to get it right than any particular smart person who watches the polls. The amount of genuine information to sift through is not so large that a mob is going to be more efficient and effective than an expert.
posted by straight at 10:14 AM on March 11, 2013 [1 favorite]

That appeared to be because people were buying stuff on intrade to move Romney's line favorably, i.e. someone buying up shares to move the line arbitrarily.

Well, it also probably had something to do with Intrade being open (until a few months ago) to Americans, while the Euro sports books aren't.
posted by Kadin2048 at 10:43 AM on March 11, 2013 [1 favorite]

Bloomberg:

Intrade’s auditor, Dublin-based Caulfield Dunne Accountants & Registered Auditors, last month raised concerns about payments made to accounts controlled by Delaney, the company founder who died in May 2011. Delaney died at 42, less than 50 yards (46 meters) from the summit of Mount Everest, the New York Times reported at the time.

The firm had “insufficient documentation regarding payments made into bank accounts in the name of a deceased former director and other third parties,” the auditor said in the report, signed off on Feb. 4. “We have not obtained all the information and explanations we consider necessary for the purpose of our audit,” it said.

posted by unSane at 12:22 PM on March 11, 2013 [1 favorite]

Furthermore, Intrade had Obama to win, so it seems to me that the argument that Intrade yielded a false prediction because it was a tool of capitalist ideology is pretty decisively undercut by the observation that Intrade did not yield a false prediction.

I take it you don't understand relative betting odds, do you?
posted by Mental Wimp at 12:36 PM on March 11, 2013 [1 favorite]

Silver: Even Without Intrade, Billions Will Be Bet on 2016 Race
posted by the man of twists and turns at 12:38 PM on March 11, 2013 [2 favorites]

Probability as certainty is subject to [constraints] of rationality that can make one certainty judgment better than another even with a single sample. It all comes down to how you use the information you're given to update your certainties.

E.g., using it to set betting odds.
posted by Mental Wimp at 12:48 PM on March 11, 2013

Irrational agents drive prediction markets by baiting rational agents to take their action. What do you do if some naive fat cat offers to buy a $1 Romney ticket for $1? You sell it happily, and the price on Romney goes back down. Attempts to "manipulate" the market merely result in the manipulator losing a bunch of money.

Also, think about this purported mechanism. The feedback loop here that supposedly rewards rational investors and punishes irrational investors doesn't kick in until after the election is over. Before the election, there's no way just looking at Intrade whether or not you're seeing a Romney bubble or an Obama bubble. One of the candidates could be overvalued and the market correction only comes on the day of the election.
posted by straight at 1:26 PM on March 11, 2013

gkhan: "Thinking the US investigation into InTrade was initiated by some crooked casino people instead of the proper and standard exercise of authority by a federal regulator seems like a rather outlandish conspiracy theory to me."

The idea that seems outlandish is that federal regulators – especially of financial markets – exercise their authority in a proper, standard way.
posted by dendrochronologizer at 3:06 PM on March 11, 2013

Well, there's also the added complication that 538's predictions almost certainly figured into trading decisions on Intrade, but not vice versa, so they're not really two independent models. If anything, you could probably describe the 538 model as a subset of the Intrade model.
posted by en forme de poire at 3:28 PM on March 11, 2013 [1 favorite]

I was thinking... if people really made bets on inTrade in favor of the guy they wanted to win, that's a classic Texas hedge... your guy wins and you make some $$$, your guy loses, and not only are you upset, but you lose money on top of it...

I like to fancy that the blip in Romney support in October was actually an Obama supporter buying some protection in big size ... in the off chance that he lost, they would have at least gotten some cash to compensate...
posted by pravit at 4:55 PM on March 11, 2013

Mental Wimp,

I take it you don't understand relative betting odds, do you?

I'd be happy to hear your substantive criticism of my view. There is an entire philosophy of statistics dedicated to the proposition that probabilities of single events are meaningless. As a "Bayesian fundamentalist", however, I am quite happy to assign probabilities to single events! I'm not claiming that a single observation is uninformative, or that there is no difference between models that estimate the true probability at 0.65 vs. 0.75. My claim is that a single observation doesn't do very much to distinguish between those models. It does a little bit to shift the posterior over those models, which we can quantify. Before the election, we had 1 bit of uncertainty as to whether Intrade or 538 was the true model (granting the assumptions I outlined above). After we observe that Obama was re-elected, we have about 0.98 bits of uncertainty. Is Obama's re-election "informative" in our model selection problem? Sure, it gives us 0.02 bits.

straight,

not even Nate Silver is right all the time. He could be 10x more accurate than the prediction markets but still not be accurate enough to make me willing to stake significant amounts of my money betting against the Intraders.

I'm not sure what you mean by 10x more accurate, but suppose it means that Intrade "knows" the probability to 10%, and Silver knows it to 1%. Suppose Intrade says 0.6, Silver says 0.65. By buying an undervalued share of Obama, you get a 15% expected ROI. That is better than you'll get with a CD. Yes, it's a single event and you can't (AFAICT) hedge that position, but life is just a succession of single events; if you face a bet with positive expected value, you should take it. Another one will come along soon enough :)

I wouldn't claim that a prediction market generates new information about the election, just that it efficiently digests it...prediction markets provide an incentive to sift through them and extract signal.

But there are lots of people with plenty of incentive to do that. I don't know why you would think the Intrade aggregate would be more likely to get it right than any particular smart person who watches the polls.

The problem was that a lot of people, including most of the people who made livings talking about the election, didn't have the incentives to accurately forecast it publicly. A prediction market aligns the incentives by forcing pundits to put their money where their mouths are. A bet, as Alex Tabarrok noted, is a tax on bullshit. See this paper by Ken Arrow et al. for more. In the worst case a prediction market does no better than the predictions of experts, but that's quite a far cry from the view that prediction markets are "free market magic".

Also, think about this purported mechanism. The feedback loop here that supposedly rewards rational investors and punishes irrational investors doesn't kick in until after the election is over. Before the election, there's no way just looking at Intrade whether or not you're seeing a Romney bubble or an Obama bubble. One of the candidates could be overvalued and the market correction only comes on the day of the election.

Again, if Romney supporters/operators try to bid Romney up to a dollar months before the election, sober investors should be happy to give them that action. Indeed, the higher the bubble goes the more incentive there is to enter the market to do just that. (You could actually see this happen sometimes on twitter, as people urged their friends to take advantage of over-enthusiastic rubes.) The true probability may fluctuate (Obama might have tanked in the debates, &c.), but actors always have the same incentive to bid the price into alignment with their best current estimates of that probability. Otherwise you'd be leaving money on the table.

Erratum:

An earlier comment should have read:

I just transposed the last digits when I typed them on my machine. With the correct conditionals, the posterior is indeed:

P(Intrade|Obama) = .609 and
P(538|Obama) = .391
rather than:

P(Intrade|Obama) = .583 and
P(538|Obama) = .416

Apologies for any confusion. Those responsible have been sacked.
posted by lambdaphage at 5:37 PM on March 11, 2013 [2 favorites]

Also, think about this purported mechanism. The feedback loop here that supposedly rewards rational investors and punishes irrational investors doesn't kick in until after the election is over. Before the election, there's no way just looking at Intrade whether or not you're seeing a Romney bubble or an Obama bubble. One of the candidates could be overvalued and the market correction only comes on the day of the election. -- straight

Just like how stocks jump up or crash when earnings reports come out depending on whether or not the stock "beats the street" or whatever - the market may be good at distilling known information, but it isn't always that good at making predictions about the future.

___

here is a list of reputable charities if you would like some reputable advice about choosing one.

Lol, I see you're new around here.

You're still making incorrect claims about Bayes' theorem. P(Obama) is the prior probability of Obama being elected, and P(L) is the prior probability of encountering someone with long hair, both of which we compute via the law of total probability. Do you deny the law of total probability?

*sigh*. You're not really making any sense. When the probability of Obama being elected goes to 1, the probability of Romney being elected goes to zero, because 1+0 = 1, the total probability stays equal to one, thus the law of total probability isn't violated _{(Obviously there is a slight probability that neither wins, like a meteor hits a debate venue or something - we can ignore that)}

Similarly, in the hair example P(L) + P(~L) = 1, and if we set P(L) =1 then P(~L) goes to zero.

The problem here is that you don't seem to think that the probability distribution of random variables can change when we get new information. In fact, they can and do change when we observe events.

This is not correct. If P(Obama|Intrade) = 0, then by Bayes' theorem P(Intrade|Obama) = 0, and P(538|Obama) = 1. That does not "make Intrade 'more accurate'". To the contrary, it rejects Intrade as strongly as possible.

Hmm, yeah the math actually favors whichever prediction was higher probability - I guess I was thinking that whichever bet was more extreme would be more certain in the sense of being closer to one or zero. My mistake.

__

This argument has become straight-up Carrollian. If you are so confident that your interpretation of Bayes' theorem is correct, let's bet on it. Whoever is wrong will pay a sum of your choice to a charity of the winner's choice. My charity of choice is Against Malaria;

Well, there's a problem with that, which you actually explain in your next comment.

The problem was that a lot of people, including most of the people who made livings talking about the election, didn't have the incentives to accurately forecast it publicly. A prediction market aligns the incentives by forcing pundits to put their money where their mouths are.

That's only true of real bets, not charity bets. For a game theoretic rational actor, the problem is that they gain nothing if they win, but actually lose if they lose. So even if they're 99% sure they're right, the expected value is still negative. From an altruistic perspective, it's even worse - the person who loses gets a "warm fuzzy" feeling, so not only do you not win, you don't even get to make the loser feel bad! The worst of all possible outcomes.

Now, on the other hand, if you offered to make a real bet, a rational actor who was sure they were more likely to be right couldn't really turn it down, right?

The other problem with this is simple: What exactly are we arguing about? I'm not clear exactly what it is you're trying to argue. It seems like you're saying you think Intrade and 538's predictions were indistinguishable somehow, and/or that the probability that Obama won the election is 0.78, while I think it's 1 (discounting nutjob conspiracy theories that ACORN stole the election, of course) - but those both seems totally crazy to me.

What do you actually think my argument is? Do you even understand it?

Now, I'll try to boil down my argument to a couple of statements I think are true:

1: I think the probability of Obama having won the election is closer to 1 then it is to 0.78

2: I think Nate Silver's prediction was "better" then Intrade's by an "amount" greater then the difference between Intrade and a coin flip.

So, which of those two statements do you think is false? If you think I'm arguing something else, then what? Can you boil your argument down to a simple statement you think is true and that you think I think is false?
posted by delmoi at 9:14 PM on March 11, 2013

(There's also a third point, which is more complex and subjective, whether or not the prior probability of Obama being elected "changes" when the election ends - now in one sense it obviously does in that the probability of neither Obama or Romney being elected goes to zero - or it could be that the probability changes over time as people make decisions and people either make it or don't make it to the polls, but that the probability the day before the election and the day after the election are very similar - or you could say that the probability was always fixed and we just didn't know it until after the election. But either way, you basically end up with P(Obama) = 1 - ε very close to the election, but my first point relates to the probability of Obama having won the election at this point in time - since you don't think probability distributions change over time this shouldn't be an issue for you, as far as I can tell)
posted by delmoi at 9:25 PM on March 11, 2013

Again, if Romney supporters/operators try to bid Romney up to a dollar months before the election, sober investors should be happy to give them that action. Indeed, the higher the bubble goes the more incentive there is to enter the market to do just that. (You could actually see this happen sometimes on twitter, as people urged their friends to take advantage of over-enthusiastic rubes.) The true probability may fluctuate (Obama might have tanked in the debates, &c.), but actors always have the same incentive to bid the price into alignment with their best current estimates of that probability. Otherwise you'd be leaving money on the table.

But the whole point of this argument here is the claim that the futures market is a good source of information for someone who wants to predict the outcome of an election. If "sober" investors already know enough to identify and take advantage of a bubble, then why are we looking at the futures market to predict the election when we could just be using whatever their source of information is?

And if you look at real-life market bubbles, they can grow for a long time before you start seeing a significant number of "sober" investors taking positions against it. And that's when you can actually see businesses failing or borrowers defaulting. Here, with the outcome of the election unknown, it seems entirely plausible to me that you could get more investors jumping onto the bandwagon for a particular candidate (urging each other to take advantage of those poor saps who don't understand that our candidate is going to win) than investors who are able to digest the polls as accurately as possible and bet against them.
posted by straight at 2:19 AM on March 12, 2013 [1 favorite]

You're still making incorrect claims about Bayes' theorem. P(Obama) is the prior probability of Obama being elected, and P(L) is the prior probability of encountering someone with long hair, both of which we compute via the law of total probability. Do you deny the law of total probability

It's been too long since I studied probability for me to figure out who's right here (if anyone) but I don't see any way at all to ascertain P(Obama) objectively under that definition. It's not at all like P(L). We don't know what the probability of Obama being elected was before the election: we only know that he was elected.

Even under that definition, our best guess of P(Obama) is the mean of all the trials - I can't imagine any other definition that makes any sense. There was one trial, and the result was 1, so the mean is 1. Because it's a single trial we have no information about the distribution so we can't establish any confidence intervals.

I think the truth is this is a degenerate case.
posted by unSane at 4:38 AM on March 12, 2013

When the probability of Obama being elected goes to 1, the probability of Romney being elected goes to zero

Folks, P(Obama) != P(Obama|Obama). P(Obama) is the prior probability of Obama being elected, assuming a prior over a class of models. This need not be, and generally is not, the true probability of Obama being elected. It is just the probability of Obama being elected according to each model, weighted by our confidence in the models. If it makes you feel any better, you can just think of it as a normalization constant that makes the posterior probabilities sum to 1.

Or we can rederive Bayes' theorem and see that there's nothing spooky going on:

O = Obama elected
I = Intrade is the true model
F = 538 is the true model
Assume I and F are the only alternatives.

(1) P(I|O) = P(I,O)/P(O) by definition.
(2) P(O) = P(O|I)P(I) + P(O|F)P(F) by law of total probability.
(3) P(I|O) = P(I,O)/(P(O|I)P(I) + P(O|F)P(F)) from (1), (2)
(4) P(I|O) = P(O|I)P(I)/(P(O|I)P(I) + P(O|F)P(F)) by definition of conditional probability.

At no point did we assume anything about P(O) except that we could decompose it according to the LTP. It need not be "ascertain[ed] objectively". Further, if you conclude that the prior probability of an event "becomes 1" when you observe the event, nothing else in probability theory makes sense. The probabilities P(F|O) and P(I|O) no longer even sum to 1 despite the assumption that F and I are exhaustive. The prior probability P(O) does not "go to 1", any more than the probability of obtaining heads from a fair coin goes to 1 when you flip it and get heads. Anyone who accepts that the prior probability P(O) = 1 after one observation should also be willing to bet that a fair coin will always land heads up after observing one heads. If anyone wants to make that bet, let me know.

Should it bother us that, in reality, I and F are not the only two possible models? No: if we are constraining our model selection problem to those two models, i.e. we want to know which one is better, then our posterior probabilities P(I|O) and P(F|O) will be off by the same error in the normalization constant, but they will still be directly comparable. Leaving out other hypotheses will not reverse our comparison of I and F.

If anyone is still unclear about this, I highly suggest sitting down in a quiet room with a cup of coffee and working a few examples until everything becomes clear. Failing to distinguish conceptually between prior and posterior distributions can lead, with all due respect, to errors such as the following:

The problem here is that you don't seem to think that the probability distribution of random variables can change when we get new information. In fact, they can and do change when we observe events.

This is precisely the point of Bayes' theorem: we update the distribution upon receipt of evidence. Crucially, we get a posterior distribution on hypotheses given the evidence, i.e. P(I|O) and P(F|O). {P(I),P(F)} was our distribution on the hypotheses before we got new information, and {P(I|O),P(F|O)} is our distribution after we got it.

Bayes' theorem may be counter-intuitive, but understanding it deep in your bones is just a life skill and should be considered part of a complete liberal arts education. I highly recommend consulting a text in probability theory (many are available online for free) if you don't trust my exposition of it, but you're not going to find anyone who will tell you different.
posted by lambdaphage at 9:08 AM on March 12, 2013 [1 favorite]

Even under that definition, our best guess of P(Obama) is the mean of all the trials - I can't imagine any other definition that makes any sense.

Wikipedia:

The Bayesian interpretation of probability can be seen as an extension of logic that enables reasoning with propositions whose truth or falsity is uncertain. [...] Bayesian probability interprets the concept of probability as "an abstract concept, a quantity that we assign theoretically, for the purpose of representing a state of knowledge or that we calculate from previously assigned probabilities,"[2] in contrast to interpreting it as a frequency or "propensity" of some phenomenon.

Stanford Encylopedia of Philosophy:

Subjectivists maintain that beliefs come in varying gradations of strength, and that an ideally rational person's graded beliefs can be represented by a subjective probability function P. For each hypothesis H about which the person has a firm opinion, P(H) measures her level of confidence (or "degree of belief") in H's truth.[6]

And if you look at real-life market bubbles, they can grow for a long time before you start seeing a significant number of "sober" investors taking positions against it. And that's when you can actually see businesses failing or borrowers defaulting. Here, with the outcome of the election unknown, it seems entirely plausible to me that you could get more investors jumping onto the bandwagon for a particular candidate (urging each other to take advantage of those poor saps who don't understand that our candidate is going to win) than investors who are able to digest the polls as accurately as possible and bet against them.

Seems to me one big difference between a futures market in commodities like oil and gold and an election prediction market is the futures market itself is a large factor in the price of the commodity. This could be a big factor in forming bubbles because participants start betting on what the market itself will do, rather than the underlying value of the asset. And value is a subjective concept that is not really a measurable quantity other than in a market, which makes it a sort of an unknown quantity at all times. A betting market should not have any impact in the outcome of a basketball game, in contrast. I don't pay much attention to it, but it doesn't seem that the same sort of huge irrational bubbles happen in sports betting lines as prices in commodities, stocks, houses, etc.

Intrade was better than most of the polls, as I recall, and the other European betting markets better than Intrade. I suspect in the next election the prediction markets will remain even closer to poll aggregators - as they have proven to be so accurate. It will be interesting to see if Silver can continue to beat the prediction markets consistently. In the financial world, most speculators do not, but it would seem that some (Warren Buffet) do.
posted by Golden Eternity at 9:19 AM on March 12, 2013 [1 favorite]

The Bayesian/frequentist thing is actually a total red herring here because Bayes' rule derives from very basic axioms of probability, which are true no matter which "school" you belong to. In other words, using Bayes' rule does not imply a commitment to a Bayesian approach. If you stare at it for a while, you can see just by drawing a Venn diagram that P(A|B) has to be P(A,B)/P(B) and that likewise P(B|A) = P(A,B)/P(A), and then Bayes' rule is basically just a little algebra on top of that. So a frequentist would also come to the same P(Obama) that lambdaphage calculated, s/he would just interpret it as the expected proportion of times that Obama would win if you were able to re-run the 2012 election over and over.
posted by en forme de poire at 11:25 AM on March 12, 2013 [2 favorites]

Though now that I think about it, I guess frequentists wouldn't be making claims about the probability of the model given the data in the first place. But yeah, the way lambdaphage calculated P(O) using Bayes' rule and marginalization is definitely correct.
posted by en forme de poire at 11:38 AM on March 12, 2013 [1 favorite]

If anyone is still unclear about this, I highly suggest sitting down in a quiet room with a cup of coffee and working a few examples until everything becomes clear.

3: there are more then two possible models for predicting the election.

I believe that is true, and you seem to be operating on the basis that it's false. Is that correct?

We are never going to figure out who's right if we can't even specify about what we disagree about.

___

Folks, P(Obama) != P(Obama|Obama). P(Obama) is the prior probability of Obama being elected, assuming a prior over a class of models. This need not be, and generally is not, the true probability of Obama being elected.

Except for the fact that we want to evaluate the models given we now know the true probability of Obama having won the election. - it sounds like you don't actually know how to calculate that.

if we are constraining our model selection problem to those two models

Except, why would we do that? As far as I can tell, it's only something you did because you don't actually know how to deal with more complex systems.

(2) P(O) = P(O|I)P(I) + P(O|F)P(F) by law of total probability.

Again, this would only be true if you assume that Intrade and 538 are the only two possible models. Which as I said, I don't, because there is no reason to do so.

Further, if you conclude that the prior probability of an event "becomes 1" when you observe the event ... The probabilities P(F|O) and P(I|O) no longer even sum to 1

Of course not. Because there are an infinite number of possible models, P(F) and P(I) should always sum to zero. Just like if you have a continuous probability distribution for a random continuous variable Y then P(Y=2) + P(Y=3) = 0. You have to use calculus to calculate things like ∫_₂^₃P(Y=x).

The probability of two events summing to zero is totally consistent with probability theory, in fact it comes up all the time. It's only a problem if you don't actually know calculus.

despite the assumption that F and I are exhaustive.

Except, again, they're not. And there is no reason to assume they are. You assumed that in your calculations, apparently, because that's all you knew how to calculate. And as a result, you completely ignored the actual results of the election

Anyone who accepts that the prior probability P(O) = 1 after one observation should also be willing to bet that a fair coin will always land heads up after observing one heads. If anyone wants to make that bet, let me know.

No, you're simply stating that the coin just landed heads up. That's it. Saying P(Obama) = 1 is simply a statement that Obama is currently the president.

Bayes' theorem may be counter-intuitive, but understanding it deep in your bones is just a life skill and should be considered part of a complete liberal arts education.

If you think Intrade and 538 are the only possible models for predicting presidential elections, and you think the sum of the probability of two events in a probability space must equal 1, then you don't understand probability theory very well.

Finally, if you want to get super-technical when we're talking about the real world, we're actually only dealing with posterior probability. There is no true P(O) there's just P(O | Everything that's ever happened in the history of the universe) - you know a quantum fluxuation a few milliseconds after the big bang caused a star to form about 8,000 light years away that would go supernova in 6428 BC, the light from which would be observed by Tycho Brahe in 1572 yadda yadda yadda Obama is president.

When I say the "probability is changing" I simply mean the 'posterior' probability is changing because more stuff is getting added to the right side of the pipe, it's just a shorthand way of saying "given the current state of the universe" - I did say that if you wanted you could think of P(Obama) = 1 as having always been the case, I never said that prior probability changed, only that you could look at it either way and in either case I was right.

But ultimately "interpretation" of probability theory is just that. there are lots of different ways to interpret the math, but as long as you do the math properly you get the right result. But you also have to pick the right probability space and events if you want to get a useful answer.

But yeah, the way lambdaphage calculated P(O) using Bayes' rule and marginalization is definitely correct. -- en forme de poire

Well, he did the calculation correctly, the second time he tried it anyway. The first time he did it wrong (which he admits). But he assumes only two possible models and doesn't include the actual result of the election in anyway, which is a calculation that is completely irrelevant. And also his math still showed that 538 was better then Intrade.

It still isn't clear at all what he's trying to say about the relative merits of Intrade and 538, because his own whack math shows that 538 is better, and better then by a larger amount then the difference between Intrade and just assuming it's a coin flip.

___

Finally, just to clarify, again, here is something I think is true that I think lambdaphage thinks is false:

3: there are more then two possible models for predicting the election

If we assume that, is no reason why P(Intrade|Obama) + P(538|Obama) isn't equal to zero. In fact I think it is actually the case. Neither one is true what I want to know is which model is produces a better prediction.
posted by delmoi at 12:54 PM on March 12, 2013

delmoi, lambdaphage doesn't think that's actually true - his point is that, taking P(Model|Obama) = P(Obama|Model)P(Model)/P(Obama), if you assume that all models are equally likely a priori, then the prior ratio P(Model)/P(Obama) is going to be the same whether you're talking about model 1 or model 2000. (By marginalization P(Obama) = P(Obama|Model1)P(Model1) + P(Obama|Model2)P(Model2)... etc.)

So the likelihood ratio P(538|Obama)/P(Intrade|Obama) remains constant even if you allow for many more unobserved equally-likely models. And given the single measurement we have, a LR of <2 is not all that great evidence in favor of 538 vs. Intrade. (If you look across the state-wide predictions the evidence might start stacking up, but I haven't looked so I don't know.)
posted by en forme de poire at 1:31 PM on March 12, 2013 [2 favorites]

So the likelihood ratio P(538|Obama)/P(Intrade|Obama) remains constant

Right, but why does this matter? If they're both near zero - what difference does it matter what the ratio is?

What we want to know is which model is more accurate, not which one has a "higher probability of being true". Like, we know for certain that Newtonian physics does not perfectly describe the universe. It breaks down at the quantum and relativistic scales. However it is very close to what we observe at human scale, it's predictions are very close to the observed results. But the probability of Newtonian physics being true is zero.

The "probability" of a model being "true" is A) Irrelevant, and B) almost certainly Zero.

One way to think about it is: Would a gambler make more money using Intrade as a model or 538? So they have a choice of betting on Obama with 95:5 odds or 61:39 odds - it's obvious in retrospect that they would have made more money with the 538 model. Now that we're in the future, we know which one would have been better - and thus we can render a verdict about which one did a better job in predicting the election.

If Romney had won, they would have lost less with the Intrade odds, and so Intrade would have done better, even though they both predicted a Romney defeat.

Now, that doesn't tell us about the future, but does tell us which method of prediction produced a better (as in "more useful") result.

I do think, though if we could hypothetically re-run the election starting from the day before, people would still make more money using the 538 odds then the Intrade odds.
posted by delmoi at 2:38 PM on March 12, 2013

Bayes' theorem may be counter-intuitive, but understanding it deep in your bones is just a life skill and should be considered part of a complete liberal arts education.

I can't speak for other criticisms, but your dismissal of small differences in posterior probabilities as being irrelevant in betting would make you a terrible, or at least very bankrupt casino owner.
posted by Mental Wimp at 3:37 PM on March 12, 2013

Hmm, I asked you to provide a simple statement that you think is true, and that you think I think is false, and you don't seem to have even bothered to try to do that. I can only conclude you've lost interest in a bet. Which is interesting, since you said yourself that not accepting a bet when you know you're right is actually a tax on stupidity or whatever

Forgive me for not being clear enough about this. My most pressing point of disagreement concerns the following claim:

0: P(Obama) = 1

I am still happy to make that bet with you, I just thought it would be rude to continue to hammer that point when we are arguing about a theorem of probability. It's like betting about whether George Washington was the first President. I would rather get everyone on the same page about that in as genial a manner as possible than rub anyone's face in it. But yes, I'd be glad to bet that any trained statistician will come to the same conclusion that en forme de poire did, Bayesian/frequentist controversy nonwithstanding :)

Clearing up statement 0 is just a prerequisite to discussing the relative merits of 538 and Intrade. As for:

1: I think the probability of Obama having won the election is closer to 1 then it is to 0.78

I take it that you're referring to 0.78 = P(Obama) in the context of the model selection problem. In the context of the model selection problem, in which we are considering only Intrade and 538 and computing the posterior distribution over those models, P(Obama) = .78 + O(10^-3) exactly. That is just a matter of arithmetic. If you're asking "what is was true probability of Obama being elected", (whether you think a probability is an objective uncertainty, a subjective credence defined as a willingness to bet, or a frequency over a large number of hypothetical trials), I don't think anyone thinks that the probability was close to 1. (You should have regarded the prediction market as free money with practically no downside risk if that were true). I don't see why you would say that the probability is "close to 1" unless you are conflating the probability of Obama being elected with the probability of Obama being elected, given that he was elected.

2: I think Nate Silver's prediction was "better" then Intrade's by an "amount" greater then the difference between Intrade and a coin flip.

My point remains, because my point is nothing more than Bayes' theorem and some arithmetic. If you are considering only Intrade and 538, are initially indifferent between them, and accept P(O|F) = .95 and P(O|I) = .61, then P(F|O) = .608 < P(O|I) = .61. If you (correctly) do not think P(O|I) is exactly decisive, you should not be impressed by P(F|O) either.

Except for the fact that we want to evaluate the models given we now know the true probability [emph added] of Obama having won the election. - it sounds like you don't actually know how to calculate that.

This is not correct. If you flip a coin once and get "heads", do you know the true bias of the coin? No. Bayes' theorem tells you exactly how to evaluate the models given the evidence, which is not the same thing as the true probability. Go back through this conversation and replace every mention of the election with a coin of unknown bias that has an "O" on one side and an "R" on the other, and perhaps it will be clearer that it is courting conceptual disaster to suppose that you can infer a probability from a single observation of a Bernoulli random variable. The heart of the confusion is that you are confusing the parameters of a random variable (in this case, the bias p of a Bernoulli) with an observation of that r.v.

Except, why would we [constrain our model selection problem to two models]? As far as I can tell, it's only something you did because you don't actually know how to deal with more complex systems...Well, he did the calculation correctly, the second time he tried it anyway. The first time he did it wrong (which he admits)....whack math...whack assumptions

This is getting cringe-inducing. We consider two models because those were the two models we were considering. Do you not recall:

Should it bother us that, in reality, I and F are not the only two possible models? No: if we are constraining our model selection problem to those two models, i.e. we want to know which one is better, then our posterior probabilities P(I|O) and P(F|O) will be off by the same error in the normalization constant, but they will still be directly comparable. Leaving out other hypotheses will not reverse our comparison of I and F.

I do this stuff for a living, dude. The reason I didn't generalize this to a beta prior is that it doesn't further illuminate the comparison, and we're having enough trouble getting some folks through the math as it is. The error in the calculation was a typo that altered the value of P(F|O) by 2.6%. So I'd appreciate it if you would refrain from making irrelevant points I have already anticipated and addressed, or playing gotcha.

The probability of two events summing to zero is totally consistent with probability theory, in fact it comes up all the time. It's only a problem if you don't actually know calculus.

You are missing the point entirely. I have specifically stated in several places that we would consider the example in which I and F exhaust the model class, because it didn't matter. Here is an exercise for you: rederive the posterior assuming a U(0,1) prior and tell me what you get. Does the ratio of the posterior probabilities change?

Saying P(Obama) = 1 is simply a statement that Obama is currently the president.

Ok, I'm serious about this bet now. I felt slightly guilty about it before, but you have helped me to get over that.
posted by lambdaphage at 5:05 PM on March 12, 2013 [1 favorite]

Terms

lambdaphage will submit the following post to stats.stackexchange.com:

------
Please help my friend and me settle a dispute:

Suppose we consider two models I and F which estimate the probability of Obama's re-election in 2012 (event O).

Given that the evidence Obama was, in fact, re-elected, we can compute the posterior distribution over these two models. We have:

P(I|O) = P(O|I)P(I)/P(O), and
P(F|O) = P(O|F)P(F)/P(O).

Now, in the above computation, is it true that P(O) = 1 because Obama did in fact get re-elected?

Or is P(O) = \int_\Theta d\theta P(O|\theta)P(\theta) (which reduces to P(O|F)P(F) + P(O|I)P(I) if we assume that I and F jointly exhaust the model space)?

Bonus question: If I and F are but two models in a continuous parameter space, do we alter the likelihood ratio P(I|O)/P(F|O) by solving the discretized problem instead?
----

If you would like to make any revisions, please do so and repost the entire thing. I will do the same until we have reached mutual agreement.

If the consensus (say, as judged by the most upvoted comment 24 hrs after posting) is that it is true that P(O) = 1, I will pay $100 dollars to a charity of your choice. If the consensus is that it is false, you will pay $100 to Against Malaria. (If you worry that Against Malaria is a bad choice, I am open to suggestions.) A screencap of the receipt should be enough verification, I think.

Sounds good?
posted by lambdaphage at 5:28 PM on March 12, 2013

Thank you for your patient and well-written comments in this thread, lambdaphage. I think maybe people are trying to use heuristics to argue with you without realizing this is a well-understood problem.
posted by no regrets, coyote at 5:34 PM on March 12, 2013 [4 favorites]

your dismissal of small differences in posterior probabilities as being irrelevant in betting would make you a terrible, or at least very bankrupt casino owner.

I think your criticism is actually directed at delmoi, who suggested that a probability of 0.61 is "basically a coin flip". I only applied that rule of thumb, not endorsed it.
posted by lambdaphage at 5:35 PM on March 12, 2013

A bet about MeFi comments about betting... ow, we're getting too meta again
posted by en forme de poire at 8:15 PM on March 12, 2013

We don't know what the probability of Obama being elected was before the election: we only know that he was elected. ...Even under that definition, our best guess of P(Obama) is the mean of all the trials - I can't imagine any other definition that makes any sense.

Even frequentists still say things like "there is a 30% chance of rain tomorrow," though, even though strictly speaking there's only one trial of whether it rains on March 13th, 2013. That day, and perhaps that exact configuration of weather, will only ever happen once. But we have data about what the weather looks like today and a model for predicting the future, so we can make some statements about probability. You can make a similar analogy to, e.g., polling data.

Incidentally, weather lulz aside, according to Nate Silver's new book, those percentages are apparently really well calibrated.
posted by en forme de poire at 8:40 PM on March 12, 2013 [1 favorite]

I think your criticism is actually directed at delmoi, who suggested that a probability of 0.61 is "basically a coin flip". I only applied that rule of thumb, not endorsed it.

Uh, no I said Intrade (@61%) was basically a coin flip compared to 538 (@95%). And even if we accept your calculation, the distance between 538 and Intrade is still greater then the distance between Intrade and a coin flip (meaning P(O|I) and 0.5).

And in fact, If we go all the way back and use your method to compare Intrade to a coin flip, we get P(I|O) = (0.61*0.5)/0.555 = 0.5495 and P(C|O) = (0.5*0.5)/0.555 = 0.4504) For some reason, you don't want to acknowledge this at all.

Compared to P(538|Obama) = (0.95 * 0.5) / 0.78 = 0.60897 And P(Intrade|Obama) (0.61*0.5)/0.78 = 0.39102 when we compare intrade and 538 using your method.
posted by delmoi at 5:03 AM on March 13, 2013

0: P(Obama) = 1

I don't specifically mean P(Obama) = 1 exactly, I believe it is very close to one. In fact, I wrote Specifically P(Obama) = 1 - ε, for some infinitesimal number ε there is some very small difference from one, of course, and I've basically been using 1 as shorthand, since it doesn't make any difference.

Now, a quick defense that P(Obama) 1- ε.

An election is not actually a random event. Coin flips aren't random either, if you know the, position, velocity, and spin a coin will have you can predict whether it will land heads or tails exactly. We just use "coin flip" as a shorthand to describe something with an equal chance of being true or false.

Similarly, if we knew the intentions of every voter, and whether or not they would show up to the polls, we would know who would win the election, just as we would know whether a coin would land face up if we knew it's spin/velocity/position.

If we spin flip a coin at 15 rpm, [0cm/s,1cm/s,10cm/s], at point [1m,1m,1m] using a precise mechanical coin flipper, and it lands on it's head, then we do know with a very high probability that if we flip the same coin with the same parameters it will land on it's head again.

Similarly if we were somehow able to re-run the election before anyone changed their minds, we'd get very close to the same result. (so for example, if you had voters fill out two ballots the same way, once you knew the winner from the first ballot, you would know with a very high degree of certainty who would win using the second ballots)

___
Now, lambdaphage, when it comes to the math I'm still somewhat confused about what you believe about here: Are you interested in the 'raw' values of P(I|O) and P(F|O) or the ratio? Obviously if we are looking at the RAW probability value, then P(I), P(F), P(I|O), and P(F|O) should all be zero.

It sounds like you just want to look at the ratio (as the number of models approaches infinity) But if that's the case, then P(O) drops out of the equation - it doesn't matter what the value is at all (so long as it's not zero, I guess) - and I don't see how you can have a real argument about the value of a term that drops out. That makes no sense.

If you are considering only Intrade and 538, are initially indifferent between them, and accept P(O|F) = .95 and P(O|I) = .61, then P(F|O) = .608 < P(O|I) = .61. If you (correctly) do not think P(O|I) is exactly decisive, you should not be impressed by P(F|O) either.

Well, you're completely ignoring P(I|O) - I calculated the values for myself here, and got:

P(538|Obama) = (0.95 * 0.5) / 0.78 = 0.60897 And P(Intrade|Obama) (0.61*0.5)/0.78 = 0.39102

That's a huge difference. And as I said in my last comment, it's a bigger difference then when you compare Intrade to a coin flip using the same method: P(I|O) = (0.61*0.5)/0.555 = 0.5495 and P(C|O) = (0.5*0.5)/0.555 = 0.4504

Now you seem to switching back to caring about the "raw" probability, instead of the ratio. I think you've confused yourself by not including the normalization constant. The probability of P(538|O) is actually 0.6087/k where k is the normalization constant, which is infinity. Of course P(538|O) is less then P(O|Intrade), because P(538|O) is zero

You could also do a 3 way comparison using Intrade, 538 and a coin flip as models. You get:

(0.95 * ⅓) / 0.71 = 0.4460, P(Intrade|Obama) (0.61*⅓)/0.71 = 0.28638, and P(Coinflip|Obama) = (0.5 * ⅓) / 0.71 = 0.23471

Now you'll notice the ratio stays the same: 0.4460/0.28638 = 0.60897/0.39102 = 1.5573

See what I mean? The ratio stays the same when you add more models, and you can tell by the ratio that 538 is 55% "better" then Intrade, while Intrade is only 22% "better" then a coin flip.

But if here you are saying we should look at the raw probability that comes out when you don't include the normalization constant.

___
There are a couple of problems with your question:

Now, in the above computation, is it true that P(O) = 1 because Obama did in fact get re-elected?

I thought we were arguing about what P(O) was in the "real world" not in "the above computation" - as I said, the actual value of P(O) is irrelevant in the computation since the ratio stays the same, and the difference between using 0.78 and 1 isn't enough to make Intrade closer to 538 then Intrade is to a coinflip anyway. You can pick whatever you want within reason, 538 is still better.

(which reduces to P(O|F)P(F) + P(O|I)P(I) if we assume that I and F jointly exhaust the model space)

Except we have no reason to make that assumption, other then that it makes the math easier for you. (We certainly could do it that way, but then you have to ignore the raw outputs and look only at the ratio. Which you're not doing.)

___

If the consensus (say, as judged by the most upvoted comment 24 hrs after posting) is that it is true that P(O) = 1, I will pay $100 dollars to a charity of your choice. If the consensus is that it is false, you will pay $100 to Against Malaria.

I also said we could do a real bet, not a charity bet :). We could use bitcoin, we'd just have to post our bitcoin addresses, and the loser could post a transaction ID (bitcoin transactions are public).

You can create an online wallet at blockchain.info but actually acquiring btc if you don't have any is kind of a pain, so if you don't want to bother no big deal (but also no bet).
___
But anyway, this is starting to take forever, and I kind of doubt we'll agree on the specific wording of a question. One way to solve it would be to have each of us write a brief overview of our views and see who is 'more right'.

In any event, here's a specific question that covers most of it:

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
Suppose we have two election prediction models I, and F and we want to determine which one was better given the final result of the election (Which we'll call event O - for Obama winning the election)

1A: Is there any value in trying to compute the posterior probability of the models being "true" using the formula:

P(I|O) = P(O|I)P(I)/P(O), and
P(F|O) = P(O|F)P(F)/P(O).
With P(I) = P(F) = 0.5, P(O|F) and P(F|P(O) = (P(O|F) + P(O|I))/2

Or is it the case that P(I) and P(F) are both zero because there are an infinite number of possible models, we know they are just approximations anyway, and if we just want the ratio P(F|O)/P(I|O) the term P(O) drops out making the actual result of the election irrelevant.

1B: Would it make more sense to look at something like the information entropy using a formula like

H_b(I) = -p_ilog₂ p_i - (1-p_i)log_2(1- p_i)
H_b(F) = -p_flog₂ p_f - (1-p_f) log_2(1- p_f),
This time using p_i and p_f as the outputs of the model, and the model with the lowest entropy being better, if the prediction was correct and worse if the prediction was wrong.

2: If the output of model F is 0.95, and the output of model I is 0.61, then does the first method actually show that F is better then I better by more then the amount which I is better then a coin flip?

3: In a philosophical sense, is it reasonable to say that P(O) is very close to 1 'in general', given the fact that Obama did in fact win the election, and if we were somehow able to re-run the election given all the same voter preferences, we would almost certainly get the same result?

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
I mentioned the entropy as a better measure up here

Is there anything you disagree with about the above questions? I'd be happy to go with just question three alone, because that's what I thought the central question about P(Obama) was about.

I do this stuff for a living, dude.

That doesn't mean you're any good at it.
posted by delmoi at 6:03 AM on March 13, 2013

I was actually about to post that whether you use Bayes' rule to compute the posteriors (i.e., P(Model|Obama)) or just stick with comparing the likelihoods directly (i.e., P(Obama|Model)), the odds ratio remains the same, since we're assuming the priors of each model are equal. 0.61/0.39 = 0.95/0.61 ≈ 1.6:1. So you're right about that. The problem is that this kind of score is generally considered to be weak evidence for one hypothesis over another. To really make a confident inference about which model is best from a single prediction, one of the predictors would really need to botch it (e.g. 0.9 vs 0.1).

Again, if you had a whole set of observations for Intrade vs. 538 (like all 50 states) you might be able to say something stronger since the probabilities for each instance would get multiplied together (well, assuming independence, which, who knows), but I'm already late for work so I'm not going to check myself at least not right now.
posted by en forme de poire at 6:12 AM on March 13, 2013

OK I lied. It looks like Intrade got the rankings correct but the estimated probabilities tend to skew Romney vs. Nate's (e.g., FL is 0.35 by Intrade but 0.53 by 538). Would still need the full dataset but so far it looks like you could probably make a case for poll aggregation being a better estimator. Again, that's assuming independence, which is almost certainly wrong for Intrade if not for 538 - I haven't looked at how his predictions were made in a while - and I don't have data for all 50 states.
posted by en forme de poire at 6:28 AM on March 13, 2013

This is becoming wearisome. You are continuing to make statements that are incorrect, and at an apparently increasing rate. I despair of correcting them all, given the track record.

Let me try to condense your question a bit:

---
Suppose we have two election prediction models I, and F and we want to determine which one was better given the final result of the election (Which we'll call event O - for Obama winning the election)

The posterior probability of the models being "true" is given by Bayes' theorem:

P(I|O) = P(O|I)P(I)/P(O), and
P(F|O) = P(O|F)P(F)/P(O).

Assume we are initially indifferent between the two models, i.e. P(I) = P(F).

Is it reasonable to say that P(O) is very close to 1, given the fact that Obama did in fact win the election, and if we were somehow able to re-run the election given all the same voter preferences, we would almost certainly get the same result?
---

I do not think I have made any substantive changes to the sense of the question.

If you're on board with that, I'll set up bitcoin.
posted by lambdaphage at 7:42 PM on March 14, 2013

*crickets*
posted by unSane at 1:43 PM on March 15, 2013

lambdaphage: You've once again assumed your answer in your question:

The posterior probability of the models being "true" is given by Bayes' theorem:

The problem is, I do not agree the posterior probability of the models being true is equal to that formula. I believe the probabilities of the models being true is zero, because there is an infinite model space, and because obviously anyone with a brain can see they are clearly not actually true.

The question about the probability of Obama having won the election is separate from the formula used to evaluate models.

Why not simply ask: "What is the probability that Obama won the 2012 election for president?"

I do not think I have made any substantive changes to the sense of the question.

If you haven't made any substantive changes, then you should be fine posting my version - which, as you say, has no substantive differences.

So go ahead and post my version.
posted by delmoi at 11:17 PM on March 15, 2013

And by the way - if it was really the case that there were two separate random processes, one with a 61% chance of picking Obama, and one with a 95% of picking Obama,

And the election were decided by randomly picking between them, and then going with the output, then of course we could say that the probability of Obama winning would be 78%, and we could use that formula to figure out the probability of those models having been chosen randomly if Obama won.

But, critically that's not what happened - There actually is one "true" model, which is the preferences of each voter in each state, whether or not they turned out to vote and were able to cast their ballots, and how many electoral college votes their state had. We can call that model E (for election) then we know P(E) = 1, P(O|E) = 1-ε, and so P(O) = 1-ε.

In other words, election prediction models are trying to estimate voter preference and turnout, and the election itself is the true measure (minus the fact that something might go wrong i.e. hanging chads in Florida, 2000)

We are not trying to determine between two "true" models, we are trying to determine which of two models, out of an infinite set, more closely approximate the real true model.

That doesn't mean you can't use that technique for evaluating models, however, if you do it clearly shows that 538 is better then Intrade, by more then the amount by which Intrade is better then a 50% guess.

The problem is you're taking the final number, 0.60897, and acting like it's the actual probability of 538, being the "true" model in reality. But again, that's not the case at all. That's only true if you make the assumption that those really are the only two models, and that one of them really is true. But that's just an assumption you made to make the math easier for yourself.

And again. If you don't have any substantive changes to my version of the question, post it as is.
posted by delmoi at 12:01 AM on March 16, 2013

Okay, still I think it's pretty unlikely we'll come to an agreement on the wording of a question, but this problem is still kind of stuck in my head, annoyingly.

I was kind of annoyed when I replied to your last question, because I thought you were just re-stating your position again, thus the snarky "just post my question if you don't disagree with it"

Rereading I noticed you dropped the requirement that P(I) = 0.5 and P(F) = 0.5, which actually a lot more compatible with what I've been saying.

The final difference though is that I think the actual quantities P(I|O) and P(F|O) are meaningless. Instead, it's the ratio that matters (i.e. the Bayes factor)

In order to compute it given P(I) = P(F) = (1/N) , where N is the number of models. The equation simplifies to:

P(I|O) / P(F|O) = (P(O|I)P(I)/P(O)) / (P(O|F)P(F)/P(O))
P(I|O) / P(F|O) = (P(O|I)(1/N)) / (P(O|F)(1/N))
P(I|O) / P(F|O) = P(O|I) / P(O|F)

The actual numerical result is the same, But the key difference is that in my view P(O) = 1 - ε and P(I|O) and P(F|O) are meangless on their own, outside of an equation,

while you seem, from what I can tell to think P(I|O) + P(F|O) = 1, that they're equal 0.61/(0.95+0.61) ~= 0.39102 and 0.95/(0.95+0.61) ~= 0.60897, respectively, and P(O) is meaningless(?) or something(?) on it's own, outside of an equation.

Is that an accurate representation of what you think?
posted by delmoi at 2:48 PM on March 18, 2013

Nevada bill would allow wagering on U.S. presidential election

Betting on presidential elections? Nevada state senator says its time has come
posted by Golden Eternity at 8:14 AM on March 26, 2013

Intrade has updated their statement:

We have now concluded the initial stages of our investigations about the financial status of the Company, and it appears that the Company is in a cash “shortfall” position of approximately US $700,000 when comparing all cash on hand in Company and Member bank accounts with Member account balances on the Exchange system.
...
We are now very confident about the reasons which caused the current circumstance of the Company; however, for legal reasons we are not yet at liberty to document them to you. I can confirm that the Company, if it is able, intends to vigorously pursue two substantial monetary claims against two distinct parties for an aggregate amount greater than $3,500,000.

It's beginning to look like my original guess for what was going on may well turn out to be true.
posted by RichardP at 9:01 PM on April 7, 2013

« Older Come! We must find the pin head! | Path of Exile Newer »

This thread has been archived and is closed to new comments

MetaFilter

no one could have predicted
March 10, 2013 6:08 PM Subscribe

Tags

Share

no one could have predicted March 10, 2013 6:08 PM Subscribe

Tags

Share

no one could have predicted
March 10, 2013 6:08 PM Subscribe