De Moivre’s equation

March 16, 2013 6:34 AM

The Most Dangerous Equation If you have never taken the time to read any statistics, then take the time to read this short chapter excerpt. In order to understand the world a bit more accurately and be able to interpret data for public policy, understanding De Moivre’s equation and sample variability is very significant.
posted by SollosQ (24 comments total)

This is a great piece! I'm writing a general-audience math book right now and adopted a couple of examples from this chapter into one of my own.
posted by escabeche at 6:55 AM on March 16, 2013

This is an excellent article. I'm pretty confident that not having a basic understanding of statistics and algorithms are going to be the illiteracies of the 21st century.
posted by mhoye at 6:57 AM on March 16, 2013 [5 favorites]

On my read, the first example (minting coins) actually does have to do with de moive's equation, but all the other examples are just instances of smaller sample sizes having larger variation, not of relating the variance of subsample averages to the variance of the whole.
posted by So You're Saying These Are Pants? at 6:58 AM on March 16, 2013 [4 favorites]

Shouldn't he have taken all the famous equations, from Ohm's law to Euler's identity, developed a danger metric and then found the variability in the danger estimates?
posted by sammyo at 6:58 AM on March 16, 2013 [5 favorites]

Every single map of a variable across the US you have ever seen is wrong.
posted by miyabo at 8:10 AM on March 16, 2013

Excerpt taken from this book.
posted by DarkForest at 8:34 AM on March 16, 2013

but all the other examples are just instances of smaller sample sizes having larger variation, not of relating the variance of subsample averages to the variance of the whole.

De Moivre's equation is generally described as one particular relationship between sampling size and variance, and the aside relating it to subsamples is just an illustration. But as I'm typing this now, I realize that you've hit on an important point that people like me, those who do stats, take completely for granted. I think you're pointing out that in the other examples, all of the relevant objects are measured.

So why should de Moivre's theorem apply at all, when there is no random selection? I think the answer is that there is great randomness when selecting a time period, or in the measurement itself. Take the cancer example. It's a definitive yes/no answer, sure, but cancer itself is a highly random event due to random muntations occurring in just the right genes. There's also randomness in when the cancer is discovered, and how we'll the records are kept and transferred to a central location for collation.

Or how about test scores for an entire school? There's no subsamples, it's the whole thing so how could De Moivre's theorem apply? Because there's randomness in the measurement itself. Students have good and bad days, days where they care, and there's always guessing going on.

But if you're a Bayesian statistician, you're probably shaking your head and saying de Moivre's theorem is all wrong, the relationship between sample size and variance is actually quite different in all these cases. And it's true, de Moivre's equation only applies here because the Central Lkmit Theorem says that you can approximate the mean of all sorts of different distributions with a Gaussian if the sample size is big enough, which is what means that de Moivre's theorem is approximately correct.

So I guess two points: nearly all aspects of modern life are better understood through probabilistic analysis than deterministic analysis, and de Moivre's type of analysis only works well with large sample sizes.
posted by Llama-Lime at 8:54 AM on March 16, 2013 [5 favorites]

As this is your first post and as I have been told this in my math posts before, it is convention to put some indication that the link leads to a PDF. Not a problem, just a courtesy.

Otherwise, I shall begin reading! Thanks!
posted by JoeXIII007 at 9:39 AM on March 16, 2013 [2 favorites]

Can any biologists or education specialists comment on the plausibility of his male/female variability argument? There is so much bad science on this stuff out there that my immediate reaction is total skepticism. And he cites Nicholas Wade, which is generally a very bad sign on gender/evolution questions.
posted by col_pogo at 9:46 AM on March 16, 2013 [1 favorite]

@col_pogo, regarding the male/female variability argument I see no reason to dismiss out of hand the cited study as "bad science." Two thoughts:

1. The author of this book chapter actually cites a summary, written by Nicholas Wade of the NYT, of work done separately by Eric Vilian and Arthur Arnold. So I'm not really sure what your beef is with Mr. Wade's characterization of the science.

2. In regards to the science, here is a review article by Arthur Arnold and others on sex differences in the brain.
posted by scalespace at 11:06 AM on March 16, 2013 [1 favorite]

but all the other examples are just instances of smaller sample sizes having larger variation, not of relating the variance of subsample averages to the variance of the whole.

Well, his point is that if they had been aware of that formula, they wouldn't have made those mistakes.
posted by delmoi at 12:05 PM on March 16, 2013

His evo-psych bullshit sounds like typical evo-psych bullshit.

It is well established that there is evolutionary pressure toward greater variation within species—within the constraints of genetic stability. his is evidenced by the dominance of sexual over asexual re­production among mammals. But this leaves us with a puzzle. Why was our genetic structure built to yield greater variation among males than females? And not just among humans, but virtually all mammals. he pattern of mating suggests an answer. In most mammalian spe­ cies that reproduce sexually, essentially all adult females reproduce, whereas only a small proportion of males do so (modern humans excepted). think of the alpha­male lion surrounded by a pride of fe­males, with lesser males wandering aimlessly and alone in the forest roaring in frustration. One way to increase the likelihood of of­spring being selected to reproduce is to have large variance among them. Thus evolutionary pressure would reward larger variation for males relative to females.

The problem with here is that he's just imagining things and assuming that because he was able to imagine something is strong evidence that it's true. A lot of evo-psych arguments work that way. Imagining humans in some kind of "state of nature" or whatever.

You could easily come up with a "Just so" story to justify the reverse thing happening. For example

Imagine an alpha male looking to spread his seed as far as possible and have the healthiest babies in all circumstances. He is going to want his offspring to be genetically diverse, and thus will seek out sex partners as different from each-other as possible.

Therefore, it would be evolutionarily advantageous for women to have greater diversity of traits, which clearly explains why they do!

You can just make up whatever goofball story you want to fit whatever data you want.
posted by delmoi at 12:26 PM on March 16, 2013 [9 favorites]

I thought he was referring to this De Moivre’s equation, and was trying my damnedest to figure out how that could be dangerous.
posted by benito.strauss at 12:34 PM on March 16, 2013 [1 favorite]

Also, this part is suprisingly innane:

Thus,from De Moivre’s equa­tion, we would expect, ceteris paribus, about 40% more variability† among males than females. he fact that we see less than 10% greater variation in NAEP demands the existence of a deeper explanation.

That would only be the case if all genes that affect intelligence differences were on the X Chromosome. If only some of them were on the x-chromosome then some would show more variability in guys and some of them would not.

The simple explanation is simple: Some genes affecting intelligence might be on the X chromosome, some are not, and more importantly specific genes actually interact with other genes on other chromosomes, and are not fully independent things with a 1:1 correlation between a gene's presence and a measurable allele.

I'm not sure what can explain that passage other then that the author somehow forgot that there were 22 other genes besides X and Y that might also have an impact on the brain.

What other explanation is there for him having written that? why would we expect 40% variability other then all relevant genes being on the X chromosome?

___

The other problem I see with this argument is that it assumes that chromosomes are just averaged together or something - in reality you have dominant and recessive traits.

So for example, suppose there were trait Z on the X chromosome that boosted math ability. And let's say it's on about 50% of X chromosomes. 50% of men would get a boost on math tests, and 50% would not. In that case, the existence of the gene increases variance among men. But what does it do to women?

Well, about 25% of women would have two copies 50% would have one, and 25% would have no copies.

Now, if it were the case that having one of two genes boosted your intelligence half as much, then you would see less variation among women then men. Only 1/4th would get the full boost, and only 1/4th would get no boost. Most would get half boost, and the average would be the same as with the guys.

But that's not how genetics works. If the trait were dominant then actually you would see the 75% of women with at least one copy get a boost, and women would actually be better at math tests then men, on average.

If the trait were recessive, then only 25% of women would get the boost, and 75% would do worse then the average man on a math test.

Now, if you had a mix of traits, some of which increased test taking ability, and some of which decreased them then that might explain why you see more variance with men then with women. But again, you certainly wouldn't expect all of those genes to be on the X chromosome.
posted by delmoi at 1:07 PM on March 16, 2013 [1 favorite]

But that's not how genetics works. If the trait were dominant then actually you would see the 75% of women with at least one copy get a boost, and women would actually be better at math tests then men, on average.

There is more to genetics than the simple complete dominance they teach in middle school. Let's not use sloppy science to fight (presumed) sloppy science.
posted by Pyry at 1:39 PM on March 16, 2013 [1 favorite]

I got the feeling that he wanted to have 5 examples (do writing schools teach a Rule of Fives that complements the Rule of Threes?). It would have been a much better article if he had left out the genetics stuff.

But I love the small schools stuff, because I made illustrating it an extra credit assignment once. I got to see a bunch of students click in to the "A-ha, but there are also more small schools in the under-performing group". Very satisfying.
posted by benito.strauss at 1:44 PM on March 16, 2013

Interesting article, but poorly written.
posted by mikeand1 at 2:04 PM on March 16, 2013

The problem with here is that he's just imagining things and assuming that because he was able to imagine something is strong evidence that it's true. A lot of evo-psych arguments work that way. Imagining humans in some kind of "state of nature" or whatever.

I think you are misreading this entirely. There are all kinds of reproductive strategies in the natural world: cloning, spawning, sexual selection of mates, harem behavior, serial monogamy, monogamy etc. If you look at the full diversity of ways things reproduce it's apparent that mammals occupy a small subset and we are all fairly similar to each other, compared to other taxa. We assume, via evolutionary theory, that these behaviors, traits and the underlying genetics have arisen from a common ancestor and changed over time. So it's totally valid to think of reasons for expressions of the mechanisms of mate selection in ancestral species in the same way you can say that the appendix used to do something but now we just have it. We may have something like greater expressed variability in the males because, at at some point, it conferred an advantage. Similarly, in many mammal species only a subset of adults reproduce and in many of those species a higher proportion of females will reproduce each cycle and will have a longer reproductive life. That's pretty well established in the zoology literature and so again, it's valid to look at. None of this has nothing at all to do with psychology. He is talking about larger mammalian traits.

Also he says right in your quote that modern humans do not show this trait of having less males reproduce than females.
posted by fshgrl at 2:38 PM on March 16, 2013 [1 favorite]

benito.strauss, don't feel bad. I thought we were going to talk about De Morgan's Law.
posted by jepler at 4:07 PM on March 16, 2013 [1 favorite]

None of this has nothing at all to do with psychology.

Well, he started out talking about intelligence, and the study of intelligence is an aspect of psychology.

Part of the problem with what he's saying is that there are lots of possible explanations - he starts of with suggestion that the increased variability might come from men having fewer copies of the genes - that certainly fits with the thesis of the rest of his article.

But then he says that's not the case because the variability isn't 40% and then he proposes this crazy theory. But that doesn't make any sense at all, unless you assume that it can't be the case that some of the genes are on the X chromosome and some are on other chromosomes.

And that's why I was asking if there is any reason for him to say this:

Thus,from De Moivre’s equation, we would expect, ceteris paribus, about 40% more variability† among males than females. he fact that we see less than 10% greater variation in NAEP demands the existence of a deeper explanation.

Other then that he thinks that either ALL of the genes that affect NEAP are on the X chromosomes or NONE are. If there is some other explanation for that sentence, what is it?

___

The other problem with a lot of evo-psych stuff is that people tend to look at traits they can actually see, and assume those specific things must have been the cause of the selective pressure. But it's possible they're a side effect of something else - like maybe some gene evolved to fight off some penis-eating fungus that still persists today and increases variability in test taking ability.

Or it could be that something else that differs between men and women is causing the variability. Take competitiveness for example, It could be that in some cases men channel that competitiveness into competing with their peers to try to do as well in school as possible -- whereas in other cases that gets channeled into rebelling disrespecting authority and not caring about school.

So in that case, a single thing: having more competitiveness might result interact with the environment and produce more variability in some other measure. You might actually be able to test this theory by seeing if competitiveness among men and women is actually correlated with higher variance in math skills. The difference in competitiveness could also be cultural rather then biological.

The Chromosome "sample size" theory could also be tested for, we could look at the variance among women who have an XY karyotype and males who have an XX karyotype due to mutations relating to the SRY gene. (It would be pretty difficult to find a large enough sample size, but it's at least in theory testable - although according to scalespace's link they now have GMO mice you can get where sex type has been split off from karyotype, so that might be another way to test)

We may have something like greater expressed variability in the males because, at at some point, it conferred an advantage.

Well, we might have it for any reason at all. If there's no way to actually test it, it's not science. It's just historical fiction.
posted by delmoi at 6:00 PM on March 16, 2013

Speaking of probability, what is the probability that, out of the universe of examples of De Moivre's equation, 2 of 5 are generic geographic region sample-size errors, and the other three are 1) touting the gold standard,* 2) debunking small schools,** and 3) explaining the biological basis for more high-IQ men?***

* In 1150, a century after the Battle of Hastings, it was recognized that the king could not just print money and assign to it any value he chose. Instead the coinage’s value must be intrinsic, based on the amount of precious materials in its makeup.

** The small schools movement seems to have arrived at one of its rec­ommendations through the examination of only one tail of the performance distribution....Expending more than a billion dollars on a theory based on ignorance of De Moivre’s equation suggests just how dangerous that ignorance can be.
      No, the hundreds of policy analysts and social scientists working in the small-schools movement over the last few decades are really not that stupid. See here for a recent debate, where nearly all of them seem to agree that there are sizes that are too large, and these sizes are quite common. No one making this claim is ignorant of standard errors.

*** Over the last few decades, however, most enlightened investigators have seen that it is not necessarily a difference in level but a difference in variance that separates the sexes....But it suggests that for some skills between 10% and 25% of the increased variability is likely to have had its genesis on the X chromosome. This observation would be invisible to those, even those with Nobel prizes for work in genetics, who are in ignorance of De Moivre’s equation.
      This one has been well-debated in this thread already. But note the faux (or perhaps real) naiveté, as if this variability hypothesis were totally new and had been overlooked by all those utterly ignorant of the idea of variation. Even a simple wikipedia article shows that this hypothesis has long existed on the right. And as far as I can see, there is no animal studies or actual genetic evidence to back it up, merely just-so evolutionary or chromosomal stories that are no different from those told to explain the also-empirically-true fact that men score on average higher on many tests. In both cases, we know the scores are higher and the variance is higher; the controversial stuff, which has nothing to do with De Moivre, is all in the explanation. And perhaps the right may turn out to be correct on this one. That still doesn't explain the marvelous coincidence of these three examples.


posted by chortly at 12:01 AM on March 17, 2013 [4 favorites]

Oh here's an example of how selective pressures can causes changes in various aspects of psychology without being selected for directly.

Early homo sapiens actually had larger brains then modern humans, and since apes with larger brains tend to be smarter then apes with smaller brains, it stands to reason that if you took a pre-historic large-brained human, raised them in our society with access to the kind of education available today, they would probably be smarter.

The way evo-psych people tend to think, they would make up some just-so story about how more intelligent people have fewer children - so there is a selective pressure against intelligence, or some other nonsense.

But in reality, most people think it simply had to do with the fact that smaller heads on infants means less risk in childbirth. So the selective pressure was really just for physically smaller heads, not less intelligence
posted by delmoi at 7:15 AM on March 17, 2013

Why Blacks are like boys and whites are like girls. (Warning: Malcolm Gladwell).

The Gaussian data assumption leads to the greatest Cramer-Rao Bound.
posted by 0rison at 9:53 PM on March 17, 2013

delmoi: "The problem with here is that he's just imagining things and assuming that because he was able to imagine something is strong evidence that it's true. A lot of evo-psych arguments work that way. Imagining humans in some kind of "state of nature" or whatever.

You could easily come up with a "Just so" story to justify the reverse thing happening."

Yes. Many bird species have polygynous mating patterns, akin to the author's "alpha-male lion surrounded by a pride of females". But birds have ZW sex-determination, where males are ZZ and females are ZW, the opposite of mammals. According to the author's theory, this would give females higher variation, and would be counter-productive—somehow birds seem to survive just fine, though.
posted by vasi at 11:54 PM on March 17, 2013 [3 favorites]