P-hacking, or inflation bias, is the selective inclusion of experimental results that suggest statistical significance, as well as the selective exclusion of results which argue against the hypothesis. Skewing work towards positive results helps investigators publish in high-profile journals, which in turn improves access to funding. In a recent PLOS publication, Michael Jennions and Megan Head use text-mining and meta-analyses to determine the extent to which this influences a broad array of published research, offering recommendations on how to reduce this practice.
"Do whatever it takes to not fool yourself, period, that's the scientific method" - Neil deGrasse Tyson. What if we can't do that?
How to tell correlation from causation - "The basic intuition behind the method demonstrated by Prof. Joris Mooij of the University of Amsterdam and his co-authors is surprisingly simple: if one event influences another, then the random noise in the causing event will be reflected in the affected event."
In a Multiverse, What Are the Odds? "Testing the multiverse hypothesis requires measuring whether our universe is statistically typical among the infinite variety of universes. But infinity does a number on statistics." (previously) [more inside]
Hyperreal numbers: infinities and infinitesimals - "In 1976, Jerome Keisler, a student of the famous logician Tarski, published this elementary textbook that teaches calculus using hyperreal numbers. Now it's free, with a Creative Commons copyright!" (pdf—25mb :) [more inside]
Network Theory Overview - "The idea: nature and the world of human technology are full of networks! People like to draw diagrams of networks. Mathematical physicists know that in principle these diagrams can be understood using category theory. But why should physicists have all the fun? This is the century of understanding living systems and adapting to life on a finite planet. Math isn't the main thing we need, but it's got to be part of the solution... so one thing we should do is develop a unified and powerful theory of networks." (via ;)
P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume.
How to think about "Science Studies Prove My Position", for politicians and all non-scientists. Any collation of measures (the effectiveness of a given school, say) will show variability owing to differences in innate ability (teacher competence), plus sampling (children might by chance be an atypical sample with complications), plus bias (the school might be in an area where people are unusually unhealthy), plus measurement error (outcomes might be measured in different ways for different schools). However, the resulting variation is typically interpreted only as differences in innate ability, ignoring the other sources. This becomes problematic with statements describing an extreme outcome ('the pass rate doubled') or comparing the magnitude of the extreme with the mean ('the pass rate in school x is three times the national average') or the range ('there is an x-fold difference between the highest- and lowest-performing schools'). League tables, in particular, are rarely reliable summaries of performance.
...to leave a smile on your face, by Helder Guimarães: Individual vs Crowd | Chaos | Freedom | Trick [more inside]
Statistics Done Wrong is a guide to the most popular statistical errors and slip-ups committed by scientists every day, in the lab and in peer-reviewed journals. Statistics Done Wrong assumes no prior knowledge of statistics, so you can read it before your first statistics course or after thirty years of scientific practice.
Is Psychometric g a Myth? - "As an online discussion about IQ or general intelligence grows longer, the probability of someone linking to statistician Cosma Shalizi's essay g, a Statistical Myth approaches 1. Usually the link is accompanied by an assertion to the effect that Shalizi offers a definitive refutation of the concept of general mental ability, or psychometric g." [more inside]
"Brain training games don't actually make you smarter." Looking at recent meta-analyses and replication attempts of studies showing increased cognitive abilities gained from brain-training games, the New Yorker article comes to the conclusion that the results are suspect and these games haven't been shown to improve cognitive abilities broadly. Currently, brain training is a multi-million-dollar business.
With a database of over 5,000 scientists, from Nobel prize winners to postdocs and PhD students, Sense About Science works in partnership with scientific bodies, research publishers, policy makers, the public and the media, to change public discussions about science and evidence. They make these scientists available for questions from civic organizations and the public looking for scientific advice from experts, campaign for the promotion of scientific principles in public policy, and publish neat guides to understanding science intended for laypeople. [more inside]
The practice of lying to one's children to encourage behavioral compliance was investigated among parents in the US (N = 114) and China (N = 85). The vast majority of parents (84% in the US and 98% in China) reported having lied to their children for this purpose. Within each country, the practice most frequently took the form of falsely threatening to leave a child alone in public if he or she refused to follow the parent. Crosscultural differences were seen: A larger proportion of the parents in China reported that they employed instrumental lie-telling to promote behavioral compliance, and a larger proportion approved of this practice, as compared to the parents in the US. This difference was not seen on measures relating to the practice of lying to promote positive feelings, or on measures relating to statements about fantasy characters such as the tooth fairy. Findings are discussed with reference to sociocultural values and certain parenting-related challenges that extend across cultures. [HTML] -- [PDF] [more inside]
291 diseases and injuries + 67 risk factors + 1,160 non-fatal complications = 650 million estimates of how we age, sicken, and die
As humans live longer, what ails us isn't necessarily what kills us: five data visualizations of how we age, sicken, and die. Causes of death by age, sex, region, and year. Heat map of leading causes and risks by region. Changes in leading causes and risks between 1990 and 2010. Healthy years lost to disability vs. life expectancy in 1990 and 2010. Uncertainties of causes and risks. From the team for the massive Institute for Health Metrics and Evaluation Global Burden of Diseases, Injuries, and Risk Factors Study 2010. [more inside]
The Nature of Computation - Intellects Vast and Warm and Sympathetic: "I hand you a network or graph, and ask whether there is a path through the network that crosses each edge exactly once, returning to its starting point. (That is, I ask whether there is a 'Eulerian' cycle.) Then I hand you another network, and ask whether there is a path which visits each node exactly once. (That is, I ask whether there is a 'Hamiltonian' cycle.) How hard is it to answer me?" (via) [more inside]
"We have little trouble recognizing that a chess grandmaster’s victory over a novice is skill, as well as assuming that Paul the octopus’s ability to predict World Cup games is due to chance. But what about everything else?" [Luck and Skill Untangled: The Science of Success]
"Parting with treasure easier said than done: Churchgoers give far less than they think" is the latest feature article from the Association of Religion Data Archives, which "strives to democratize access to the best data on religion." The site includes a browsable archive of religious survey data, a quick statistical roundup, international religious profiles, feature articles on topics like the rise of Mormons, Muslims and nondenominational churches in the USA ("nondenominational and independent churches may now be considered the third largest religious group in the country...Only the Catholic Church and the Southern Baptist Convention are larger"), links to sources like the 2010 U.S. Religious Census, a Religion Research Hub (with tutorials and helpful advice on best practices when theorizing, conceptualizing and measuring religious behavior) and lots more.
galton.org is an exhaustive website devoted to the life and works of the statistical pioneer and "father of eugenics" Francis Galton, inventor of the scatterplot, the correlation coefficient, fingerprint identification, and who knows what else. Almost all of Galton's books and papers are reproduced here, some in scanned form and some in searchable .pdf, from his major books to his letters to Pigeon Fancier's Journal. A short selection after the fold. [more inside]
The rise and fall of personal computing - A neat (and in some ways, stark) visualization of the impact of mobile devices on computing
Trials and Errors. Jonah Lehrer's latest piece in Wired is a sort of sequel to his earlier article in the New Yorker on the decline effect (previously). Where that article focused on the institutional factors interfering with the accumulation of truth, this one focuses on the philosophical issues of causation and correlation in modern science. [Via]
The year was 1945. Two earthshaking events took place: the successful test at Alamogordo and the building of the first electronic computer. Their combined impact was to modify qualitatively the nature of global interactions between Russia and the West. No less perturbative were the changes wrought in all of academic research and in applied science. On a less grand scale these events brought about a [renaissance] of a mathematical technique known to the old guard as statistical sampling; in its new surroundings and owing to its nature, there was no denying its new name of the Monte Carlo method (PDF). -N. MetropolisConceptually talked about on MeFi previously, some basic Monte Carlo methods include the Inverse Transform Method (PDF) mentioned in the quoted paper, Acceptance-Rejection Sampling (PDFs 1,2), and integration with and without importance sampling (PDF).
OpenCPU provides a RESTful interface to the popular open-source statistical package R, enabling the user to perform calculations and create publication-quality or web-embeddable visualizations via standard web requests.
Larry Gonick is a veteran American cartoonist best known for his delightful comic-book guides to science and history, many of which have previews online. Chief among them is his long-running Cartoon History of the Universe (later The Cartoon History of the Modern World), a sprawling multi-volume opus documenting everything from the Big Bang to the Bush administration. Published over the course of three decades, it takes a truly global view -- its time-traveling Professor thoroughly explores not only familiar topics like Rome and World War II but the oft-neglected stories of Asia and Africa, blending caricature and myth with careful scholarship (cited by fun illustrated bibliographies) and tackling even the most obscure events with intelligence and wit. This savvy satire carried over to Gonick's Zinn-by-way-of-Pogo chronicle The Cartoon History of the United States, along with a bevy of Cartoon Guides to other topics, including Genetics, Computer Science, Chemistry, Physics, Statistics, The Environment, and (yes!) Sex. Gonick has also maintained a few sideprojects, such as a webcomic look at Chinese invention, assorted math comics (previously), the Muse magazine mainstay Kokopelli & Co. (featuring the shenanigans of his "New Muses"), and more. See also these lengthy interview snippets, linked previously. Want more? Amazon links to the complete oeuvre inside! [more inside]
Statistical hypothesis testing with a p-value of less than 0.05 is often used as a gold standard in science, and is required by peer reviewers and journals when stating results. Some statisticians argue that this indicates a cult of significance testing using a frequentist statistical framework that is counterintuitive and misunderstood by many scientists. Biostatisticians have argued that the (over)use of p-vaues come from "the mistaken idea that a single number can capture both the long-run outcomes of an experiment and the evidential meaning of a single result" and identify several other problems with significance testing. XKCD demonstrates how misunderstandings of the nature of the p-value, failure to adjust for multiple comparisons, and the file drawer problem result in likely spurious conclusions being published in the scientific literature and then being distorted further in the popular press. You can simulate a similar situation yourself. John Ioannidis uses problems with significance testing and other statistical concerns to argue, controversially, that "most published research findings are false." Will the use of Bayes factors replace classical hypothesis testing and p-values? Will something else?
Following the Journal of Personality and Social Psychology's decision to publish Daryl Bem's writeup of 8 studies (PDF) purporting to show evidence for precognition (previously), researchers from the University of Amsterdam have written a rebuttal (PDF) which finds methodological flaws not only in Bem's research, but in many other published papers in experimental psychology. Could this prove to be psychology's cold fusion moment? [more inside]
An examination of the differences between the literary and scientific cultures, by John Allen Poulos.
'Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong.' Dr. John P. A. Ioannidis, adjunct professor at Tufts University School of Medicine is a meta-researcher. 'He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences.' [more inside]
Do we live in a world where there is magic and meaning, or is it all just chance? Radiolab meets two young women who share a nearly unbelievable story of coincidence and fate. Then they consult with statisticians for a very different take on the same story. This short audio documentary is charming and delightful. A Lucky Wind won a Best Documentary: Honorable Mention Award in the 2009 Third Coast / Richard H. Driehaus Foundation Competition as well as the 2009 AAAS Kavli Science Journalism Award (Radio Documentary). [more inside]
Significantly what?...Or how our most common statistical methods really weren't meant to be used that way and why that study result is likely spurious. Since mefites like to argue about stats, here's some background for us all (and I'm not talking correlation vs causation)!
Mercenary Epidemiology: Data Reanalysis and Reinterpretation for Sponsors With Financial Interest in the Outcome. (.pdf link) When should scientists be required to release their raw data for (potentially hostile) re-analysis? A letter to the editors of Annals of Epidemiology from David Michaels, Ph.D., MPH, public health blogger, author of the book Doubt Is Their Product, and, as of December 2009, the Assistant Secretary of Labor for OSHA, unanimously confirmed by the Senate despite the dismay of some. Michaels interviewed at Science Progress about Doubt Is Their Product (podcast, with transcript.)
C0nc0rdance [sytl] asks; How far should we trust common sense? A less than 9 min video on Common Sense as it relates to Science. Enjoy.
Whatever one's opinion of its possible limitations, the 2006 Iraq mortality survey produced epidemiological evidence that coalition forces have failed to protect Iraqi civilians... If, for the sake of argument, the study is wrong and the number of Iraqi deaths is less than half the infamous figure, is it acceptable that "only" 300,000 have died? Last November, with no explanation, the Iraqi Ministry of Health suddenly began citing 150,000 dead, five times its previous estimate. Is that amount of death acceptable? In January, the United Nations reported that more than 34,000 Iraqis were killed violently in the last year alone. Is that acceptable?Regarding The Number, the result of what one of the study's authors calls an episode more deadly than the Rwandan genocide... [more within]
...Would it surprise you to learn that if the Johns Hopkins estimates of 400,000 to 800,000 deaths are correct -- and many experts in the survey field seem to suggest they probably are -- that the supposedly not-yet-civil-war in Iraq has already cost more lives, per capita, than our own Civil War (one in 40 of all Iraqis alive in 2003) ? And that these losses are comparable to what some European nations suffered in World War II ? You'd never know it from mainstream press coverage in the U.S. "Everybody knows the boat is leaking, everybody knows the captain lied," Leonard Cohen once sang. The question the new study raises: How many will go down with the ship, and will the press finally hold the captain fully accountable ?Iraqi Death Rate May Top Our Civil War -- But Will the Press Confirm It ?
See also Debating the Body Count in Iraq
See also Deaths in Iraq: how many, and why it matters
See also The Science of Counting the Dead
See also How the Media Covered The Lancet’s Iraqi Casualty Study
See also More deadly than Saddam
The Logic of Diversity "A new book, The Wisdom of Crowds [..:] by The New Yorker columnist James Surowiecki, has recently popularized the idea that groups can, in some ways, be smarter than their members, which is superficially similar to Page's results. While Surowiecki gives many examples of what one might call collective cognition, where groups out-perform isolated individuals, he really has only one explanation for this phenomenon, based on one of his examples: jelly beans [...] averaging together many independent, unbiased guesses gives a result that is probably closer to the truth than any one guess. While true — it's the central limit theorem of statistics — it's far from being the only way in which diversity can be beneficial in problem solving." (Three-Toed Sloth)
"Chance favours the prepared mind" (Pasteur) but can a science of n = 1 be credible? Seth Roberts is a UCBerkeley Psychology Professor who is into generating novel scientific ideas from self-experimentation. He has written a very serious journal article (abstract) in Behaviour and Brain Science in which he alleges: Seeing faces in the morning on television decreased mood in the evening and improved mood the next day . . . Standing 8 hours per day reduced early awakening and made sleep more restorative . . . Drinking unflavored fructose water caused a large weight loss that has lasted more than 1 year.. among other things. The entire paper was published along with formal peer reviews and a response from Roberts [warning: 63page .pdf] (Peers came down about 50:50 in support/dissenting) A short review/discussion of the article and followup and a short followup Roberts paper with experimental replications (pdf) via
Canada's "Brain Drain" has been a growing concern among Canadians for a number of years. There are a number of reports (PDF) indicating that an increasing number of "highly skilled graduates in fields such as health, engineering and natural and applied sciences" have been heading south for work. There are guides to assist, first hand accounts, and even profiles of people who have left.