MetaFilter posts tagged with data

The map of music

escabeche — Tue, 30 Apr 2013 20:01:23 -0800

Every Noise At Once. A map of musical genres, built by Glenn McDonald of The War Against Silence and the Echo Nest. Click on a genre name to hear a sound sample, or pop it open to see a map of bands within that genre.

Intelligence Tests

kliuless — Thu, 11 Apr 2013 10:40:00 -0800

Is Psychometric g a Myth? - "As an online discussion about IQ or general intelligence grows longer, the probability of someone linking to statistician Cosma Shalizi's essay g, a Statistical Myth approaches 1. Usually the link is accompanied by an assertion to the effect that Shalizi offers a definitive refutation of the concept of general mental ability, or psychometric g." Myths, Sisyphus and g - "Over the years I have not encountered a single endorser of Shalizi's article who actually understands the relevant subject matter. His article is loved for its reassuring conclusions, not the strength of its arguments. I am sure many 'thinkers' resisted Darwinism, the abandonment of geocentrism, and even the notion that the Earth is a sphere, for similar psychological reasons." Nuthin' but a 'g' thang

So I've always had the intuitive hypothesis that there are different types of intelligence; that different people tend to process information in different ways, whether due to habit or nature. But then there are all those people who say that intelligence can be boiled down to a single factor, the mysterious "g" (which I assume stands for either "general intelligence" or "gangsta"). Since this went against years of casual observation, I was somewhat pleased to see the eminent Cosma Shalizi write an essay debunking the notion of "g". But then I saw this blog post defending the notion of "g", and claiming that Shalizi makes a bunch of errors. Basically, the disagreement revolves around the question of why most or all psychometric tests and tasks seem positively correlated with each other. Shalizi points out that this correlation structure will naturally lead to the emergence of a "g"-like factor, even if one doesn't really exist; his opponent points out that if no "g" exists, it should be possible to design uncorrelated psychometric tests, which so far has proven extremely difficult to do. The latter post, by a pseudonymous blogger calling himself "Dalliard", contains a bunch of references to psychometric research that I don't know about and have neither the time nor the will to evaluate, so I'm a bit stumped. Normally I'd leave the matter at that, shrug, and go read something else, but I realized that my intuitive hypothesis about intelligence didn't really seem to be explicitly stated in either of the posts. So I thought I'd explain my conjecture about how intelligence works. In a nutshell, it's this: What if there are multiple "g's"? ...just imagine several dozen hyperplanes, and project them all onto one hyperplane... Remember that psychometric tests are simple mental tasks, but most of the mental tasks we do are complex, like computer programming or chess or writing. And for those tasks, learning and practice matter as much as innate skill, or more (for example, see this study about the neurology of chess players). Therefore, everyone can be "smart" in some way, if "smart" means "good at some complex mental task".

also btw To Make Mice Smarter, Add A Few Human Brain Cells (previously)

"A single cow yields a very large amount of beef..."

cthuljew — Tue, 09 Apr 2013 22:18:14 -0800

Snack Data is a publicly–accessible database of food. It serves as a definitive resource for snack enthusiasts throughout the world.

Yet another reason books are awesome.....as if we needed one.

littleap71 — Tue, 02 Apr 2013 11:37:18 -0800

Mining books to map emotions through a century. Emotion words aren't consistently used through time, it seems. Things got scary in the 80's.

Incommensurable values

kliuless — Mon, 25 Mar 2013 20:42:30 -0800

Economists and the theory of politics - "why unions were often well worth any deadweight cost" abstract:

The standard approach to policymaking and advice in economics implicitly or explicitly ignores politics and political economy, and maintains that if possible, any market failure should be rapidly removed. This essay explains why this conclusion may be incorrect; because it ignores politics, this approach is oblivious to the impact of the removal of market failures on future political equilibria and economic efficiency, which can be deleterious. We outline a simple framework for the study of the impact of current economic policies on future political equilibria — and indirectly on future economic outcomes. We then illustrate the mechanisms through which such impacts might operate using a series of examples. The main message is that sound economic policy should be based on a careful analysis of political economy and should factor in its influence on future political equilibria.

viz. What kind of mass movement with truly powerful institutional support can take the place of unions? - "I agree with just about everything they say about the value of unions, but I also feel forced to acknowledge that it doesn't matter. As a truly powerful mass movement, unions are dead and they aren't coming back. This has left a gaping hole in American politics: Corporations and the rich continue to have enormous institutional power, while the working and middle classes have almost no one to speak for them. I figure that filling this hole is the most important problem the left has to address over the next decade or so. Unfortunately, I don't know how." cf. The Tyranny of Political Economy - "In reality, our contemporary frameworks for political economy are replete with unstated assumptions about the system of ideas underlying the operation of political systems. Make those assumptions explicit, and the decisive role of vested interests evaporates. Policy design, political leadership, and human agency come back to life... Expand the range of feasible strategies (which is what good policy design and leadership do), and you radically change behavior and outcomes." also btw...

Markets in almost nothing - "One of the first things I noticed when I started studying economics was that goods that can't be bought and sold are basically ignored."
Wealth and Motivations for Saving - "teach the public more about how wealth builds over time"
The Supply and Demand for Safe Assets - "Where do safe assets come from? Empirical evidence suggests that the private sector creates more near riskless assets when the supply of government debt is low and reduces privately-created near riskless assets when the supply of government debt is high."
The safe asset shortage - "Safe debts – or what is often called information insensitive assets, as they do not suffer from the types of financial frictions that are characteristic to other financial assets – play a major role in facilitating transactions for institutional investors. And, as we have learned in the recent years, they also play a major role in triggering financial crises when they lose their safety status and turn into information sensitive assets."
GDP, welfare and the rise of data-driven activities - "The worry today is not that investment in technology might not be as productive as we thought (the so-called computer paradox), but the fact that the economic value of the fast growing consumption and production of online data may not be adequately captured in official statistics."
Uncle Sam, venture capitalist - "AMERICA, like much of the world, is facing a crisis of innovation. Its roots rest in several significant challenges: an awareness that rapid technological progress and growth will be crucial in weathering demographic headwinds and the threat of climate change among them. But there is very little consensus in Washington on just what the government ought to be doing to help."
What Google Reader tells us about banking and nationalisation - "Which is why the government taking charge of a service like RSS for the benefit of the public good — or for that matter providing the country with universal internet or high quality media — should not necessarily be treated with suspicion or mistrust. In the civilized world there is a perfectly reasonable way to ensure arm's length detachment and to protect such institutions from the political meddling of government."
If You Can't Fix It, You Don't Own It - "Who owns our stuff? The answer used to be obvious. Now, with electronics integrated into just about everything we buy, the answer has changed. We live in a digital age, and even the physical goods we buy are complex. Copyright is impacting more people than ever before because the line between hardware and software, physical and digital has blurred. The issue goes beyond cellphone unlocking, because once we buy an object — any object — we should own it. We should be able to lift the hood, unlock it, modify it, repair it... without asking for permission from the manufacturer. But we really don't own our stuff anymore (at least not fully); the manufacturers do. Because modifying modern objects requires access to information: code, service manuals, error codes, and diagnostic tools."

like think about sovereign debt -- that is safe assets -- more as a national equity: [1,2,3,4,5]

The US national debt is in truth - like all national debts - a complete and surreal fiction: it is a national equity, the greater part of which is interest-bearing either as claims over public or private revenues. At least two-thirds of the quasi tax credits created by banks came into existence as mortgage loans, and are therefore backed by claims over the productive value of the US land and buildings which they fund. Much of the rest consists of claims over the value of US assets which fund the productive capacity of US corporations. The remainder - which provides the credit necessary to finance the circulation of goods and services in the US - is based upon the magnificent productive capacity of the US people. Only by liquidating US Incorporated could this National Equity ever be redeemed... There is no shortage of dollars because every dollar's worth of productive capacity - public or private; productive people or productive assets - in the US is the capacity to issue a dollar credit, which reflects the increase in the US national wealth which underpins the US national equity. President Barack Obama and his government should get busy creating national equity by instructing the Fed to create and issue the necessary finance for the creation of a new generation of US infrastructure; the transition to a low carbon future which the US can, and should, be leading; and in increasing the capacity of the US people to do so.

(or how the government budget constraint is different than a household's or corporation's -- namely that they can tax and can't be liquidated, unless extraordinarily mismanaged or conquered, I guess...)

Knock, knock. Who's there? Banana. Banana who?

jquinby — Mon, 18 Mar 2013 08:02:50 -0800

"While playing around with the Nmap Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. " After completing the scan of roughly one hundred thousand IP addresses, we realized the number of insecure devices must be at least one hundred thousand. Starting with one device and assuming a scan speed of ten IP addresses per second, it should find the next open device within one hour. The scan rate would be doubled if we deployed a scanner to the newly found device. After doubling the scan rate in this way about 16.5 times, all unprotected devices would be found; this would take only 16.5 hours. Additionally, with one hundred thousand devices scanning at ten probes per second we would have a distributed port scanner to port scan the entire IPv4 Internet within one hour. Don't miss the browsable Hilbert map.

The Daily Viz

frimble — Tue, 12 Mar 2013 06:09:41 -0800

Matt Stiles is a data journalist for NPR. He tries each day to create a data visualization, or post those he finds online. e.g. A heatmap of birthdays, baseball standings at a glance, and health insurance rates by county.

Finding info on your state lawmakers has never been easier

joedan — Sat, 02 Mar 2013 10:06:36 -0800

Open States , the latest project of the nonprofit Sunlight Foundation, is an easy to navigate comprehensive database of activities from all state capitols that makes it easy to find your state lawmaker, review their votes, search for legislation, track bills and much more.

The Quantified Man

instinkt — Tue, 26 Feb 2013 15:22:37 -0800

The culture of hyper capitalism. "Our work is being re-quantified — in a big way — and Chris Dancy, a director in the office of the chief technology officer at BMC Software, thinks it’s time for employees to take these metrics into their own hands." [Previously on MF] “If you can measure it, someone will,” he says, “and that somebody should be you". Quantified Self Douglas Rushkoff on quantification (listening)

Selecting Instagram Filter "Supreme Leader Glow"

Wordshore — Mon, 25 Feb 2013 15:01:09 -0800

Visitors to, and other non-residents in, North Korea are now able to tweet and instagram, as mobile data services are gradually opened up. (Probably) the first tweet sent in this way appeared earlier today. (Probably) the first Instagram picture out of North Korea, and another one a few hours later. Russia Today reports that "... upon arrival to North Korea tourists will be required to fill a special form to purchase a 3G SIM-card for the price of 50 euro and will have to pay hefty 0.38 – 1.43 euro per minute of talk". ...though the Wall Street Journal says that "The intial ‘set-up’ cost to enable mobile devices to connect to Koryolink’s network is 75 euros for a USB modem or 150 euros for a SIM card, according to the report. Then the charges for data usage range from 150 euros for 2 gigabytes to 400 euros for 10 gigabytes. The 2 gigabyte package appears to be the cheapest option available, meaning visitors would face a total bill of 225 euros, or around $300, if using a USB modem." Newsday reports that: "Late leader Kim Jong Il was revealed to have been a Mac user. His Macbook Pro, or a replica, is enshrined at the Kumsusan mausoleum where his body lies in state."

This is Who They Are

misozaki — Wed, 20 Feb 2013 16:50:08 -0800

Infographic about Japanese teens aged 12-15 (SLYT) "2.8% are students between 12-15" means out of the entire population of Japan. "Middle School Recognizes" means the percentage of junior high schools that were aware of bullying problems. "The Number of School Refusal" means the number of kids who stopped going to school (futoko).

U.S. Chamber of Commerce influence in European Parlement

jeffburdges — Tue, 12 Feb 2013 05:56:49 -0800

E.U. Data Protection Directive has many proposed amendments that MEPs cut and pasted directly from American right-wing lobbyists group and ALEC member the U.S. Chamber of Commerce (previously). The U.S. Chamber of Commerce spent over $136M last year lobbying for U.S. corporate interests. They have opposed action on climate change, worked against healthcare reform, opposed financial regulations like Sarbanes-Oxley, supported SOPA, and lobbied to weaken the Foreign Corrupt Practices Act (previously).

Masters of the Internet, Le Monde Diplomatique

infini — Tue, 12 Feb 2013 03:47:14 -0800

To understand what is at stake we need to make our way through the rhetorical smog. For months prior to the WCIT, the Euro-American press trumpeted warnings that this was to be an epochal clash between upholders of an open Internet and would-be government usurpers, led by authoritarian states like Russia, Iran and China. The terms of reference were set so rigidly that one European telecom company executive called it a campaign of “propaganda warfare” (2). ~ Masters of the Internet, Le Monde Diplomatique

Mapping Data

kaibutsu — Tue, 12 Feb 2013 00:41:29 -0800

In December, the Philadelphia police department released a csv database of major crimes (murder, rape, burglary, etc) since 2006. Since then, community software developers have been mapping the data. The community involvement is hoped to spur the future release of large city data sets. The mapping projects: PHL Crime Mapper Philly Crime Map Philly Rap Sheet (Using court data) AXIS Philly: Changes in Crime by Neighborhood Philly Homicides Animation

Everyone wants to pet Riker

urbanwhaleshark — Tue, 08 Jan 2013 16:39:01 -0800

Original ST:TNG makeup and wardrobe footage from 1987

Visualization of large scale datasets looks darn pretty

legweak — Mon, 07 Jan 2013 09:18:02 -0800

null_sets is a new body of artwork aimed at exploring the gap between data and information. Consisting of a set of images, this project stems from our interest in glitches, code-breaking, and translation. our custom script encodes text files as images, making it possible to visualize both the size and architecture of large-scale data sets through an aesthetic lens. So if you ever wanted to see hamlet as a jpeg and find artistic merit hiding within its code, here's your chance. A collaborative project that examined the interplay of data, information, and knowledge won the jury prize for the Distributed Microtopias exhibition at the 15th Annual Finger Lakes Environmental Film Festival. The University of Tennessee's Remote Data Analysis and Visualization Center (RDAV) collaborated with University of Tennessee Knoxville artist Evan Meaney on the project, called Null_Sets, which is a collection of artwork that visualizes the size and structure of data. The artwork was created using an open source script developed at RDAV in which whole bodies of text, from classic literature to HTML to genomic data, can be exported as digital images. Null_Sets uses encoding to represent the changes in pixel color and intensity, and might be adapted to explore how values in a data set change. "This project makes it possible to visualize both the size and architecture of large-scale data sets through an aesthetic lens," Meaney says. RDAV's Amy Szczepanski notes "the techniques we developed in this project laid the groundwork for a larger project that will likely use the Nautilus supercomputer (managed by the National Institute for Computational Sciences) in the future."

Can you find your dot?

hat_eater — Sat, 05 Jan 2013 12:20:16 -0800

Census Dotmap is the visual representation of all persons counted in the 2010 US and 2011 Canadian censuses (via).

Could I BE having any less sex?

crossoverman — Fri, 04 Jan 2013 17:13:44 -0800

The One Where Someone Calculated How Many People The Friends Slept With and a graph. (via)

291 diseases and injuries + 67 risk factors + 1,160 non-fatal complications = 650 million estimates of how we age, sicken, and die

hat — Fri, 14 Dec 2012 11:08:08 -0800

As humans live longer, what ails us isn't necessarily what kills us: five data visualizations of how we age, sicken, and die. Causes of death by age, sex, region, and year. Heat map of leading causes and risks by region. Changes in leading causes and risks between 1990 and 2010. Healthy years lost to disability vs. life expectancy in 1990 and 2010. Uncertainties of causes and risks. From the team for the massive Institute for Health Metrics and Evaluation Global Burden of Diseases, Injuries, and Risk Factors Study 2010. The data visualizations page also includes a large number of static figures from the GBD 2010 study. The GBD 2010 team at the Institute for Health Metrics and Evaluation (IHME), an independent global health research center at the University of Washington, "spent almost 5 years building [the database of causes of death]; we have included almost 800 million deaths from 1950 to 2010, and the data come from different sources," resulting in "the biggest database for cause of death analysis in the world," according to Rafael Lozano at the University of Washington. The GBD 2010 includes:

291 diseases and injuries
67 risk factors
1,160 sequelae (nonfatal health consequences)
Estimates for 21 regions
Estimates for 20 age groups

Takeaway points from the press release (emphasis mine):

Child mortality has dropped by more than 60%, falling "so quickly that it has beaten every published prediction," though diarrhea and other infectious diseases still kill well over a million children under the age of 5 every year.
Deaths among adults aged 15 to 49 increased by 44% between 1970 and 2010, "in part because of increases in violence and the ongoing challenge of HIV/AIDS."
Undernutrition has "successfully been cut by two-thirds," and now physical inactivity and macronutrient overnutrition contribute to a 10%-and-rising fraction of the disease burden.

However,

The trends identified in GBD 2010 occur across regions with one notable exception: sub-Saharan Africa, where infectious diseases, childhood illnesses, and maternal causes of death account for as much as 70% of the burden of disease. By comparison, these conditions account for only one-third of the burden in south Asia and Oceania, and less than 20% in all other regions.

Research findings from the main publication summary (emphasis mine):

In 2010, the three leading risk factors for global disease burden were high blood pressure [...]; followed by tobacco smoking, including secondhand smoke [...]; and alcohol use [...] This reflects a substantial change from 1990 when the leading risk factors were childhood underweight [...]; followed by household air pollution from use of solid fuels [...]; and tobacco smoking, including secondhand smoke [...].
Regional differences are significant. While much of the world is burdened by obesity and high body mass index, underweight is still the leading risk factor in sub‐Saharan Africa. Other prevalent risk factors in the region include household air pollution and nonexclusive and discontinued breastfeeding.
Although child undernutrition has fallen significantly as a risk factor for all ages, it remained the leading risk factor worldwide in 2010 for children under 5, accounting for 12.4% of global DALYs [disability‐adjusted life years], followed by nonexclusive or discontinued breastfeeding at 7.6%.
A number of risks that primarily affect childhood communicable diseases, including unsafe water and sanitation and micronutrient deficiencies, declined in significance in the past 20 years, with unsafe water and sanitation accounting for only 0.9% of global DALYs in 2010.
At the same time, GBD 2010 findings show the importance of household air pollution from solid fuels and ambient particulate matter pollution as major risk factors. One or both rank in the top 10 as causes of disease burden in 13 of the 21 regions. In south Asia, they are the leading cause of burden.

From the Guardian's reporting (more: 1, 2, 3):

Since 1970 the largest gains in life expectancy have taken place in the Maldives (27.3 years for men and 29.4 years for women) and improvements in life expectancy at birth in excess of 20 years were recorded in Bangladesh, Bhutan, Iran, and Peru. But life expectancy fell by one to seven years in Zimbabwe and Lesotho, where populations were severely affected by HIV/Aids, and for men in Ukraine and Belarus, where an alcohol crisis took hold.

If you'd like to read the papers themselves, the full text is currently available for free (but registration required) at The Lancet. In addition to the data visualizations, the IHME provides GBD 2010 publication summaries: Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010

Healthy life expectancy for 187 countries, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010

A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010

Age‐specific and sex‐specific mortality in 187 countries, 1970–2010: a systematic analysis for the Global Burden of Disease Study 2010

Years lived with disability (YLDs) for 1,160 sequelae of 289 diseases and injuries, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010

Disability‐adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010

Common values in assessing health outcomes from disease and injury: disability weights measurement study for the Global Burden of Disease Study 2010
Each of the data visualizations again:

Causes of death by age, sex, region, and year
Heat map of leading causes and risks by region
Changes in leading causes and risks between 1990 and 2010
Healthy years lost to disability vs. life expectancy in 1990 and 2010
Uncertainties of causes and risks

The Institute for Health Metrics and Evaluation adds,

The findings are being announced at the Royal Society in London on Dec. 14 and published in The Lancet, the first time the journal has dedicated an entire triple issue to one study.

The Lancet includes The story of GBD 2010: a “super-human” effort:

“Running the programs to map the data to our cause list of 291 causes and correcting the bias can take days, even using a powerful cluster of more than 100 computers. The data that we have to store after the modelling process can take 3 terabytes.” [Rafael] Lozano estimates that the storage needed for the causes of death data was 400 times bigger than that for GBD 1990 [...] Majid Ezzati, chair in global and environmental health at the School of Public Health, Imperial College London, UK, told The Lancet, As researchers, we tend to believe that more ‘data’ are better than less. I still believe so. But more data, but not all the perfect data we could wish for, means that we need to fundamentally think differently about when to stop searching for more and how to use it [...] The study brought out the well-known but far too frequently overlooked issue that people of different scientific traditions—clinicians, basic scientists, epidemiologists, and quantitative scientists—think and speak differently about the same problem. This can of course be a very powerful resource for bringing together different ways of looking at a problem and solving in the most comprehensive and interesting way—a true systems approach. It can also be a challenge, and at times a source of tension.

(Previously on the Institute for Health Metrics and Evaluation: Is there a market for years?)

Stress at MIT

Rustic Etruscan — Mon, 10 Dec 2012 19:22:21 -0800

The Tech, a newspaper at MIT, has published a report about MIT students' stress. (via) The paper's introduction to this issue. The blog post that prompted the study. The open letter from MIT's president urging students to read said blog post.

DNA data storage

latkes — Sun, 25 Nov 2012 11:13:43 -0800

Just think about it for a moment: One gram of DNA can store 700 terabytes of data. That’s 14,000 50-gigabyte Blu-ray discs… in a droplet of DNA that would fit on the tip of your pinky. To store the same kind of data on hard drives — the densest storage medium in use today — you’d need 233 3TB drives, weighing a total of 151 kilos. In Church and Kosuri’s case, they have successfully stored around 700 kilobytes of data in DNA — Church’s latest book, in fact — and proceeded to make 70 billion copies (which they claim, jokingly, makes it the best-selling book of all time!) totaling 44 petabytes of data stored. via

The Library of Babel in 140 characters (or fewer)

0bvious — Sat, 27 Oct 2012 08:15:36 -0800

The universe (which others call The Twitter) is composed of every word in the English language; Shakespeare's folios, line-by-line-by-line; the Exegesis of Philip K. Dick, exploded; Constantine XI, in 140 character chunks; Sun Tzu's Art of War, in its entirety; the chapter headings of JG Ballard, in abundance; and definitive discographies of Every. Artist. Ever... All this, I repeat, is true, but one hundred forty characters of inalterable wwwtext cannot correspond to any language, no matter how dialectical or rudimentary it may be. [previously]

Most children: 13 - Samwise Gamgee & Rose Cotton

the man of twists and turns — Tue, 23 Oct 2012 01:05:08 -0800

Lord Of The Rings: Statistics (part of LOTRProject)

honeypot

lalex — Sun, 14 Oct 2012 13:05:17 -0800

HoneyMap is an interesting data visualization project depicting cyberattacks. Details.

High Tech's Dirty Little Secret

nowhere man — Sun, 23 Sep 2012 06:43:32 -0800

"Of all the things the Internet was expected to become, it is safe to say that a seed for the proliferation of backup diesel generators was not one of them." Power, Pollution and the Internet [sl NY Times]