MetaFilter posts tagged with data and information

Intelligence Tests

kliuless — Thu, 11 Apr 2013 10:40:00 -0800

Is Psychometric g a Myth? - "As an online discussion about IQ or general intelligence grows longer, the probability of someone linking to statistician Cosma Shalizi's essay g, a Statistical Myth approaches 1. Usually the link is accompanied by an assertion to the effect that Shalizi offers a definitive refutation of the concept of general mental ability, or psychometric g." Myths, Sisyphus and g - "Over the years I have not encountered a single endorser of Shalizi's article who actually understands the relevant subject matter. His article is loved for its reassuring conclusions, not the strength of its arguments. I am sure many 'thinkers' resisted Darwinism, the abandonment of geocentrism, and even the notion that the Earth is a sphere, for similar psychological reasons." Nuthin' but a 'g' thang

So I've always had the intuitive hypothesis that there are different types of intelligence; that different people tend to process information in different ways, whether due to habit or nature. But then there are all those people who say that intelligence can be boiled down to a single factor, the mysterious "g" (which I assume stands for either "general intelligence" or "gangsta"). Since this went against years of casual observation, I was somewhat pleased to see the eminent Cosma Shalizi write an essay debunking the notion of "g". But then I saw this blog post defending the notion of "g", and claiming that Shalizi makes a bunch of errors. Basically, the disagreement revolves around the question of why most or all psychometric tests and tasks seem positively correlated with each other. Shalizi points out that this correlation structure will naturally lead to the emergence of a "g"-like factor, even if one doesn't really exist; his opponent points out that if no "g" exists, it should be possible to design uncorrelated psychometric tests, which so far has proven extremely difficult to do. The latter post, by a pseudonymous blogger calling himself "Dalliard", contains a bunch of references to psychometric research that I don't know about and have neither the time nor the will to evaluate, so I'm a bit stumped. Normally I'd leave the matter at that, shrug, and go read something else, but I realized that my intuitive hypothesis about intelligence didn't really seem to be explicitly stated in either of the posts. So I thought I'd explain my conjecture about how intelligence works. In a nutshell, it's this: What if there are multiple "g's"? ...just imagine several dozen hyperplanes, and project them all onto one hyperplane... Remember that psychometric tests are simple mental tasks, but most of the mental tasks we do are complex, like computer programming or chess or writing. And for those tasks, learning and practice matter as much as innate skill, or more (for example, see this study about the neurology of chess players). Therefore, everyone can be "smart" in some way, if "smart" means "good at some complex mental task".

also btw To Make Mice Smarter, Add A Few Human Brain Cells (previously)

Incommensurable values

kliuless — Mon, 25 Mar 2013 20:42:30 -0800

Economists and the theory of politics - "why unions were often well worth any deadweight cost" abstract:

The standard approach to policymaking and advice in economics implicitly or explicitly ignores politics and political economy, and maintains that if possible, any market failure should be rapidly removed. This essay explains why this conclusion may be incorrect; because it ignores politics, this approach is oblivious to the impact of the removal of market failures on future political equilibria and economic efficiency, which can be deleterious. We outline a simple framework for the study of the impact of current economic policies on future political equilibria — and indirectly on future economic outcomes. We then illustrate the mechanisms through which such impacts might operate using a series of examples. The main message is that sound economic policy should be based on a careful analysis of political economy and should factor in its influence on future political equilibria.

viz. What kind of mass movement with truly powerful institutional support can take the place of unions? - "I agree with just about everything they say about the value of unions, but I also feel forced to acknowledge that it doesn't matter. As a truly powerful mass movement, unions are dead and they aren't coming back. This has left a gaping hole in American politics: Corporations and the rich continue to have enormous institutional power, while the working and middle classes have almost no one to speak for them. I figure that filling this hole is the most important problem the left has to address over the next decade or so. Unfortunately, I don't know how." cf. The Tyranny of Political Economy - "In reality, our contemporary frameworks for political economy are replete with unstated assumptions about the system of ideas underlying the operation of political systems. Make those assumptions explicit, and the decisive role of vested interests evaporates. Policy design, political leadership, and human agency come back to life... Expand the range of feasible strategies (which is what good policy design and leadership do), and you radically change behavior and outcomes." also btw...

Markets in almost nothing - "One of the first things I noticed when I started studying economics was that goods that can't be bought and sold are basically ignored."
Wealth and Motivations for Saving - "teach the public more about how wealth builds over time"
The Supply and Demand for Safe Assets - "Where do safe assets come from? Empirical evidence suggests that the private sector creates more near riskless assets when the supply of government debt is low and reduces privately-created near riskless assets when the supply of government debt is high."
The safe asset shortage - "Safe debts – or what is often called information insensitive assets, as they do not suffer from the types of financial frictions that are characteristic to other financial assets – play a major role in facilitating transactions for institutional investors. And, as we have learned in the recent years, they also play a major role in triggering financial crises when they lose their safety status and turn into information sensitive assets."
GDP, welfare and the rise of data-driven activities - "The worry today is not that investment in technology might not be as productive as we thought (the so-called computer paradox), but the fact that the economic value of the fast growing consumption and production of online data may not be adequately captured in official statistics."
Uncle Sam, venture capitalist - "AMERICA, like much of the world, is facing a crisis of innovation. Its roots rest in several significant challenges: an awareness that rapid technological progress and growth will be crucial in weathering demographic headwinds and the threat of climate change among them. But there is very little consensus in Washington on just what the government ought to be doing to help."
What Google Reader tells us about banking and nationalisation - "Which is why the government taking charge of a service like RSS for the benefit of the public good — or for that matter providing the country with universal internet or high quality media — should not necessarily be treated with suspicion or mistrust. In the civilized world there is a perfectly reasonable way to ensure arm's length detachment and to protect such institutions from the political meddling of government."
If You Can't Fix It, You Don't Own It - "Who owns our stuff? The answer used to be obvious. Now, with electronics integrated into just about everything we buy, the answer has changed. We live in a digital age, and even the physical goods we buy are complex. Copyright is impacting more people than ever before because the line between hardware and software, physical and digital has blurred. The issue goes beyond cellphone unlocking, because once we buy an object — any object — we should own it. We should be able to lift the hood, unlock it, modify it, repair it... without asking for permission from the manufacturer. But we really don't own our stuff anymore (at least not fully); the manufacturers do. Because modifying modern objects requires access to information: code, service manuals, error codes, and diagnostic tools."

like think about sovereign debt -- that is safe assets -- more as a national equity: [1,2,3,4,5]

The US national debt is in truth - like all national debts - a complete and surreal fiction: it is a national equity, the greater part of which is interest-bearing either as claims over public or private revenues. At least two-thirds of the quasi tax credits created by banks came into existence as mortgage loans, and are therefore backed by claims over the productive value of the US land and buildings which they fund. Much of the rest consists of claims over the value of US assets which fund the productive capacity of US corporations. The remainder - which provides the credit necessary to finance the circulation of goods and services in the US - is based upon the magnificent productive capacity of the US people. Only by liquidating US Incorporated could this National Equity ever be redeemed... There is no shortage of dollars because every dollar's worth of productive capacity - public or private; productive people or productive assets - in the US is the capacity to issue a dollar credit, which reflects the increase in the US national wealth which underpins the US national equity. President Barack Obama and his government should get busy creating national equity by instructing the Fed to create and issue the necessary finance for the creation of a new generation of US infrastructure; the transition to a low carbon future which the US can, and should, be leading; and in increasing the capacity of the US people to do so.

(or how the government budget constraint is different than a household's or corporation's -- namely that they can tax and can't be liquidated, unless extraordinarily mismanaged or conquered, I guess...)

Visualization of large scale datasets looks darn pretty

legweak — Mon, 07 Jan 2013 09:18:02 -0800

null_sets is a new body of artwork aimed at exploring the gap between data and information. Consisting of a set of images, this project stems from our interest in glitches, code-breaking, and translation. our custom script encodes text files as images, making it possible to visualize both the size and architecture of large-scale data sets through an aesthetic lens. So if you ever wanted to see hamlet as a jpeg and find artistic merit hiding within its code, here's your chance. A collaborative project that examined the interplay of data, information, and knowledge won the jury prize for the Distributed Microtopias exhibition at the 15th Annual Finger Lakes Environmental Film Festival. The University of Tennessee's Remote Data Analysis and Visualization Center (RDAV) collaborated with University of Tennessee Knoxville artist Evan Meaney on the project, called Null_Sets, which is a collection of artwork that visualizes the size and structure of data. The artwork was created using an open source script developed at RDAV in which whole bodies of text, from classic literature to HTML to genomic data, can be exported as digital images. Null_Sets uses encoding to represent the changes in pixel color and intensity, and might be adapted to explore how values in a data set change. "This project makes it possible to visualize both the size and architecture of large-scale data sets through an aesthetic lens," Meaney says. RDAV's Amy Szczepanski notes "the techniques we developed in this project laid the groundwork for a larger project that will likely use the Nautilus supercomputer (managed by the National Institute for Computational Sciences) in the future."

The Library of Babel in 140 characters (or fewer)

0bvious — Sat, 27 Oct 2012 08:15:36 -0800

The universe (which others call The Twitter) is composed of every word in the English language; Shakespeare's folios, line-by-line-by-line; the Exegesis of Philip K. Dick, exploded; Constantine XI, in 140 character chunks; Sun Tzu's Art of War, in its entirety; the chapter headings of JG Ballard, in abundance; and definitive discographies of Every. Artist. Ever... All this, I repeat, is true, but one hundred forty characters of inalterable wwwtext cannot correspond to any language, no matter how dialectical or rudimentary it may be. [previously]

honeypot

lalex — Sun, 14 Oct 2012 13:05:17 -0800

HoneyMap is an interesting data visualization project depicting cyberattacks. Details.

use value vs. exchange value

kliuless — Sat, 01 Sep 2012 00:12:16 -0800

What Is Value? What Is Money? (via via)

What is value about? and how do we measure value? Traditionally, the way of measuring value has been not been through measures of value, but actually through measures of appropriation: measures of the amount of money that you can appropriate through that business, not the value that it generates in society. We're starting to see this difference.

Reinventing Society In The Wake Of Big Data

With Big Data we can now begin to actually look at the details of social interaction and how those play out, and are no longer limited to averages like market indices or election results. This is an astounding change. The ability to see the details of the market, of political revolutions, and to be able to predict and control them is definitely a case of Promethean fire -- it could be used for good or for ill, and so Big Data brings us to interesting times. We're going to end up reinventing what it means to have a human society.

more on 'data science' -No, Really, Some of My Best Friends Are Data Scientists -Data Science: The Problem Isn't Statisticians, It's Too Many Poseurs more on 'big data' -Moving from Big Data to Smart Data -Top 10 categories for Big Data sources and mining technologies -Hadoop 2.0 -Grid computing -NSA Overview of Cloud Computing When Central Is Essential - "economist Glen Weyl* views changes in technology as leveling the playing field between central governments and free markets"

In his famous 1945 article, "The Use of Knowledge in Society," F. A. Hayek argued that despite their inequity and inefficiency, free markets were necessary in order to allow the incorporation of information held by dispersed individuals into social decisions. No central planner could hope to collect and process all the information necessary for social decisions; only markets allowed and provided the incentives for disaggregated information processing. Yet, increasingly, information technology is leading individuals to delegate their most "private" decisions to automated processing systems... While these information systems are [currently] mostly nongovernmental, they are sufficiently centralized that it is increasingly hard to see how dispersed information poses the challenge it once did to centralized planning.

The Optimist's Case For Information Technology

Schools have poured a ton of money into ICT and tech-focused startups are circling the education sector aiming to disrupt it. Something's going to change here. IBM built a supercomputer that can win at Jeopardy which is cool but useless and now they're trying to turn it into a medical diagnostic engine. Computer-driven cars are already a real thing. Right now, price is a big barrier to adoption but if there's anything we know about ICT hardware it's that the price will fall... I don't know that doing a medical consult with your home computer and then having your prescription automatically disatched by autonomous vehicle equals a change comparable to indoor plumbing, but it's a big deal.

Two Types of Knowledge: Human Capital and Information - "Human capital is knowledge that is hard to transfer. Information is knowledge that is easy to transfer." Desire Modification: the ultimate technology - "With humanity divided into clades by motivation and personality type, evolution would be very important." [1,2] --- *The Situation Bears Watching - "Hermann Weyl, the mathematician who was Einstein's friend, was a great uncle."

Digital Images are SomeThing to aspire to? (A reflection on Hito Steyerl's proposal)

0bvious — Thu, 16 Feb 2012 05:13:43 -0800

Artist and film-maker, Hito Steyerl, asks us to stand shoulder to shoulder with our digital equivalents. Digital images are Things (like you and me) - a plethora of compressed, corrupted representations pushed and pulled through increasingly policed and capitalised information networks. If 80% of all internet traffic^* is SPAM - a liberated excess withdrawn^** from accepted channels of communication - perhaps it is in The Poor Image we find our closest kin? * In her recent October Journal article (Digital Debris: Spam and Scam), Steyerl claims that "80% of today's email messages are spam" ** This piece by Finn Brunton, (Roar so wildly: Spam, technology and language), is referenced by Steyerl in the same October Journal article

JavaScript InfoVis Toolkit

Blazecock Pileon — Sun, 12 Feb 2012 14:31:07 -0800

JavaScript InfoVis Toolkit (JIT) - providing tools for creating interactive data visualizations for the web

Less data viz, more dinner viz.

carsonb — Tue, 04 Oct 2011 16:49:24 -0800

Taste Buds is a visualization of complementary flavors, sourced from patterns found in lots and lots of recipes. For those of us not blessed with a chef's instincts.

Bubbles and Public Facts

kliuless — Mon, 23 May 2011 14:23:01 -0800

The Destruction of Economic Facts - "Renowned Peruvian economist Hernando de Soto argues that the financial crisis wasn't just about finance—it was about a staggering lack of knowledge" (via)

During the second half of the 19th century... To prevent the breakdown of industrial and commercial progress, hundreds of creative reformers concluded that the world needed a shared set of facts. Knowledge had to be gathered, organized, standardized, recorded, continually updated, and easily accessible... The result was the invention of the first massive "public memory systems" to record and classify—in rule-bound, certified, and publicly accessible registries, titles, balance sheets, and statements of account—all the relevant knowledge available... for investors to infer value, take risks, and track results... Over the past 20 years, Americans and Europeans have quietly gone about destroying these facts. The results are hardly surprising. In the U.S., trust has broken down...

BONUS

The Gaussian copula function tattoo
The financial crisis was a major event which was caused by Wall Street's shortsighted greed — a greed which is epitomized in the way that the copula function became ubiquitous even though risk managers and even Li himself knew full well that it was extremely limited in how it should be used. If we want to 'be vigilant and not forget' the destructive potential of Wall Street, then the Gaussian copula function is actually a really good thing to get as a tattoo. There's an irony here too. Elms got this tattoo, in part, because of its very incomprehensibility — the way it epitomizes the way that Wall Street is 'predicated in obfuscation'. But Wall Street embraced Li's formula for the opposite reason — that it was very tractable and easy to understand, at least if you were a quant with a degree in finance. Elms's tattoo is the version of the formula which I used in the Wired article — but it's not, actually, the version of the formula which was used day-to-day on Wall Street... Most representations of the copula look nothing like that, and are much harder to understand. All of which shows that Elms is absolutely right, at heart, about the copula function and what it represents. To Wall Street, it's simple and easy — disastrously so. To the rest of us, however, it makes a Semiotext(e) book look like a Sesame Street lyric. And that, I think, is why Levin and Elms are going to be disappointed, and Blankfein is going to remain out of jail. Back in 1933, when Ferdinand Pecora uncovered huge scandals on Wall Street, they were easy for all Americans to understand, and easy to protect against. This time around, Wall Street's activities are incomprehensible not only to the lay person but even to senior bankers: a big part of the reason why the crisis was so big and so bad is precisely that people like Stan O'Neal and Bob Rubin failed at their job of understanding the risks their banks were taking. They were knavishly foolish, but still more fools than knaves — which means that it's extremely hard to make a strong case in front of a jury that what they did was criminal.
The new argument against financial innovation - "It is from the not yet but soon to be famous Alp Simsek, at Harvard, and smart people tell me it is important and already influential..." cf. Speculation and Risk Sharing with New Financial Assets
While the traditional view of financial innovation emphasizes the risk sharing role of new financial assets, belief disagreements about these assets naturally lead to speculation... Financial innovation always decreases the uninsurable variance because new assets increase the possibilities for risk sharing. My main result shows that financial innovation also always increases the speculative variance. This is true even if traders completely agree about the payoffs of new assets. The intuition behind this result is the hedge-more/bet-more effect: Traders use new assets to hedge their bets on existing assets, which in turn enables them to place larger bets and take on greater risks. This effect suggests that financial innovation is more likely to be destabilizing in more complete financial markets and when it concerns derivative assets... A question emerges as to how new assets should be introduced to minimize their short run impact on the speculative variance. I show that staggering (or delaying) the introduction of new assets is not effective because it reduces traders learning simultaneously with their speculation. A viable alternative is to set temporary position limits (or taxes) on new assets.
Financial Repression - "There are only a few ways to bring down sovereign debt burdens. Growth is one, but growth may be impaired by high debt burdens. Austerity is another, but to cut debts this way requires a long period of painful policy, of the sort that's rarely tolerable to electorates. Default is another way. And rapid inflation is another way still... a fifth option—financial repression—was key to quickly and relatively painlessly addressing large sovereign debts... The tight financial controls associated with post-Depression financial regulation and the introduction of the Bretton Woods system enabled a period of financial repression that persisted from the end of the war to around 1980. This period was characterised by low real interest rates (during this time they were quite often negative) persistently, modestly high inflation rates, and rapid reduction in debt levels thanks largely to this 'financial repression tax'. It was an incredibly effective mix of policies." cf. The Liquidation of Government Debt [1,2,3]

The Information Sage

Blazecock Pileon — Tue, 17 May 2011 15:53:53 -0800

“If you display information the right way, anybody can be an analyst,” Tufte once told me. “Anybody can be an investigator.” - The Washington Monthly interviews informaticist Edward Tufte [via]

Information Is Soil

Bora Horza Gobuchul — Thu, 10 Feb 2011 02:56:00 -0800

Jer Thorp is the New York Times' current Data Artist in Residence. He creates information-rich animations, most recently of the latest Kepler candidate extrasolar planets [previously]; also a global render of people's uses of Twitter.

Lee Byron is a designer, artist, and biker: his work includes visualisations of Facebook breakups over the course of a year and Hollywood box office revenues, 1986 - 2008.

David McCandless is an "information journalist"; his blog, InformationisBeautiful.net, has been linked to plenty of times on the blue, but you might enjoy this overview of his work and others at TED. Similarly, Hans Rosling, also mentioned previously. A lot of the content is extremely Flash-heavy; full-screen mode is recommended for the videos. Most of the visualizations were built using the Processing language, but if you wish to play around with simple data visualisation with no programming, you might want to try GapMinder.

Wrangler

chunking express — Fri, 04 Feb 2011 08:18:53 -0800

Stanford's Visualization Group has produced a data cleanup web app called Wrangler that works like straight up magic.

Data Tools of the Fuuuuture ... fuuture ... future ... uture... ture ... re ...

stratastar — Tue, 11 Jan 2011 13:23:32 -0800

Dataists give their hopes and dreams for data, data tools and data science in 2011. Already, Google has provided Google Refine (previously) to help clean your datasets. While great visualizations can be created with online tools or by combining R (great posts previously), with ggplot2, GGobi, and even Google Motion Charts With R (already built into Google Spreadsheets). Need data? Needlebase, helps non-programmers scrape, harvest, merge, and data from the web. Or if you’re introspective, Your Flowing Data and Daytum provide tools to measure and chart details of your own life.

Can a person disappear in surveillance Britain?

zarq — Thu, 29 Apr 2010 14:15:14 -0800

It's been estimated that the average UK adult is now registered on more than 700 databases and is caught many times each day by nearly five million CCTV cameras. So how hard would it be for an average citizen to disappear completely? That’s the subject of a new documentary film: Erasing David, (Trailer: YouTube, Vimeo) which premieres this evening in the UK on More4. It's also now available worldwide online at the iTunes store and through several Video On Demand services, as well as through Good Screenings. Erasing David: Facebook / Twitter / Downloadable PDFs can be found at their Protect Yourself page / Education Materials Review: Erasing David and the fight for privacy rights. Good Screenings is a new site dedicated to facilitating screenings of "the best, award-winning social justice filmmaking... films that aren't just good - they do good too..." The CCTV camera statistics in the first line of this FPP were lifted from the TimesOnline article, and may have been sourced from the documentary's marketing materials. In the interest of accuracy, I rephrased their mention as an estimate, because the official statistics which have been released lead to varying conclusions as to how many cameras and databases there really are.

Number of cats I own: 2

defenestration — Sat, 10 Apr 2010 13:09:44 -0800

Infographics — 2010's Animated GIF

Twenty Times a Day

shoesfullofdust — Fri, 24 Apr 2009 19:57:07 -0800

...the Department of Transportation will not keep secret the data we collect on birds striking airplanes. - Ray LaHood, United States Secretary of Transportation

From the dreaded mourning dove to the nefarious Canada goose to the humble armadillo, the FAA's recently released National Wildlife Strike Database ON-LINE contains information on aircraft/wildlife strikes from over 100,000 reported incidents between 1990 and 2008. The database was referenced previously in the Birdstrike in the Hudson thread, but until this week the the F.A.A. provided only aggregate data about wildlife strikes - collisions by state, not by airport. Submit your query here or download the current version(15 MB MS Access format) from this page. Federal Aviation Administration - Wildlife Hazard Mitigation Home Page. Wildlife Strikes to Civil Aircraft in the United States 1990 - 2007 - A comprehensive analysis of the database (pdf format). Sharing the Skies (TP 13549E) is an online book about the subject of aircraft/wildlife strikes from Transport Canada.

visualizing data

nickyskye — Thu, 06 Nov 2008 13:01:53 -0800

20 Useful Visualization Libraries from the excellent A Beautiful WWW. Well, not entirely limited to libraries. Useful stuff for visualization practitioners sounded a little non-specific, though. These are all freely available. A Beautiful WWW created by Todd Holloway.

Data Visualization on the Web

nthdegx — Tue, 07 Aug 2007 15:00:44 -0800

Data Visualization: Modern Approaches is a Smashing Magazine article examining a variety of increasingly popular or novel information visualization employed on modern websites.

blinded me with (beautiful) science

oneirodynia — Mon, 09 Apr 2007 12:19:01 -0800

"To determine whether a diagram is good or bad, one needs to determine for what context it was designed for." PingMag (1, 2) interviews Andrew Vande Moere of infosthetics . A quick, informative read which includes pretty pictures of some MeFi faves.

Rare elements

fatllama — Sun, 07 Jan 2007 22:24:58 -0800

A periodic table of visualization methods.

Google starts an internal futures market

rzklkng — Mon, 26 Sep 2005 05:51:22 -0800

Google's Crystal Ball::NYTimes. Quite interesting...Via TechDirt:

Google has created a predictive market system, basically a way for its employees to bet on the likelihood of possible events. Such markets have long been used to predict world events, like election results. Intrade, part of the Trade Exchange Network, allows people to bet on elections, stock market indexes and even the weather, for example.

I wonder how accurate the aggregated content of blogs would be to measure the likelihood of prospective real world events? The economist they consulted, Hal R. Varian, has some interesting links on his web page as well. I think that the internet better get their anti-spam technology up to par before we have people "gaming" the future through blogspam. For an explanation of Futures Markets (charts), see this page at the US Commodity Futures Trading Commission.

The memespread project

cbrody — Thu, 08 Apr 2004 02:19:16 -0800

The memespread project. How does a meme spread? What part does MetaFilter play in the process? [via waxy.org]

Do the Terrorists Care about Teenage Smoking?

archimago — Thu, 19 Dec 2002 09:50:48 -0800

Information deemed useful to terrorists is disappearing from government Web sites. I know this is old news, but this article details some of the specifics of whas has been happening. "The previous presumption, that publicly-funded information is the rightful property of the public until proven otherwise, has been replaced by the presumption that the public has to prove to a suspicious government that it deserves the information." I understand that as a nation we are hypersensitive now to terrorism, but isn't this just what the terrorists want? The loss of our freedoms to information?

adrianhon — Wed, 28 Nov 2001 08:36:10 -0800

"The Web, left to its own devices, would be the exact opposite of that: It's like a giant city with no neighborhoods; it needs these kind of meta-filters, these second-level kind of things, whether it is Yahoo or Google or Slashdot, to rein in that chaos and turn it to something more organized." From the second page of an interview with the author of Emergence, Steven Johnson (also co-founder of Feed).