Information Wants to be Free

March 4, 2009 3:18 PM

WikiLeaks: every current Congressional Research Service report in a Torrent (2.2 GB).h/t Jessamyn's twitter. Americans spend $100 million a year on the Congressional Research Service, a private think tank for members of Congress and their staffs. While technically available to the public, their reports were never posted on the Internet by the government.

In the past, we've counted on the painstaking, one-at-a-time collection efforts of folks like the Open CRS Network (previously). Now, WikiLeaks and the Pirate Bay (legally) delivers to your desktop some fine research on mountain pine beetles, China's role in the global financial crisis, NATO in Afghanistan, and anything else you can think of. The 110th Congress' HR 2545 would have required that CRS give the public the same (relatively) easy access to its reports Congressional staff has.

posted by l33tpolicywonk (18 comments total)

Impressive find.
posted by bz at 3:34 PM on March 4 [1 favorite has favorites]

This is pretty great, for a lot of reasons. I can't wait to get home and start seeding.
posted by box at 3:37 PM on March 4

Excerpt from a 2003 CRS internal memo regarding making CRS products publicly available (from the third (wikipedia) link):

Impairment of Member Communication with Constituents – The danger of placing CRS, a support agency, in an intermediate position responding directly to constituents instead of preserving the direct relationship between constituents and their elected representatives. This threatens the dialog on policy issues between Members and their constituents that was envisioned by the Constitution.
posted by kisch mokusch at 3:40 PM on March 4 [2 favorites has favorites]

(re-posted in case anybody was wondering, as I was, why this hadn't been made available earlier)
posted by kisch mokusch at 3:43 PM on March 4

Even more interesting is this letter, written today by Senator Joe Lieberman urging Congress to make all CRS reports public.

Of course, that letters comes just a couple days after the same Senator sent a letter to the US courts asking them why they still haven't made the PACER e-filing system available to the public for free.

Out of nowhere, Lieberman seems to somehow be channeling the spirits of the open access and transparency community.

After his back-stabbing of the Dems last year, I had come to really dislike Lieberman. Well, I still dislike him. However, it just goes to show, bad people can sometimes do nice things.
posted by genome4hire at 3:44 PM on March 4 [1 favorite has favorites]

I can't wait to get home and start seeding.

Ewwww, gross
posted by Hoopo at 3:52 PM on March 4 [5 favorites has favorites]

Something about the intersection of The Pirate Bay [ZOMG! Aren't they on trial? Right now? For being criminals?] and the dissemination of government documents that the government seemed not particularly eager to disseminate gives me a sinking feeling that something terrible is going to happen here.
posted by Joe Beese at 4:16 PM on March 4

When you share information, you're sharing TERRORism.
posted by blue_beetle at 4:23 PM on March 4

>However, it just goes to show, bad people can sometimes do nice things.

I'm certain, in Lieberman's case, it was purely by accident.
posted by blastrid at 4:27 PM on March 4 [4 favorites has favorites]

Something about the intersection of The Pirate Bay [ZOMG! Aren't they on trial? Right now? For being criminals?] and the dissemination of government documents that the government seemed not particularly eager to disseminate gives me a sinking feeling that something terrible is going to happen here.

I would have believed that about a month and a half ago.

I don't now.
posted by eyeballkid at 4:35 PM on March 4 [1 favorite has favorites]

This has been sort of slowly moving forward for the past few years with more and more people getting on the bandwagon. I personally have gotten my elected officials to procure two CRS reports that went into OpenCRS but what a pain in the ass. While I totally understand not wanting the CRS people to have to do customer service, I'm astonished that it's taken as long as it did to get this information public. Also, my heads up came from Justin.
posted by jessamyn at 5:10 PM on March 4

Also I'm a little surprised that there's a Pirate Bay link on the from page of MeFi and I dn't mind a bit.
posted by jessamyn at 5:13 PM on March 4 [1 favorite has favorites]

For some reason, Azureus 4.1.0.2 fails to fetch the .torrent from TPB. [It complains that a route to the host could not be found.] (Running under Gentoo Linux w/ Sun's JDK 1.6.0.12) Wget or Firefox on the same machine have no such issues.

Azureus also fails to connect to the torrent's tracker.

Conspiracy?
posted by simoncion at 5:18 PM on March 4

I didn't have any trouble bringing down the torrent, and it went a bit faster than usual -- about 990KB/s.

The linked torrent contains a valid zip file -- a strange choice since ZIP is a very fragile archive format -- with 20,194 files inside. Each report seems to consist of three files: a PDF, a plain text version (weirdly formatted to 100 columns and containing ASCII form feed controls) and a metadata file consisting of Key: value pairs.

Adding it all up, there are 6,731 reports. The most recent is "Underground Carbon Dioxide Sequestration: Frequently Asked Questions" from January 21 of this year. The oldest is "Fish and Wildlife Service: Compensation to Local Governments" from March 6, 1990.

Included is an index.html file, seemingly tacked on by the WikiLeaks folks, containing title, document ID, and date information for each report, and linking to the PDF. It seems to be derived from one of the indexes at WikiLeaks itself where they also identify several browsable archives [1, 2, 3] just in case you didn't feel the need to download the entire 2GB torrent to read one document.


posted by majick at 5:28 PM on March 4 [2 favorites has favorites]

Oh, this is brilliant. Thank you so much!
posted by cereselle at 8:05 AM on March 5

Conspiracy?

Well didn't Azureus turn into a huge corporate content-spammer etc thing after it changed into Vuze? Wouldn't be surprised if there was other screwy things going on with it as well.
posted by FatherDagon at 9:55 AM on March 5

That's a lot of documents. Anyone care to look through and find the highlights?
posted by JHarris at 4:40 PM on March 5

If you search OpenCRS for topics you are interested in, you can figure out which reports might be worth checking out. A few I've been interested in

- Income and Poverty Among Older Americans in 2007
- Internet Privacy: Overview and Legislation in the 109th Congress
- Barriers to Corporate Fraud: How They Work, Why They Fail (pdf)
- The Effect of State-Legalized Same-Sex Marriage on Social Security Benefits and Pensions
- Homosexuals and the U.S. Military: Current Issues
- Vermont Emergency Management and Homeland Security Statutory Authorities Summarized
- Obscenity and Indecency: Constitutional Principles and Federal Statutes
posted by jessamyn at 6:57 PM on March 5