The Internet Archive releases a torrent of torrents
August 8, 2012 7:31 PM   Subscribe

The Internet Archive is now offering over 1,000,000 torrents including our live music concerts, the Prelinger movie collection, the librivox audio book collection, feature films, old time radio, lots and lots of books, and all new uploads from our patrons into Community collections (with more to follow). ... BitTorrent is the now fastest way to download items from the Archive, because the BitTorrent client downloads simultaneously from two different Archive servers located in two different datacenters, and from other Archive users who have downloaded these Torrents already. The distributed nature of BitTorrent swarms and their ability to retrieve Torrents from local peers may be of particular value to patrons with slower access to the Archive, for example those outside the United States or inside institutions with slow connections. (previously)
Interestingly, the Archive’s plans for BitTorrent are not limited to providing an alternative download link for their files. Founder Brewster Kahle says that they are also working on turning it into a storage mechanism.

“The next step is to make BitTorrent a distributed preservation system for content like ours,” Kahle told us. Kahle believes that the Internet Archive and the BitTorrent community can help each other and hopes to get the discussion on the preservation idea started.

“I think this whole thing will be awesome, and possibly very important,” he adds.

In the wake of recent news featuring raids, crackdowns, DDoSes and lawsuits, this announcement from the Internet Archive brings some very welcome positive news about BitTorrent. For those who are interested in tracking how many people are leeching from the archive, here are some fancy graphs.
posted by Egg Shen (41 comments total) 49 users marked this as a favorite
 
FWIW, here is the µTorrent graph of my download speed for Night of the Living Dead.
posted by Egg Shen at 7:31 PM on August 8, 2012


This is awesome! Now I have something even harder to explain than "Linux ISOs" when I'm arguing with my ISP about why they've throttled my bandwidth again...
posted by trackofalljades at 7:36 PM on August 8, 2012 [3 favorites]


This is wonderful and I wish them all the best, but I don't think BitTorrent will work well for long term storage. There seems to be a correlation between how long it takes people to download something and how long they keep it available online for others.
posted by Kevin Street at 8:00 PM on August 8, 2012


My biggest complaint about torrents is that carry very little natural metadata, so you can't quite be sure what you're downloading until hours later. I downloaded "Debbie Gets Her Bandwidth Throttled" and only later found out that it wasn't at all about internet network performance, as I had expected.

Archive.org at least provides a little context before you download, so we don't end up wasting everyone's time. The big providers should stop worrying about the technology of delivery, should stop shooting the messenger, and should start leveraging their respectability as content selectors, because credible selection becomes more valuable as access becomes easier.

Anyway, I'm off to watch the rest of "Debbie Gets Her Bandwidth Throttled" because even though it wasn't what I expected, it's still pretty entertaining.
posted by twoleftfeet at 8:03 PM on August 8, 2012 [5 favorites]


I downloaded "Debbie Gets Her Bandwidth Throttled" and only later found out that it wasn't at all about internet network performance, as I had expected.

He didn't fix the cable (internet)?
posted by Blazecock Pileon at 8:05 PM on August 8, 2012 [4 favorites]


Can I get porn this way? I mean free porn. Not so much worried about legality.
posted by cjorgensen at 8:07 PM on August 8, 2012 [2 favorites]


This is wonderful and I wish them all the best, but I don't think BitTorrent will work well for long term storage. There seems to be a correlation between how long it takes people to download something and how long they keep it available online for others.

As someone without metered bandwidth, I'm happy to seed legal material for quite a while. Since the archive is going to keep stuff seeded this should be a good long term solution, the extra bandwidth from people uploading is just a bonus.
posted by furiousxgeorge at 8:07 PM on August 8, 2012


There seems to be a correlation between how long it takes people to download something and how long they keep it available online for others.

I'm certainly guilty of this, as I generally prefer to download (legally available) torrented material locally (i.e. on a laptop) and then quickly move it to a NAS drive, at which point my torrent client can no longer find it and stream it.

I'm why we can't have nice things.
posted by ShutterBun at 8:09 PM on August 8, 2012


Search Results
Results: 1 through 0 of 0 (0.018 secs)
You searched for: Debbie Gets Her Bandwidth Throttled AND format:"Archive BitTorrent" Advanced search
Your search did not match any items in the Archive.

Drat.
posted by mrbill at 8:10 PM on August 8, 2012 [3 favorites]


Can I get porn this way? I mean free porn. Not so much worried about legality.

Considering that on the Archive's Bittorent stat's page, the most snatched part has several links to My Little Pony porn collections, so I wouldn't put it past some people.
posted by drw------- at 8:13 PM on August 8, 2012 [1 favorite]


He didn't fix the cable (internet)?

Oh boy, how you gonna keep 'em down on the server farm once they've seen Torrent Fungus.
posted by cog_nate at 8:16 PM on August 8, 2012 [1 favorite]


Considering that it seems that this announcement equates to little more than "stuff that you used to be able to watch streaming, or download sequentially (i.e. were able to view it as it downloaded) you can instead download via torrenting, which is non-sequential, probably has very few seeds (apart from Archive's own servers) and must be completely downloaded before viewing.

Unless they can get some other major partners to mirror their collection and increase the number of continuous seeds, it's going to take some major changes before we see any tangible benefits from this, no?
posted by ShutterBun at 8:20 PM on August 8, 2012


My bandwidth's been throttled
my brain is all mottled
my words have been glottaled
my whiskey un-bottled
posted by flapjax at midnite at 8:22 PM on August 8, 2012 [3 favorites]


This is wonderful and I wish them all the best, but I don't think BitTorrent will work well for long term storage.

I'm pretty sure the folks at IA would agree, which is probably only one of several reasons they would never, ever intend these torrents to function as "long term" storage or any other kind of storage. The point is to make things more easily accessible while wasting as little bandwidth as possible, if I understand it correctly.

My biggest complaint about torrents is that carry very little natural metadata, so you can't quite be sure what you're downloading until hours later.

I'm not aware of any torrent clients that refuse you the ability to browse through all the files they're downloading...which are all created and named within minutes if not seconds of starting the torrent? With large files such as video, it shouldn't be very hard for a user to watch at least part of it while it's coming down, at least I've never had trouble doing this. If you're super curious about something specific, you could always just download that file to check it out before getting the rest (most clients support doing this as well).
posted by trackofalljades at 8:23 PM on August 8, 2012


He didn't fix the cable (internet)?

Don't be fatuous.
posted by Pope Guilty at 8:29 PM on August 8, 2012 [2 favorites]


Founder Brewster Kahle told TorrentFreak that turning BitTorrent into a distributed preservation system for the Internet is the next step.

I can kind of see how someone would misinterpret that as "we're intending for torrents to act as storage" but I'd disagree with that reading personally...to me it sounds a lot more like he's discussing how torrent-similar technology could be used in the future.
posted by trackofalljades at 8:32 PM on August 8, 2012


Most downloaded film from the Prelinger Archives by far - About Bananas. So I watched it and can't for the life of me figure out why it would have 19,385,457 downloads. The review from (MetaFilters?) stbalbach was great though [excerpts]:
Zemurray, a Russian Jewish immigrant from the working class, addressed the stodgy Boston Brahmin board of directors of United Fruit: “You gentlemen have been fucking up this business long enough. I’m going to straighten it out,” and immediately fired them all.

This propaganda film is from the school of Edward Bernays, the originator of modern public relations (the Nazi's were avid followers of Bernays techniques but for ends more nefarious than eating bananas).

…the CIA today uses propaganda to destabilize or help overthrow governments all over the world for American business and political interests. It all goes back to bananas and Zemurray and United Fruit and the banana republics of the old days.
So great in fact, that it was commented on by one Chip Cogswelle: "Before clicking play, I usually look at reviews. This film has a long (large charCount) post that is WAY OFF topic, and a little scary. What's goin' on??"
posted by unliteral at 8:32 PM on August 8, 2012 [3 favorites]


I actually worked on a research project five or six years ago that intended to create a system so a distributed, multi-university Bittorrent system would be balanced and feasible. The real motivation was research for an e-cash scheme that didn't work out, so it didn't really go anywhere, but this idea's been floating around.

The problem, though, is that Bittorrent as a protocol has a relatively large amount of overhead to take advantage of a lot of unreliable, incomplete sources. This is really great when you're in that situation - for example, grabbing the latest isos and DVD rips - but it's kind of useless when you have a small number of fast sources (like Archive's servers) and basically nothing else.

The reason you have basically nothing else with Archive is that, unlike new releases to The Pirate Bay or something, everyone who might be interested in stuff released a long time ago is going to be pretty evenly distributed through time. This means that the most useful feature of Bittorrent - the ability to productively connect swarms of unreliable peers - isn't good for anything since nobody's downloading at the same time.
posted by 23 at 8:45 PM on August 8, 2012 [7 favorites]


it's going to take some major changes before we see any tangible benefits from this, no?

My impression is that the move is about bolstering the perceived legitimacy of bittorrent as a communications medium as much as it's about anything else.

In which case, "tangible" wouldn't come into it.
posted by Egg Shen at 9:02 PM on August 8, 2012


Egg Shen: i disagree. even if .1% of the collection is popular the actual bandwidth of serving up these could be a substantial fraction of IA's bandwidth. they dont actually have a huge web infrastructure---they serve their pages out of a slightly restyled Christian Science church in sanfran.... i think they said they dont even have a colo anywhere.
posted by dongolier at 9:44 PM on August 8, 2012 [2 favorites]


Hey, I found a show I was at! Neat.
posted by hypersloth at 9:59 PM on August 8, 2012


The inherent technology of torrents is disruptive, and this freaks people out. If I want milk, there should be a milkman who brings a bottle of milk to me. It doesn't make sense that my request for milk is met through a distributed process, with carbohydrates arriving from Australia, calcium from the White Cliffs of Dover, Alpha lactalbumin from the choicest sources in Sweden...

That just doesn't make sense. I want a product, some kind of object, that I can exchange for my dollars. I don't know what it means if I have to do that exchange with a zillion infinitesimal components from all over the planet.

People haven't quite evolved an appropriate sense of object recognition, of gestalt, for the myriad products of the modern age, because the objects of consumption are increasingly spread out around the world.
posted by twoleftfeet at 10:07 PM on August 8, 2012 [2 favorites]


whoa, sa-weeeeet - 42 Butthole Surfers concerts!!
posted by mannequito at 10:10 PM on August 8, 2012


My impression is that the move is about bolstering the perceived legitimacy of bittorrent as a communications medium as much as it's about anything else.

Definitely a worthwhile goal, although this still seems so "fringe-y" that there are going to be 3 likely scenarios, of which only 2 are likely in 99% of cases:

1. A person who is both familiar with bittorrent and is looking for some obscure gems as can be found at Archive,

2. A person who is not familiar, but has an interest in obscure audio/video, and simply opts for the direct download

3. A person who is not familiar, but for some inexplicable reason decides on the (probably) slower distributed download

The latter would, of course, be a rare case.

There have been times when software distribution wherein a large number of people were compelled to download a large file near-simultaneously (i.e. World of Warcraft updates) and torrent-style distribution was both forced on customers, as well as being a simple one-click endeavor on the part of the consumer. From what I recall, a lot of people reacted to this the same way they react to any kind of socialism, and it took a lot of time before anyone made peace with the whole idea of "the more you share, the more everyone benefits". But the bottom line was: there was no other way to distribute such a large download to so many users at the same time.

If Archive succeeds in spreading that philosophy, it could be a great thing indeed. Of course, first they'll have to shut down all direct downloads and enforce torrenting, along with maybe partnering with a software developer to create a dummy-proof torrent client program.

I think a real turning point will be when Apple or Microsoft begins distributing OS upgrades (not just updates, but genuine 8GB upgrades) via torrenting, which seems inevitable in the near future.
posted by ShutterBun at 10:25 PM on August 8, 2012 [2 favorites]


ShutterBun: They do mention they're using multiple servers of their own for BitTorrent, so it should presumably be the faster download method even without outside cooperation.

The real turning point is not OS upgrades (which would be done without people being aware of it), but when standard browsers integrate torrent clients (like Opera did years ago). Once you make it so clicking "magnet link" just works everyone will be happy.
posted by 23 at 10:29 PM on August 8, 2012


Again though, with torrenting, unless the user has a client which specifically targets segments sequentially (I know there are some premium clients out there that will do this, but not sure about free ones) but in general, the nature of torrenting is to distribute things so that "you get what everyone else needs, and vice versa" and since video and audio files rely on all the data being in order, you generally won't be able to enjoy your movie until it is complete.

Even if Archive is using its own bandwidth to seed torrents, it wouldn't be any faster than simply clicking on a direct download link (unless they start throttling those) and a direct download file is viewable (by several viewer applications) as they are in progress.

Ideally, a Limewire style system, whereby files are distributed, but the pieces are downloaded in (relative) sequence could be implimented, or more torrent clients could be configured to behave that way.

The real turning point is not OS upgrades (which would be done without people being aware of it)

Again, I am not talking about "automatic updates", but instead paid upgrades such as "OSX Liger" or whatever. Apple seems committed to eliminating boxed software, so it would definitely be interesting if they implemented some kind of torrenting in order to improve performance with their App store for large software distributions.
posted by ShutterBun at 11:13 PM on August 8, 2012 [2 favorites]


if they implemented some kind of torrenting in order to improve performance with their App store for large software distributions.

Microsoft at least has dipped its toe in the water with respect to aspects of P2P distribution of software. See BITS, peer distribution and BranchCache. However, all of these techniques are designed for environments where you trust your peers and want to conserve external bandwidth (i.e. on the same corporate network)

The fundamental problem is that P2P doesn't really make sense for premium digital distribution when the networks that most clients connect through are asymmetric and the economics is geared against outbound transfer. The available bandwidth between my client device and either a data-center or a CDN or even a cache in my ISP is always going to be far, far faster and more reliable than piecemeal transfer from hundreds of peers who most likely aren't on the same network as me and probably don't have outbound bandwidth that can compete with a data-center. I really can't overemphasize how much more efficient it is (in the current Internet) for most clients to connect to a data-center in Virginia or Oregon than it is for them to establish 20-odd TCP connections to a bunch of random machines connected through various ISPs with varying upload speeds and congestions issues to deal with.

It's a lazy argument on my part, but if BitTorrent really made sense from an economic, reliability or performance stand-point then I'm pretty sure we'd already see Netflix, YouTube, iTunes and Windows Update using P2P transfer for the content they distribute. However, what we see is companies like Netflix distributing their content through backbone providers like Level 3 and from CDNs within ISPs. There really isn't much incentive to distribute content through P2P right now.
posted by rh at 12:16 AM on August 9, 2012 [1 favorite]


The distributors favor streaming because they don't want to let people download actual copies. I'd signup for Netflix but slower but with actually downloading the content in a second.
posted by furiousxgeorge at 12:19 AM on August 9, 2012


ShutterBun, uTorrent (at the very least) has what it calls a "streaming" mode to download blocks sequentially. Many shady torrent clients (BitComet, for example) have this as a feature as well; some other big ones don't because there's a feeling it screws up the Bittorrent ecosystem. It's not hard to get if you want it, and then is functionally the same as a direct download from the perspective of players. Also, I've never heard of premium torrent clients. Is that a thing?

I assumed torrenting from Archive would be faster than a direct download simply because then you can pull from more than one of their servers at once.

Even with large, OS level upgrades, my impression is that that would be achieved in a manner that never put the word "torrent" in front of the user - you wouldn't fire up uTorrent and use a magnet link, you'd run "MacOSX Version Updater" without being told it's using BitTorrent in the background. That will be good for BitTorrent as a technical and legitimizing step, but it wouldn't be particularly helpful to the BitTorrent ecosystem of existing clients and legal contents (which the Archive setup seems closer to).
posted by 23 at 12:19 AM on August 9, 2012




(feeling an unsettling temptation to run away to San Francisco and sit on the doorstep of the Internet Archive until they let me in as a new apprentice/monk.)
posted by kaibutsu at 12:55 AM on August 9, 2012 [1 favorite]


Also, I've never heard of premium torrent clients. Is that a thing?

Vuze (Azureus) offers "watch now" (i.e. streaming mode) under its "Vuze Plus" client, which involves some kind of payment. I should have probably suspected that clients like uTorrent would offer this as a standard feature, though.

Granted, most torrent content I tend to encounter seems to fall under two categories: "enough seeders to download a 700mb file in 10 minutes", or "so few seeders that attempting to 'watch now' would undoubtedly result in tons of pauses and hiccups.'"
posted by ShutterBun at 1:28 AM on August 9, 2012


Even with large, OS level upgrades, my impression is that that would be achieved in a manner that never put the word "torrent" in front of the user

Probably not. Blizzard's Patch Downloader is little more than a proprietary bittorrent client, so I suspect that other major companies would incorporate something similar, being careful to avoid the stigma of the whole "bittorrent = piracy" thing.
posted by ShutterBun at 1:34 AM on August 9, 2012


"With The Pirate Bay under fire, and Demonoid taken offline and gift-wrapped for the US, the Internet Archive better be careful."

Gizmodo really do suck compared to the Verge these days. That piece is both based around a misunderstanding of the technology and is barely coherent beyond that.
posted by jaduncan at 4:18 AM on August 9, 2012


This is awesome, because I only recently realized how much music is on IA. (I'd been told, but didn't believe.) Much of it is, of course, terrible. But then much of commercial music is also terrible.

Right now, I'm listening to Wakka Chikka Wakka Chikka: Porn Music For The Masses, which is hilarious and awesome.
posted by DU at 4:30 AM on August 9, 2012 [1 favorite]


Gizmodo really do suck compared to the Verge these days. That piece is both based around a misunderstanding of the technology and is barely coherent beyond that.

It's a straight rip from TorrentFreak. How is it wrong? Demonoid is offline, and the servers (and I assume the database) have apparently been over to the US Government.
posted by Mezentian at 5:09 AM on August 9, 2012


How is it wrong?

Does IA contain anything even the most zealous copyright defender cares about?
posted by DU at 5:52 AM on August 9, 2012


My mistake, I was thinking about a different article.

That said, a href="http://yro.slashdot.org/story/12/08/09/007212/legitimate-ebook-lending-community-closed-after-copyright-complaints">over-zealous copyright defenders are everywhere.
posted by Mezentian at 7:05 AM on August 9, 2012


Does IA contain anything even the most zealous copyright defender cares about?

You might be surprised.
posted by Egg Shen at 7:16 AM on August 9, 2012


The archive.org FAQ has a bunch of technical detail on how this works. The most interesting part to me is they're using Webseeding; the torrents include HTTP URLs for the media. In fact, they just point to archive.org's existing HTTP servers. It's a clever bit of opportunistic serving; they didn't have to do much work to their existing setup to add BitTorrent support and some P2P bandwidth. Clever!
posted by Nelson at 10:15 AM on August 9, 2012 [1 favorite]


rh writes "It's a lazy argument on my part, but if BitTorrent really made sense from an economic, reliability or performance stand-point then I'm pretty sure we'd already see Netflix, YouTube, iTunes and Windows Update using P2P transfer for the content they distribute. However, what we see is companies like Netflix distributing their content through backbone providers like Level 3 and from CDNs within ISPs. There really isn't much incentive to distribute content through P2P right now."

Seems that the only thing stopping netflix using bit torrent is the fact that the clients would then need copies of the content and of course the owners aren't on board with that. Otherwise in parallel with the current system it would pretty well be a win for everyone.
posted by Mitheral at 7:50 PM on August 12, 2012


« Older The top ten new species 2012   |   Huge, Creaky Applications Newer »


This thread has been archived and is closed to new comments