Downloaders delimma

December 9, 2012 10:40 AM

Long-time favorite usenet indexing site NzbMatrix has closed its site as part of a recent sweep of DMCA related takedowns on similar sites. Other recent shutdowns include Newzbin, Newzbin2, and NZBsRUS.

While primarily a blow to people making use of such software as Sickbeard, CouchPotato, Headphones, and Sabnzbd+to automate their media downloads, the indexing sites provided a quick, user-friendly way to find media.

NZBmatrix and similar sites make use of the .nzb format, originally associated with the site Newzbin (defunct)to automatically associate all individual postings required to make a complete file. Newsgroup posting are historically broken into dozens-to-hundreds of individual files, often only 1.44Mb each, from back in the days when portable storage was limited to what could be found on a HD Floppy disk.

Some usenet users have always felt the sites presented a danger to the usenet community at large, simply because their ease of use attracted "n00bs" to the service and made it easy for copyright holders to issue DMCA takedowns for illicit media.

Other indexing sites exist, and enterprising users can take it upon themselves to host their own indexes via LAMP-based software like Newznab.

Initial reports indicate that the Morganelli group may have played a role in the takedown, however theirtwitter feed says differently. That same twitter feed is currently being subjected to much hate from users of the offending websites. The Morganelli Group exists to aid copyright holders by monitoring "pirate" networks for copyrighted media and then disabling the files as soon as possible.

posted by TomMelee (50 comments total)

The first rule of usenet is you do not talk about usenet.
posted by palbo at 10:58 AM on December 9, 2012 [22 favorites]

Could we maybe have posts on awesome-but-unknown-to-me things BEFORE they shutdown?

Thanks in advance.
posted by DU at 11:03 AM on December 9, 2012 [15 favorites]

I'm not sure I'm ready or able to host my own index using Newznab. Perhaps I can do it, and perhaps it's easy, but there's so much I don't understand about indexes, and, indeed, much of the process. It's a shame, though, because when it works, it's phenomenal!
posted by newfers at 11:06 AM on December 9, 2012

Sadly, I haven't set-foot in Usenet since Comcast took-away included access years ago. I kind of miss it. I kind of don't.
posted by Thorzdad at 11:08 AM on December 9, 2012

.ftd and .nzb pretty much killed Usenet. Upload bots would use any and all newsgroups to drop files, which caused considerable grief to the human users of that newsgroup. Our posts would get flooded off the servers, leaving only their binaries. I used a pretty high retention server, but I frequently saw newsgroups that had about 2Gb of space get flooded with about 10Gb of posts. The uploader was killing off their own binaries by uploading too much, too fast. I could watch the segments upload and then roll off the server and expire. We called this "carpet bombing" because it totally devastated any newsgroup that it happened to, making it completely unusable.
posted by charlie don't surf at 11:13 AM on December 9, 2012 [5 favorites]

Haven't checked in on Usenet in nearly 7 years. Are any of the big 8 still in use or did everyone eventually jump ship to web-based forums?

I imagine the alt.* heiarchy is mostly binaries now, whether the group wants to be or not based upon charlie don't surfs comment.
posted by wcfields at 11:18 AM on December 9, 2012

Well, son of a bitch.
posted by broken wheelchair at 11:18 AM on December 9, 2012 [1 favorite]

Nyg.Flfnqzva.Erpbirel still stands, unsullied.
posted by boo_radley at 11:26 AM on December 9, 2012 [3 favorites]

It feels like it did when Napster died bro.
posted by fullerine at 11:39 AM on December 9, 2012 [1 favorite]

I'm curious why "policing our indexing bots" was an "impossible task." It doesn't take very much to comply with a DMCA takedown in most cases. Is the problem that NzbMatrix didn't have a fully automated DMCA process? That the takedown request came in a format that would have been prohibitive to input? That NzbMatrix felt it needed to proactively police for reposts (not ordinarily required under the DMCA)? Something else?

Something doesn't fully add up; there are missing facts here.
posted by grimmelm at 11:43 AM on December 9, 2012 [1 favorite]

When I first set up sick beard I thought it was interesting that it used NZB indexes to look up binaries. I had been on usenet for over a decade and it was my first time using them.

Most of the big usenet providers now have a web frontend with a search interface that you can use to find files, and those files typically have already been automatically unpacked. It seemed kind of silly that sick beard had to interface with sabnzb to grab all the files and unpack them when there is already a working search and RSS interface to the full files.
posted by grizzly at 11:47 AM on December 9, 2012

I always thought usenet would survive, last man standing, the cockroach that survives armageddon. But they've really launched an all-out multifaceted campaign on it - nzb indexers closing, the elimination of PayPal and other payment options for usenet providers, automated DMCA take-downs on , massive amounts of spamming using identical filenames and viruses, it's really been amazing to watch. All within a few months.

I guess we can talk about it now.
posted by Auden at 11:47 AM on December 9, 2012 [1 favorite]

The interesting aspect of this is that most users of Usenet for content downloading are willing to pay monthly fees to newsgroup server providers, membership fees to index sites, license fees to software makers, and even dedicate money to special hardware to use just for downloading.

For many, the willingness to pay is not the issue, it's the unavailability of content outside the distribution windows. Sure there's a bunch of people that want something for free, like anything desirable.

It's unfortunate that most of the major content providers haven't been able to adopt a windowing strategy that's more flexible that what exists now. The windows have gotten a bit smaller, but you still run into hard to find problems where content isn't available digitally or only as full purchase.

When I worked inside one of the major media companies, I often discussed that when my children were little, I would have paid $50 to watch a new movie at home that was still in the theaters, because the costs of babysitting, movie tickets, food, etc. all far exceeded that. Of course, I was viewed as a heretic.

Making something unavailable, at any price, is a sure way to guarantee piracy.
posted by Argyle at 11:51 AM on December 9, 2012 [14 favorites]

Morganelli Group has such a shady-sounding description on their homepage. I wonder what they’re doing to ensure the preservation of the problem to which they are the solution?
posted by migurski at 11:54 AM on December 9, 2012 [1 favorite]

Also, with the relative ease of running a personal Newznab install, it's easy to see the formation of private indexes, going darknet style with a trusted group of friends.

Heck, it wouldn't be to hard to sell/give away Amazon AMIs with Newznab ready to run, needing only Usenet provider credentials...
posted by Argyle at 11:55 AM on December 9, 2012

the elimination of PayPal and other payment options for usenet providers

Hadn't heard about that. From November 21st:

Paypal Bans Usenet Providers Over Piracy Concerns

Just how far PayPal is prepared to go was explained to us by Putlocker. The UK-based company had its PayPal account frozen several months ago after it refused to allow the payment provider to snoop on files uploaded by its users.

“They basically wanted access to the backend to monitor all the files being uploaded, and listing all files of users if they wanted, regardless of the privacy setting that the user might have selected,” Putlocker told TorrentFreak.

“This is a complete invasion of privacy on PayPal’s part, as it’s none of their business what files users keep in their account. We have a solid abuse handling policy already, and we don’t feel a 3rd party company has any business snooping on our users,” the company added.

MediaFire, another large cyberlocker, also said it stopped accepting PayPal after the company was unable to reach an agreement with the payment company. While there are still file-hosting services and Usenet providers that accept PayPal payments, this number is expected to decline further in the months to come. It is clear that PayPal’s new policies are in part the result of the copyright lobby.
posted by mediareport at 12:07 PM on December 9, 2012 [1 favorite]

I specifically started using sick beard for Homeland ; I bought the first season on amazon, wanted to buy the second. I am someone who would completely pay for immediate access to selected shows....I'm just not subscribing to the whole crappy catalog.
posted by das_2099 at 12:11 PM on December 9, 2012 [1 favorite]

Well fuck.
posted by Sternmeyer at 12:13 PM on December 9, 2012

This was not how I wanted to spend the last day of not frigid weather for a week or more.
posted by wierdo at 12:24 PM on December 9, 2012

NZB piracy always seemed like it was gonna make usenet worthless eventually, I'm pretty surprised it took this long for law talkin' people to get excited about em. Back when Bittorrent was just a relatively-shitty-time-investment-ratio method to get Linux isos, I had NomadNews cranking down TV shows and Movies automatically and at great speed direct from my ISP. NZB and specifically CouchPotato/Sickbeard and the fact that anybody can buy a consumer class NAS which runs apps like some kind of automatic low power piracy appliance for me seemed like an ostentatious celebratory endzone dance, practically begging for this kind of takedown.

I do really want the networks to come around on this, because I'd love at this point to get some per-show subscriptions or catalog access deals, some of the television shows that my significant other enjoys are simply not in the internet TV piracy attention orbit, and I'm getting tired of learning new ways to steal TV.
posted by jarvitron at 12:30 PM on December 9, 2012 [2 favorites]

It surprises me more that courts are buying into the idea that you can go after mere indexers who ingest everything. To me, that would be no different than going after the usenet servers themselves.
posted by wierdo at 12:44 PM on December 9, 2012

I just finished configuring my newznab+ install about 5 minutes ago, started about 15 minutes after I posted this. If you've ever done anything with apache before, it's pretty simple. I dropped it on my Windows Server 2011 machine via Xampp.

I've never in my life seen a private community who is so freaking nice to noobs. Seriously, I spent the last hour in /msg with a random, unpaid member of the community who walked me through everything like I was a complete n00b. (I am a noob, just maybe not a complete noob.)

FWIW, The Sickbeard Index still works fine, and alternative sites are newznab providers whose api's you can inject into Sickbeard. We shall carry on...

I'd be happy to pitch in on a metafilter darknet on an amazon AMI if such things were ok to talk about---which I am led to believe they are not.
posted by TomMelee at 1:09 PM on December 9, 2012 [4 favorites]

> I'm curious why "policing our indexing bots" was an "impossible task." It doesn't take very much to comply with a DMCA takedown in most cases. Is the problem that NzbMatrix didn't have a fully automated DMCA process?

I assume it's because they knew that so much of the content they indexed is pirated that they wouldn't have a sustainable business after complying with every DMCA filing.
posted by ardgedee at 1:56 PM on December 9, 2012

I've never in my life seen a private community who is so freaking nice to noobs.

Obviously you weren't actually on Usenet.

That was my primary objection to ftd and nzb. It took the discussions out of Usenet and put them on a front end that nobody on Usenet could locate. So when our newsgroup got carpet bombed with off topic material, there was no way to contact the source and ask them to post elsewhere. It was all happening on some obscure IRC channel or a private forum or some place full of noobs that didn't give a shit that they were disrupting communities that had existed for many years before they came along.
posted by charlie don't surf at 2:08 PM on December 9, 2012 [2 favorites]

It's unfortunate that most of the major content providers haven't been able to adopt a windowing strategy that's more flexible that what exists now. The windows have gotten a bit smaller, but you still run into hard to find problems where content isn't available digitally or only as full purchase.

I'm patient enough that windowing doesn't matter at all to me. What I hate is that everything has to somehow be wrapped in shit. I know it sounds crude but frankly the things they do to stuff disgusts me.

Watch a movie you have bought ? First endure the shit: Commercials for other movies, accusations of theft, awful and annoying menu screens.

Want to watch a movie on Demand? Endure a menu UI designed by someone who hates people while commercials or some E! online commercial plays over and over with the same drivel for months on end. With xfinity on demand they can't even maintain consistency in their menu structure - some shows can only be found by search, others alphabetically and still others by network only. So hard to find and more commercial time! Weeee! Then once you find your show I hope you don't get interrupted because if you pause it will only stop for about 4 minutes. Then it kicks back to the menu with the commercial (at about 1.5X the volume of the show you were watching). Another 10 minutes and you are kicked out of the ondemand back to realtime cable. You better resume that day or your resume bookmark is gone. If it is pay per view you have just 24 hours from when you started to watch. So an emergency and then your job and you are out the cost of the view. Then there is the fact that they list the channels you don't get on your guide with no indicator that you don't get them until you select the channel. They also list things you can't access under the Free category on your menu.

With downloads I get no commericals, no threats, the ability to pause as long as I want and most critically the use of my preferred well designed user friendly media manager instead of some corporate cost minimized profit maximized crud.

So when you argue for them to sell it to you somehow I think you are underestimating how awful they will make it for you when they do.
posted by srboisvert at 2:10 PM on December 9, 2012 [8 favorites]

So if I understand correctly, newznab just cuts out the indexer middleman?

I can get on board with that.
posted by Lord_Pall at 2:16 PM on December 9, 2012

Newzbin and Nzbmatrix had indexes going back several years. That was one of the best reasons to pay them rather than run your own indexer. I don't have the disk space to let newznab do a 1500 day backfill on any popular group. Maybe some other folks do.
posted by wierdo at 2:22 PM on December 9, 2012

Just a few comments:
Obviously you weren't actually on Usenet

I get really perturbed by the notion that any given open network is a cool kids club where other people aren't welcome. I first hopped onto Usenet back in about 1992-1998 when I was learning how to be a phone phreak, back when research included Usenet or Gopher or both. We simply didn't turn our group into a binary distribution group and we had no problems, conversations continued as always, and as necessary we coded messages. We had sub groups and IRC's and all the other fun subsidiaries of an active usenet community. Fwiw, there are only about 400 or so active binary distribution newsgroups of the 50-odd-thousand active groups.

So if I understand correctly, newznab just cuts out the indexer middleman?
Yup, that's about the skinny. FWIW, pretty much all the heavy indexers are or were newznab powered, even the free version gives you an API option.

disk space on 1500 day backfill
I'm not backfilling 1500 days, but remember you're only backfilling headers, the binaries themselves reside on the servers. As I look at todays repository for 1000+ day automatic backfill across all categories, it's only something like 60Gb for all indexed groups.
posted by TomMelee at 3:52 PM on December 9, 2012 [1 favorite]

I always thought they were hand curated.. Shows what I know.

I'm fighting with Newznab in Wampp, but the more I dork with this, I think I'm going to have to convert this whole box over to ubuntu. I was running win7, sabnzbd, etc on a lowish power turion neo (hp n40l). I'm hoping it's got enough punch to run newznab+ as well...
posted by Lord_Pall at 4:23 PM on December 9, 2012

Obviously you weren't actually on Usenet

No, I mean just now on the setup channel. That's how the ftd and nzb world work, with a remote channel that is not directly connected to NNTP servers.
posted by charlie don't surf at 4:39 PM on December 9, 2012

Lord_Pall: "I always thought they were hand curated.. Shows what I know."

Newzbin was. Others were not.
posted by wierdo at 4:53 PM on December 9, 2012

Charlie---sorry I misunderstood you. I was on the IRC channel.

Lord_Pall, I have exactly the same box, HPN40L, it's had about 4 os's but I settled on Win Server 2011 a long time ago. Newznab is running just fine via XAMPP, although granted Xampp is really NOT for web-facing application, just local. I use this box for a ton of stuff. There are people succesfully running newznab+ off of a raspberri pi, although not exactly well.

There are a handfull of php.ini fixes you need to make, but that's about it.
posted by TomMelee at 5:14 PM on December 9, 2012

Scrapped wampp, went with xampp, and it was smooth sailing. Wamp is a piece of crap.

Got everything up and running now, waiting for backfill. Quite cool stuff.
posted by Lord_Pall at 7:31 PM on December 9, 2012

Just as I was about to fork out my $10, this week. I guess it's good that I hung on to that $10. Still ... fuck.
posted by Xere at 7:39 PM on December 9, 2012

Did NZBsRUS go down? I thought it was still up.
posted by meehawl at 9:23 PM on December 9, 2012

I think the funny thing here is that I, as a proud member of the Napster generation, have seen this same scenario play out, time after time after time after time. It's like a chapter from Genesis, with a couple dozen names that only a select few people really know about.

Or better yet, it's the exact opposite of the Dutch story of the little boy and the dyke, since we all know that there are going to be an infinite number of new cracks in that dam.

Anyway, it's just fun to see them plug individual holes, knowing that there are dozens more opening up new every day.
posted by Blue_Villain at 9:27 PM on December 9, 2012 [1 favorite]

I assume the next step is to go after the providers themselves.. They already DMCA takedown continually on giganews, but if you carry 2 providers, you can fill without trouble.

The localized index thing is fascinating. Other than taking down all of usenet, I don't see a solution.
posted by Lord_Pall at 9:32 PM on December 9, 2012

ANd this kind of thing always reminds me of the King of the Hill episode where they destroy a fireant mound with the lawnmower.. And wake up to hundreds of fireant piles the next day.
posted by Lord_Pall at 9:33 PM on December 9, 2012

meehawl - NZBsrus is gone.
posted by kuanes at 4:35 AM on December 10, 2012

Other than taking down all of usenet, I don't see a solution.

I wonder how the postponed "6 strikes" will play out if they extend it to usenet traffic.
posted by Enron Hubbard at 9:17 AM on December 10, 2012

So any usenet provider worth their salt uses SSL connections. Set up a random port, use SSL, and all they have is a stream of uuencoded text data going over the pipe. Maybe they can use the headers to figure out what it is, but I'm not sure what's possible, and what's boogeyman future stuff...
posted by Lord_Pall at 10:16 AM on December 10, 2012

I had just ditched my Astraweb subscription on Saturday. Went with a server based in the Netherlands and things seem to be hunkey-dorey, for now. Still getting a steady 5.7MBps (~45Mbits of a 50Mbit connection?) out of the European server vs. Astra's US servers.

Fortunately my web-accessible, no-frills, free, and nzb-supporting usenet search engine seems to be humming along, too.

I wonder how much resistance non-US based servers will (be able to) put up against DMCA notices?
posted by porpoise at 2:28 PM on December 10, 2012

So how hard is it to get newznab up and running on win7/64 for someone who's never installed or run apache etc?

And, for that matter, will it eat all the resources on a ~2007 C2D box with 4gb ram?
posted by ROU_Xenophobe at 6:11 AM on December 11, 2012

We're gonna need a bigger disk.
posted by jbroome at 9:37 AM on December 11, 2012

kuanes: "NZBsrus is gone."

What is this then?
posted by meehawl at 10:10 AM on December 11, 2012

Matrix, Reloaded.
posted by meehawl at 10:40 AM on December 11, 2012

ROU_Xenophobe: "And, for that matter, will it eat all the resources on a ~2007 C2D box with 4gb ram?"

Disk IO seems to be the most significant bottleneck. Damn database-driven software. :P
posted by wierdo at 1:16 PM on December 11, 2012

NZBZRUS is back up today. Interesting.

Newznab is not for noobs. If you've never edited php or niggled with server settings, I don't recommend it.

SpotWeb seems to be much easier/faster, but requires fiddling in Dutch until installed, when you can switch to another language.

And yea, keeping even 15 groups fully updated and propagated is pretty much a 20 hour a day intensive process, although it only eats as much ram as your php.ini will let it and ~50% processing power. There is no threaded option for windows servers, there is for *nix.
posted by TomMelee at 1:02 PM on December 12, 2012

It occurs to me that clever people might index usenet binaries, automatically generate nzbs, and then post those nzbs themselves to usenet.

Then someone else could index that index, generate an nzb for the day's nzbs, and post *that* to a different newsgroup.
posted by ROU_Xenophobe at 11:59 AM on December 13, 2012

That's basically what Spotweb does, fyi.
posted by TomMelee at 6:58 PM on December 13, 2012