DDoS on SoCal Time Warner

February 28, 2009 9:42 AM

It seems that there is a large DDoS attack targeting Time Warner's DNS servers in Southern California. JeffTWC's latest statement. A solution for those affected?

Previous DNS posts, especially this one, highlight a dangerous weakness of the current implementation of the internet: DNS Servers.

An alternate location for OpenDNS implementation instructions.

posted by wayofthedodo (46 comments total)

Wow, thanks for this. We've indeed had horrible Internet service here in LA. It's been so bad that we've started searching for alternatives to Time Warner. Now that I know it's an attack and not just sheer incompetence (always a good guess with TWC) I'll give them another week or so before I seriously consider bailing.
posted by Bookhouse at 9:48 AM on February 28

I so wanted the "solution" to be a link to a page that just said, "GO OUTSIDE!"

Then I realized that I've been online and unproductive all day, and I felt kind of silly.
posted by Navelgazer at 9:48 AM on February 28

The PoRN. It is so SLoW. HoPE me.
posted by hal9k at 10:26 AM on February 28 [1 favorite]

I'm a happy openDNS user, but adopters should be aware that there are potential privacy concerns. The OpenDNS folks will be able to see all the domain names for the sites that you're visiting; one of their optional add-on services is adult site blocking at the network level. They seem to be trustworthy (and are probably more trustworthy than the local cable company), but it is probably worth looking at the wikipedia discussion of openDNS's privacy issues before switching everything over to them.
posted by jenkinsEar at 10:26 AM on February 28

Shit, man, I wondered what was going on but too lazy to find out-- I'm at the same crossroads as Bookhouse. Thanks for this.
posted by hifiparasol at 10:44 AM on February 28

Count me also as noting erratic DNS issues.
posted by drpynchon at 10:50 AM on February 28

When this first started, I called up Time Warner and they sent someone out, who ended up replacing my cable modem. When it stopped working again later that night, I called them up again and by this time there was a message stating they were working on fixing the problem. I guess they aren't too good about keeping their repairmen informed. It seems to be working today, but it's gone in and out over the past week so who knows how long it'll last.
posted by fishmasta at 10:54 AM on February 28

*red light starts flashing, alarm starts ringing*

"Hackers!"

*shakes fist*
posted by The Card Cheat at 10:56 AM on February 28

The PoRN. It is so SLoW. HoPE me.

Am I the only one who heard this in Torgo's voice?
posted by EarBucket at 11:07 AM on February 28 [1 favorite]

I recommend trying OpenDNS before bailing. DNS is a problem for most ISPs.
posted by Argyle at 11:08 AM on February 28

Time Warner!! When I called them about this, their response was - no shit - "Our records indicate that you have had no interruption of service in the last 12 days!"

Meanwhile, I'm staring at the eleventy-billionth "page cannot be displayed" error of the day.

HATE!

/tries decaf
posted by Space Kitty at 11:10 AM on February 28

The Card Cheat: "*red light starts flashing, alarm starts ringing*

"Hackers!"

*shakes fist*"

"They've penetrated our code walls. They're stealing the internet!"
posted by PontifexPrimus at 11:41 AM on February 28

In San Diego time warner's DNS servers have been extremely unreliable since the second week of November. I've been relying on OpenDNS for the past 3 months. I'm surprised it hasn't gotten more press, but its good to see them acknowledging something.
posted by ethansr at 12:00 PM on February 28

Is this only affecting LA? I haven't had any problems this week in San Diego, and I've had Transmission running pretty much constantly the past couple of days.
posted by Thoughtcrime at 12:00 PM on February 28

Ah, I guess not. And I guess DNS has nothing to do with bittorrent. Never mind me. Haven't noticed any problems loading web pages either, though.
posted by Thoughtcrime at 12:01 PM on February 28

I think it's really misleading to say that this is an attack on the DNS "system". This is an attack on a specific set of caching recursive resolvers, NOT on any authoritative hosts. It is like saying that an attack on hotmail is an attack on "the email system", when you can use any of a large number of alternative email services or even run your own server.

And that brings me to my second point: I hate seeing these debates presented as the false dilemma of "whose DNS server should I use"? There is nothing stopping you from running your own DNS recursive resolver on your own local machine, making you completely isolated from any of the bullshit that your ISP's DNS servers might bring. You don't need a separate machine, and you don't need linux, as ISC has hancy bind binaries for Win32. (And OS X has them in fink, etc.)
posted by Rhomboid at 12:07 PM on February 28 [1 favorite]

"I recommend trying OpenDNS before bailing."

I recommend reading the entire post before commenting.
posted by mr_crash_davis mark II: Jazz Odyssey at 12:21 PM on February 28 [1 favorite]

I recommend smearing spinach paste liberally on one's genitalia before noon.
posted by Hovercraft Eel at 12:33 PM on February 28 [3 favorites]

I don't suggest using OpenDNS at all; they massage the results they return for their own benefit, which interferes with software that depends on DNS servers doing what they're supposed to do. (ie, return NXDOMAIN for domains that don't exist, instead of redirecting you to an ad-laden search page.)

Use one or two of the servers from 4.2.2.1 to 4.2.2.6. These are open DNS servers that should be highly resistant to DOS attacks. They're free and, as far as I know, operate correctly.

I'm not sure who provides these servers or why, but they've been up for a long time.
posted by Malor at 12:40 PM on February 28 [2 favorites]

Hovercraft Eel, that pretty much goes without saying.
posted by mr_crash_davis mark II: Jazz Odyssey at 12:40 PM on February 28

They are run by Level3 and you should consider avoiding them, because they will eventually be locked down to non-L3 customers (not to mention the possibility that they've been logging, datamining, and profiting from the traffic the whole time.)
posted by Rhomboid at 12:50 PM on February 28

I recommend smearing spinach paste liberally on one's genitalia before noon.

*checks watch*
Awww
posted by graventy at 12:51 PM on February 28

Yeah, OpenDNS sucks balls. I'm continually surprised how many techie types who should know better evangelize it. Their business model is to serve you ads when you mistype a domain, and they disallow you from blocking their ad servers at the DNS level like you can with any other site. Also, one of the higher-ups has a Kibo-like ability to show up in any thread on any forum and yell at people who dare to criticize his wonderful ad-serving company. If MetaFilter didn't cost $5 to join, he'd probably show up here an hour after I post this. And who knows what they're using the record of every website you access for? Probably to serve you more ads. Just use your ISP's DNS servers, they're fine 99% of the time. Or acquire a junk PC, install Linux and djbdns, and run your own DNS server if you really want to.
posted by DecemberBoy at 1:09 PM on February 28

While they are being lauded for their "openness" in disclosing the cause of the problem, it is entirely possible they are just lying. My experiences with Pacbell, Adelphia, Time Warner, Roadrunner, Comcast, SBC, and AT&T don't give me much confidence in their ability or willingness to find and fix a problem or in their willingness or ability to be genuinely candid about it.
posted by Xoebe at 1:14 PM on February 28 [2 favorites]

OpenDNS is not Open, not DNS, and does not have faster servers than most ISPs.

DNS latency is pretty hard to fuck up in software — the real problem with most ISP DNS servers is your route to them, and Level3 is fucking ace in that regard. I did some benchmarks yesterday, with some surprising results.

What's especially lulzy about OpenDNS is that they've successfully marketed the shit 'nerds' hate right back to them. The Cory Doctorows of the world will rail against Verizon and in the same sentence recommend fucklers doing the same shit masquerading as 'open'.
posted by blasdelf at 1:27 PM on February 28

I'm astonished that there's a DDoS against Time Warner. What would be the motivation?

I'd recommend people use pretty much any DNS server other than OpenDNS. What OpenDNS is doing is wrong and dangerous. You can freeload off of pretty much any DNS server out there though, very few filter traffic.
posted by Nelson at 1:47 PM on February 28

I also was surprised there was a DDoS against TWC's recursive resolvers. What is the point of this? If anyone can out-capacity a DDoS attack, it has to be a network company like TWC. Anyone want to shed light on what the attacks are trying to accomplish, why go after this?
posted by geoff. at 1:57 PM on February 28

Yeah, OpenDNS sucks balls. I'm continually surprised how many techie types who should know better evangelize it.

Indeed. Returning incorrect DNS results for your own benefit is pretty solidly in black-hat territory.
posted by oaf at 1:59 PM on February 28

Is bittorrent traffic either 1) increasing the traffic to the point where TWC cannot adequately defend against DDoS? 2) encouraging TWC to masquerade a general traffic of service as DDoS in order to deny service to bittorrenters 3) a topic which would only be brought into the conversation by conspiracy lovers such as yours truly?

Secondly, is this DDoS attack against TWC DNC going to the new wave of internet worm? Is it already really really popular? So is there a solution to DNS server vulnerability which is end-user implementable? or is the customer SOL unless they're a business willing to pay for a huge pipeline to the DNS? And is pay-for-play DNS access a new way to limit net neutrality?

Or once again, too much conspiracy?
posted by wayofthedodo at 2:00 PM on February 28

*facepalm*
posted by Rhomboid at 2:05 PM on February 28

Add a "me too, Bob" to the chorus of TWC haters. (Semi-long story to follow.)

A couple of months ago, I had an internet outage, and called TWC to ask what the problem was -- and got a recording telling me they knew there was an internet problem, and that they were taking care of it. So I turned off the computer and went outside for a while -- and there was a cable truck just outside my apartment building. The guy in the truck told me that the outage covered my whole area (however big that is) and that they'd have it fixed in about two or three hours.

So I got home about three hours later and tried to log on -- nothing at first, but after about half an hour everything worked OK. So the next day I called TWC to get a refund on my bill -- and the CSR on the other end spent ten minutes "looking to see if there were any outages" in my area. Finally I got his supervisor on the line, who told me that it was standard procedure to look at outage history if someone asks for a refund, and finally (after I turned off the honey and turned on the vinegar) just gave me the refund, which wound up being less than a dollar.

So TWC knew enough about the problem to set up a phone message AND to send out a truck (probably several), yet they had no record of any outage.

Seriously, what a completely fucked up system they must have.
posted by hifiparasol at 2:25 PM on February 28

Hovercraft Eel wrote: I recommend smearing spinach paste liberally on one's genitalia before noon.

Hello. You seem to have mistyped a URL. I believe the site you are looking for is called 'kuro5hin.'
posted by wierdo at 2:31 PM on February 28

Is there something like a trustworthy list of DNS servers we can use, preferably with geographical location so one can choose by proximity? Down here in Argentina the servers tend to suck badly, and I had reluctantly set up OpenDNS after being fed up with failures on resolving even the BBC News site an annoying number of times per week. Failing that, is that ISC software hard to set up and/or resource intensive? Is some other alternative preferred, for those who have tried doing it in your own everyday computer?
posted by Iosephus at 3:30 PM on February 28

My Comcast DNSs stopped working for a couple of hours last night. They'd respond to pings but wouldn't do DNS lookups.

I wonder if it was the same lot of black-hats.
posted by Chocolate Pickle at 3:46 PM on February 28

Really? So far my Comcast service is unaff
posted by Horace Rumpole at 3:55 PM on February 28 [2 favorites]

Iosephus: Route locality is much more important than geographical locality, and they do not map directly at all. The speed of light in copper & glass is really fast, circumnavigating the globe in ~200ms. Routing packets over Comcast's shitty network from WA to GA and back often takes longer.

Don't trust geographical location to be useful, or even other people's experiences via other ISPs. You pretty much have to test for yourself which servers will have lower latencies. traceroute is a great tool for figuring out what backbones you're connected through, and dig will let you find out how long a DNS query actually takes.

If you're going to run something on your own computer, don't run the full ISC Bind, a simple proxy like dnsmasq or dnscache would be more appropriate. I have dnsmasq running on my router, acting as a cache between my computers and Level3's servers.
posted by blasdelf at 4:10 PM on February 28 [1 favorite]

When used as a simple caching recursive resolver, bind has a memory footprint of around 5MB, and DNS is one of the lowest bandwidth internet protocols in use.

While I agree that using bind for a recursive resolver is a bit overkill, there are no native Win32 ports of DJB-ware. Besides, the whole point of this exercise is to make yourself independent from other people's DNS server problems, and just sticking dnsmasq in front of 4.2.2.1 does not accomplish that at all.
posted by Rhomboid at 4:17 PM on February 28

"I recommend reading the entire post before commenting."

I recommend reading the first comment before barfing out the usual weak snark.
posted by Argyle at 7:10 PM on February 28

All I can say is "yes", Comcast had problems and they seem to be in cahoots with TWC.
posted by zengargoyle at 8:57 PM on February 28

@fishmasta, @Space Kitty, all others who called in and got a conflicting message from call centers -- When did you call, and where are you located? If our phone reps aren't getting this news, we really need to know about it.

@wayofthedodo Thanks for posting this. For real. We need to get the message out, and this is going to help. But you may want to add a few more layers of foil to your hat -- judging by your comment re: BitTorrent traffic, I'd say some of the pink lasers from outer space are still getting through to your skull.
posted by jeffsimmermonTWC at 10:57 PM on February 28

jeffsimmermonTWC: Check your mefi mail for the details.
posted by fishmasta at 11:40 PM on February 28

jeffsimmermonTWC: wow, you're a member of this site? never in a million years did I expect that. amazement aside, what's the good word on the network status? What's going on? A blow-by-blow post on the attack, with gory nerd details would be awesome.
posted by wayofthedodo at 12:25 AM on March 1

@wayofthedodo: I've been a member of MeFi since 2002 or so under another username. Mathowie/Cortex/Jessamyn let me set this account up for work purposes only when I got this job.

I'll share what I can on Monday -- for now, things are quiet. Follow me on Twitter if you're not already, that's where I'm getting most of this stuff out.
posted by jeffsimmermonTWC at 6:52 AM on March 1

Anyone here noticing these kinds of issues with Bresnan Communications here in the mountain states?

Just asking because I have been experiencing these same kinds of issues for the past three and half weeks ... and had started doing some simple testing of my own ...
posted by aldus_manutius at 9:33 AM on March 1

@jeffsimmermonTWC - I'm in West Hollywood and we called a week ago. Color me impressed to see you replying here. (Is there anything MeFi can't do?)
posted by Space Kitty at 1:28 PM on March 2

For what it's worth, I've posted another update -- share far and wide, please: http://a.longreply.com/65742
posted by jeffsimmermonTWC at 8:09 PM on March 3