Comments on: DDoS on SoCal Time Warner

DDoS on SoCal Time Warner

wayofthedodo — Sat, 28 Feb 2009 09:42:22 -0800

It seems that there is a large DDoS attack targeting Time Warner's DNS servers in Southern California. JeffTWC's latest statement. A solution for those affected?

Previous DNS posts, especially this one, highlight a dangerous weakness of the current implementation of the internet: DNS Servers. An alternate location for OpenDNS implementation instructions.

By: Bookhouse

Bookhouse — Sat, 28 Feb 2009 09:48:05 -0800

Wow, thanks for this. We've indeed had horrible Internet service here in LA. It's been so bad that we've started searching for alternatives to Time Warner. Now that I know it's an attack and not just sheer incompetence (always a good guess with TWC) I'll give them another week or so before I seriously consider bailing.

By: Navelgazer

Navelgazer — Sat, 28 Feb 2009 09:48:28 -0800

I so wanted the "solution" to be a link to a page that just said, "GO OUTSIDE!" Then I realized that I've been online and unproductive all day, and I felt kind of silly.

By: hal9k

hal9k — Sat, 28 Feb 2009 10:26:42 -0800

The PoRN. It is so SLoW. HoPE me.

By: jenkinsEar

jenkinsEar — Sat, 28 Feb 2009 10:26:42 -0800

I'm a happy openDNS user, but adopters should be aware that there are potential privacy concerns. The OpenDNS folks will be able to see all the domain names for the sites that you're visiting; one of their optional add-on services is adult site blocking at the network level. They seem to be trustworthy (and are probably more trustworthy than the local cable company), but it is probably worth looking at the wikipedia discussion of openDNS's privacy issues before switching everything over to them.

By: hifiparasol

hifiparasol — Sat, 28 Feb 2009 10:44:08 -0800

Shit, man, I wondered what was going on but too lazy to find out-- I'm at the same crossroads as Bookhouse. Thanks for this.

By: drpynchon

drpynchon — Sat, 28 Feb 2009 10:50:56 -0800

Count me also as noting erratic DNS issues.

By: fishmasta

fishmasta — Sat, 28 Feb 2009 10:54:44 -0800

When this first started, I called up Time Warner and they sent someone out, who ended up replacing my cable modem. When it stopped working again later that night, I called them up again and by this time there was a message stating they were working on fixing the problem. I guess they aren't too good about keeping their repairmen informed. It seems to be working today, but it's gone in and out over the past week so who knows how long it'll last.

By: The Card Cheat

The Card Cheat — Sat, 28 Feb 2009 10:56:35 -0800

*red light starts flashing, alarm starts ringing* "Hackers!" *shakes fist*

By: EarBucket

EarBucket — Sat, 28 Feb 2009 11:07:30 -0800

The PoRN. It is so SLoW. HoPE me. Am I the only one who heard this in Torgo's voice?

By: Argyle

Argyle — Sat, 28 Feb 2009 11:08:27 -0800

I recommend trying OpenDNS before bailing. DNS is a problem for most ISPs.

By: Space Kitty

Space Kitty — Sat, 28 Feb 2009 11:10:58 -0800

Time Warner!! When I called them about this, their response was - no shit - "Our records indicate that you have had no interruption of service in the last 12 days!" Meanwhile, I'm staring at the eleventy-billionth "page cannot be displayed" error of the day. HATE! /tries decaf

By: PontifexPrimus

PontifexPrimus — Sat, 28 Feb 2009 11:41:25 -0800

The Card Cheat: "*red light starts flashing, alarm starts ringing* "Hackers!" *shakes fist*" "They've penetrated our code walls. They're stealing the internet!"

By: ethansr

ethansr — Sat, 28 Feb 2009 12:00:04 -0800

In San Diego time warner's DNS servers have been extremely unreliable since the second week of November. I've been relying on OpenDNS for the past 3 months. I'm surprised it hasn't gotten more press, but its good to see them acknowledging something.

By: Thoughtcrime

Thoughtcrime — Sat, 28 Feb 2009 12:00:09 -0800

Is this only affecting LA? I haven't had any problems this week in San Diego, and I've had Transmission running pretty much constantly the past couple of days.

By: Thoughtcrime

Thoughtcrime — Sat, 28 Feb 2009 12:01:40 -0800

Ah, I guess not. And I guess DNS has nothing to do with bittorrent. Never mind me. Haven't noticed any problems loading web pages either, though.

By: Rhomboid

Rhomboid — Sat, 28 Feb 2009 12:07:58 -0800

I think it's really misleading to say that this is an attack on the DNS "system". This is an attack on a specific set of caching recursive resolvers, NOT on any authoritative hosts. It is like saying that an attack on hotmail is an attack on "the email system", when you can use any of a large number of alternative email services or even run your own server. And that brings me to my second point: I hate seeing these debates presented as the false dilemma of "whose DNS server should I use"? There is nothing stopping you from running your own DNS recursive resolver on your own local machine, making you completely isolated from any of the bullshit that your ISP's DNS servers might bring. You don't need a separate machine, and you don't need linux, as ISC has hancy bind binaries for Win32. (And OS X has them in fink, etc.)

By: mr_crash_davis mark II: Jazz Odyssey

mr_crash_davis mark II: Jazz Odyssey — Sat, 28 Feb 2009 12:21:20 -0800

"I recommend trying OpenDNS before bailing." I recommend reading the entire post before commenting.

By: Hovercraft Eel

Hovercraft Eel — Sat, 28 Feb 2009 12:33:24 -0800

I recommend smearing spinach paste liberally on one's genitalia before noon.

By: Malor

Malor — Sat, 28 Feb 2009 12:40:03 -0800

I don't suggest using OpenDNS at all; they massage the results they return for their own benefit, which interferes with software that depends on DNS servers doing what they're supposed to do. (ie, return NXDOMAIN for domains that don't exist, instead of redirecting you to an ad-laden search page.) Use one or two of the servers from 4.2.2.1 to 4.2.2.6. These are open DNS servers that should be highly resistant to DOS attacks. They're free and, as far as I know, operate correctly. I'm not sure who provides these servers or why, but they've been up for a long time.

By: mr_crash_davis mark II: Jazz Odyssey

mr_crash_davis mark II: Jazz Odyssey — Sat, 28 Feb 2009 12:40:05 -0800

Hovercraft Eel, that pretty much goes without saying.

By: Rhomboid

Rhomboid — Sat, 28 Feb 2009 12:50:26 -0800

They are run by Level3 and you should consider avoiding them, because they will eventually be locked down to non-L3 customers (not to mention the possibility that they've been logging, datamining, and profiting from the traffic the whole time.)

By: graventy

graventy — Sat, 28 Feb 2009 12:51:45 -0800

I recommend smearing spinach paste liberally on one's genitalia before noon. *checks watch* Awww

By: DecemberBoy

DecemberBoy — Sat, 28 Feb 2009 13:09:57 -0800

Yeah, OpenDNS sucks balls. I'm continually surprised how many techie types who should know better evangelize it. Their business model is to serve you ads when you mistype a domain, and they disallow you from blocking their ad servers at the DNS level like you can with any other site. Also, one of the higher-ups has a Kibo-like ability to show up in any thread on any forum and yell at people who dare to criticize his wonderful ad-serving company. If MetaFilter didn't cost $5 to join, he'd probably show up here an hour after I post this. And who knows what they're using the record of every website you access for? Probably to serve you more ads. Just use your ISP's DNS servers, they're fine 99% of the time. Or acquire a junk PC, install Linux and djbdns, and run your own DNS server if you really want to.

By: Xoebe

Xoebe — Sat, 28 Feb 2009 13:14:32 -0800

While they are being lauded for their "openness" in disclosing the cause of the problem, it is entirely possible they are just lying. My experiences with Pacbell, Adelphia, Time Warner, Roadrunner, Comcast, SBC, and AT&T don't give me much confidence in their ability or willingness to find and fix a problem or in their willingness or ability to be genuinely candid about it.

By: blasdelf

blasdelf — Sat, 28 Feb 2009 13:27:14 -0800

OpenDNS is not Open, not DNS, and does not have faster servers than most ISPs. DNS latency is pretty hard to fuck up in software — the real problem with most ISP DNS servers is your route to them, and Level3 is fucking ace in that regard. I did some benchmarks yesterday, with some surprising results. What's especially lulzy about OpenDNS is that they've successfully marketed the shit 'nerds' hate right back to them. The Cory Doctorows of the world will rail against Verizon and in the same sentence recommend fucklers doing the same shit masquerading as 'open'.

By: Nelson

Nelson — Sat, 28 Feb 2009 13:47:00 -0800

I'm astonished that there's a DDoS against Time Warner. What would be the motivation? I'd recommend people use pretty much any DNS server other than OpenDNS. What OpenDNS is doing is wrong and dangerous. You can freeload off of pretty much any DNS server out there though, very few filter traffic.

By: geoff.

geoff. — Sat, 28 Feb 2009 13:57:44 -0800

I also was surprised there was a DDoS against TWC's recursive resolvers. What is the point of this? If anyone can out-capacity a DDoS attack, it has to be a network company like TWC. Anyone want to shed light on what the attacks are trying to accomplish, why go after this?

By: oaf

oaf — Sat, 28 Feb 2009 13:59:16 -0800

Yeah, OpenDNS sucks balls. I'm continually surprised how many techie types who should know better evangelize it. Indeed. Returning incorrect DNS results for your own benefit is pretty solidly in black-hat territory.

By: wayofthedodo

wayofthedodo — Sat, 28 Feb 2009 14:00:50 -0800

Is bittorrent traffic either 1) increasing the traffic to the point where TWC cannot adequately defend against DDoS? 2) encouraging TWC to masquerade a general traffic of service as DDoS in order to deny service to bittorrenters 3) a topic which would only be brought into the conversation by conspiracy lovers such as yours truly? Secondly, is this DDoS attack against TWC DNC going to the new wave of internet worm? Is it already really really popular? So is there a solution to DNS server vulnerability which is end-user implementable? or is the customer SOL unless they're a business willing to pay for a huge pipeline to the DNS? And is pay-for-play DNS access a new way to limit net neutrality? Or once again, too much conspiracy?

By: Rhomboid

Rhomboid — Sat, 28 Feb 2009 14:05:06 -0800

*facepalm*

By: hifiparasol

hifiparasol — Sat, 28 Feb 2009 14:25:33 -0800

Add a "me too, Bob" to the chorus of TWC haters. (Semi-long story to follow.) A couple of months ago, I had an internet outage, and called TWC to ask what the problem was -- and got a recording telling me they knew there was an internet problem, and that they were taking care of it. So I turned off the computer and went outside for a while -- and there was a cable truck just outside my apartment building. The guy in the truck told me that the outage covered my whole area (however big that is) and that they'd have it fixed in about two or three hours. So I got home about three hours later and tried to log on -- nothing at first, but after about half an hour everything worked OK. So the next day I called TWC to get a refund on my bill -- and the CSR on the other end spent ten minutes "looking to see if there were any outages" in my area. Finally I got his supervisor on the line, who told me that it was standard procedure to look at outage history if someone asks for a refund, and finally (after I turned off the honey and turned on the vinegar) just gave me the refund, which wound up being less than a dollar. So TWC knew enough about the problem to set up a phone message AND to send out a truck (probably several), yet they had no record of any outage. Seriously, what a completely fucked up system they must have.

By: wierdo

wierdo — Sat, 28 Feb 2009 14:31:12 -0800

Hovercraft Eel wrote: I recommend smearing spinach paste liberally on one's genitalia before noon. Hello. You seem to have mistyped a URL. I believe the site you are looking for is called 'kuro5hin.'

By: Iosephus

Iosephus — Sat, 28 Feb 2009 15:30:08 -0800

Is there something like a trustworthy list of DNS servers we can use, preferably with geographical location so one can choose by proximity? Down here in Argentina the servers tend to suck badly, and I had reluctantly set up OpenDNS after being fed up with failures on resolving even the BBC News site an annoying number of times per week. Failing that, is that ISC software hard to set up and/or resource intensive? Is some other alternative preferred, for those who have tried doing it in your own everyday computer?

By: Chocolate Pickle

Chocolate Pickle — Sat, 28 Feb 2009 15:46:02 -0800

My Comcast DNSs stopped working for a couple of hours last night. They'd respond to pings but wouldn't do DNS lookups. I wonder if it was the same lot of black-hats.

By: Horace Rumpole

Horace Rumpole — Sat, 28 Feb 2009 15:55:38 -0800

Really? So far my Comcast service is unaff

By: blasdelf

blasdelf — Sat, 28 Feb 2009 16:10:46 -0800

Iosephus: Route locality is much more important than geographical locality, and they do not map directly at all. The speed of light in copper & glass is really fast, circumnavigating the globe in ~200ms. Routing packets over Comcast's shitty network from WA to GA and back often takes longer. Don't trust geographical location to be useful, or even other people's experiences via other ISPs. You pretty much have to test for yourself which servers will have lower latencies. traceroute is a great tool for figuring out what backbones you're connected through, and dig will let you find out how long a DNS query actually takes. If you're going to run something on your own computer, don't run the full ISC Bind, a simple proxy like dnsmasq or dnscache would be more appropriate. I have dnsmasq running on my router, acting as a cache between my computers and Level3's servers.

By: Rhomboid

Rhomboid — Sat, 28 Feb 2009 16:17:36 -0800

When used as a simple caching recursive resolver, bind has a memory footprint of around 5MB, and DNS is one of the lowest bandwidth internet protocols in use. While I agree that using bind for a recursive resolver is a bit overkill, there are no native Win32 ports of DJB-ware. Besides, the whole point of this exercise is to make yourself independent from other people's DNS server problems, and just sticking dnsmasq in front of 4.2.2.1 does not accomplish that at all.

By: Argyle

Argyle — Sat, 28 Feb 2009 19:10:39 -0800

"I recommend reading the entire post before commenting." I recommend reading the first comment before barfing out the usual weak snark.

By: zengargoyle

zengargoyle — Sat, 28 Feb 2009 20:57:58 -0800

All I can say is "yes", Comcast had problems and they seem to be in cahoots with TWC.

By: jeffsimmermonTWC

jeffsimmermonTWC — Sat, 28 Feb 2009 22:57:04 -0800

@fishmasta, @Space Kitty, all others who called in and got a conflicting message from call centers -- When did you call, and where are you located? If our phone reps aren't getting this news, we really need to know about it. @wayofthedodo Thanks for posting this. For real. We need to get the message out, and this is going to help. But you may want to add a few more layers of foil to your hat -- judging by your comment re: BitTorrent traffic, I'd say some of the pink lasers from outer space are still getting through to your skull.

By: fishmasta

fishmasta — Sat, 28 Feb 2009 23:40:50 -0800

jeffsimmermonTWC: Check your mefi mail for the details.

By: wayofthedodo

wayofthedodo — Sun, 01 Mar 2009 00:25:24 -0800

jeffsimmermonTWC: wow, you're a member of this site? never in a million years did I expect that. amazement aside, what's the good word on the network status? What's going on? A blow-by-blow post on the attack, with gory nerd details would be awesome.

By: jeffsimmermonTWC

jeffsimmermonTWC — Sun, 01 Mar 2009 06:52:26 -0800

@wayofthedodo: I've been a member of MeFi since 2002 or so under another username. Mathowie/Cortex/Jessamyn let me set this account up for work purposes only when I got this job. I'll share what I can on Monday -- for now, things are quiet. Follow me on Twitter if you're not already, that's where I'm getting most of this stuff out.

By: aldus_manutius

aldus_manutius — Sun, 01 Mar 2009 09:33:49 -0800

Anyone here noticing these kinds of issues with Bresnan Communications here in the mountain states? Just asking because I have been experiencing these same kinds of issues for the past three and half weeks ... and had started doing some simple testing of my own ...

By: Space Kitty

Space Kitty — Mon, 02 Mar 2009 13:28:39 -0800

@jeffsimmermonTWC - I'm in West Hollywood and we called a week ago. Color me impressed to see you replying here. (Is there anything MeFi can't do?)

By: jeffsimmermonTWC

jeffsimmermonTWC — Tue, 03 Mar 2009 20:09:28 -0800

For what it's worth, I've posted another update -- share far and wide, please: http://a.longreply.com/65742