Wave of Cyber Attacks Hit Major US Websites
October 21, 2016 10:08 AM   Subscribe

Many sites including Twitter, Shopify and Spotify suffering outage. USA Today and Fortune Magazine offer additional details. White House Press Secretary Josh Earnest said the Department of Homeland Security was “monitoring the situation" but that “at this point [we] don’t have any information about who may be responsible for this malicious activity.”
posted by Hermione Granger (265 comments total) 29 users marked this as a favorite
 


Me, this morning, on a loop: "Wow, Twitter's down! I should check Twitter to see what's happening! Gah!"
posted by Celsius1414 at 10:16 AM on October 21, 2016 [40 favorites]


Pinterest, Etsy, and Squarespace are also down.
posted by Hermione Granger at 10:17 AM on October 21, 2016 [1 favorite]


When so many popular sites are using one platform, taking out that platform can knock out a large part of the Web and cause a lot of disruption.

In this case it seems to be the Dyn DNS platform that's been attacked, but there are other central points of failure these days, notably AWS and CloudFlare. Yes, they're big, but so are the attacks.

The whole point of the internet is that it's a decentralised network - the current trend towards centralising hosting and infrastructure services on a few large 'cloud' companies goes against that and makes the network vulnerable to large-scale disruption.
posted by winterhill at 10:18 AM on October 21, 2016 [58 favorites]


Kickstarter was affected, too. I feel bad for anyone with a funding campaign at a critical moment.
posted by GameDesignerBen at 10:18 AM on October 21, 2016 [1 favorite]


Don't worry; for whatever problem is vexing you today, there's an App for that.
posted by armoir from antproof case at 10:18 AM on October 21, 2016 [2 favorites]


That helps. I could see Twitter on the desktop but not the app. And when I tried to go to status.twitter.com it was down.
posted by larrybob at 10:19 AM on October 21, 2016


I haven't heard this elaborated much, but I'm pretty confident that Clinton's campaign getting hacked will turn into one of the best things to happen to internet security in forty years.

The next administration will have come into office having survived an attempt to sabotage their campaign mostly through vanilla infosec failures, and will (hopefully) be invested in making sure that doesn't happen again.
posted by mhoye at 10:19 AM on October 21, 2016 [27 favorites]


Whatsapp seems down to for me at the moment. I'm happy MeFi is still up.
posted by motdiem2 at 10:20 AM on October 21, 2016 [2 favorites]


I just read the Fortune article and decided to follow them on twitter, it's not available right now. I was just on it earlier. This will be an interesting day.
posted by reedcourtneyj at 10:20 AM on October 21, 2016


I sometimes check out but even that isn't working.
posted by larrybob at 10:20 AM on October 21, 2016 [4 favorites]


Was the sarcasm too subtle?
(Maybe I can google how to do sarcasm better. Is teh Google still up?)
posted by armoir from antproof case at 10:21 AM on October 21, 2016 [1 favorite]


But seriously , what winter hill said.

The commercialization of the Internet has come full circle. We have made our bed, and we may very well burn in it.
posted by armoir from antproof case at 10:24 AM on October 21, 2016 [2 favorites]


Down.is is up if you want to check if a site is down larrybob
posted by motdiem2 at 10:25 AM on October 21, 2016 [3 favorites]


We've been dealing with Dynect's managed DNS issues all day long. ( strangely , the things I have on their standard DNS ( unmanaged ) haven't seemed to have any issues.

Half-jokingly, I suggested to my co-workers that it's Russian hackers warming up for election day.
posted by mikelieman at 10:27 AM on October 21, 2016 [3 favorites]


Twitter is working for me, but images are all broken. Github is also down.

Seems like there were two waves to this thing? One started early this morning and was resolved and now we're in the midst of the second wave?

Back in the day when there were DNS problems, I had a text file with IP addresses of my favorite sites. But nowadays, that doesn't seem to work for most sites anymore (present company excluded of course) I wonder why that is? Maybe the mitigation measures that big companies need to take against attacks like these means that having a single IP address to your site no longer makes sense?
posted by gwint at 10:27 AM on October 21, 2016 [3 favorites]


Your favorite website may have been up, but its developers were in emergency mode because they couldn't deploy due to Github being down.

Thankfully, it's also given developers a nice carte blanche excuse for why we're behind today. "Well, the whole internet is basically down at the moment, so..." ¯\_(ツ)_/¯
posted by theraflu at 10:27 AM on October 21, 2016 [7 favorites]


Google and Youtube are still up, Amazon was down briefly for me earlier, and eBay's image server is partially down. Tumblr works on desktop but is having outages on mobile. Parse appears to be up still, which is good because a lot of mobile apps still use that as a backend.

Ironically both downrightnow and downdetector were down about an hour ago, too!
posted by Hermione Granger at 10:28 AM on October 21, 2016 [1 favorite]


The commercialization of the Internet has come full circle. We have made our bed, and we may very well burn in it.

I think it's probably best that we avoid catastrophising. Not every modern problem is an existential threat
posted by howfar at 10:28 AM on October 21, 2016 [31 favorites]


The next administration will have come into office having survived an attempt to sabotage their campaign mostly through vanilla infosec failures, and will (hopefully) be invested in making sure that doesn't happen again.

Hopefully, there will be better password protection. Using passwords like

Runner4567
p@ssw0rd

doesn't give me confidence that internet security will a priority.
posted by Roentgen at 10:28 AM on October 21, 2016 [3 favorites]


I'm happy MeFi is still up.

I had a little trouble reaching it earlier today, but it resolved quickly.
posted by thelonius at 10:29 AM on October 21, 2016 [3 favorites]


Oh, and decentralize your DNS by using multiple providers:

When github comes back, here's a link to Netflix's tool:

https://github.com/Netflix/denominator
posted by mikelieman at 10:29 AM on October 21, 2016 [6 favorites]


MetaFilter won't load for me or let me post comments.
posted by radicalawyer at 10:29 AM on October 21, 2016 [36 favorites]


copies gwint's mefi ip adress into a txt file just in case
posted by motdiem2 at 10:30 AM on October 21, 2016 [1 favorite]


Decentralizing won't save you if you're on the receiving end of a big attack. Let's say you're Kickstarter. You host all your own infrastructure, etc. Then this hits you. How can you even start to respond? The big companies have response teams that are larger than your entire company by an order of magnitude. Big attacks drive people to services like AWS, because they become the problem of someone else who has a lot more technical expertise and resources than you do.
posted by phooky at 10:30 AM on October 21, 2016 [12 favorites]


(brb; busy painting a portrait of myself, face painted blue, standing atop of dead hackers, captioned, "THEY'LL NEVER TAKE MY SPOTIFY.")
posted by DirtyOldTown at 10:30 AM on October 21, 2016 [2 favorites]


I had a little trouble reaching it earlier today, but it resolved quickly.

We had a presumably coincidental run-in with a deeply over-excited scraper bot; frimble nuked it from orbit, load is back to late-election weekday normal.
posted by cortex at 10:30 AM on October 21, 2016 [36 favorites]


Is there any more detailed coverage of the attack yet?
posted by Joe Chip at 10:31 AM on October 21, 2016


Achewood was down earlier. And on a "Fuck You Friday."

This. Means. War.
posted by SansPoint at 10:31 AM on October 21, 2016 [11 favorites]


frimble is awesome.
posted by Melismata at 10:32 AM on October 21, 2016 [10 favorites]


You host all your own infrastructure, etc. Then this hits you. How can you even start to respond?

Call my upstream ISP? This whole thing reminds me of why I stopped compiling my own BIND.


https://www.dynstatus.com/
Update - This DDoS attack may also be impacting Dyn Managed DNS advanced services with possible delays in monitoring. Our Engineers are continuing to work on mitigating this issue.
Oct 21, 16:48 UTC
posted by mikelieman at 10:34 AM on October 21, 2016


as mentioned on hacker news, porn sites have better tech - better dns redundancy in this instance - than almost anyone else.
posted by Foci for Analysis at 10:34 AM on October 21, 2016 [4 favorites]


Is there any more detailed coverage of the attack yet?

Nothing from Schneier yet. Who do people follow lately for netsec stuff?
posted by gwint at 10:34 AM on October 21, 2016 [1 favorite]


Hopefully, there will be better password protection. Using passwords like

Runner4567
p@ssw0rd

doesn't give me confidence that internet security will a priority.


Any system that relies on users to come up with secure passwords is a broken system.
posted by straight at 10:35 AM on October 21, 2016 [5 favorites]


Is this sort of thing going to continue until Killary concedes the election?
posted by Flashman at 10:35 AM on October 21, 2016 [1 favorite]


I'm happy MeFi is still up.
...
I had a little trouble reaching it earlier today


Same; cat was sitting on my laptop
posted by Greg Nog at 10:35 AM on October 21, 2016 [52 favorites]


frimble nuked it from orbit, load is back to late-election weekday normal.

Invoke the frimble doctrine. It's the only way to be sure.
posted by mandolin conspiracy at 10:37 AM on October 21, 2016 [7 favorites]


The Russians try to help Trump's campaign by preventing him from tweeting.
posted by acb at 10:39 AM on October 21, 2016 [75 favorites]


If this is another internet-of-things attack maybe we can eventually get some regulation passed. The FCC regulates people that would pollute the airwaves with radio frequencies and "products that, by design, contain circuitry operating in the radio frequency range need to demonstrate FCC compliance". Seems like they could apply the same reasoning to devices that can disrupt cyberspace.
posted by dilaudid at 10:40 AM on October 21, 2016 [13 favorites]


Hmm...seems like someone (*cough* Putin!) doesn't want anyone reposting the sick burns Hilary dished out at the Alfred E Smith dinner last night...
On a lighter note, looking over the list of sites affected, apparently only the garbage parts of the internet are on fire.
posted by sexyrobot at 10:41 AM on October 21, 2016


This made my morning... interesting. I hope it doesn't fuck up my weekend.
posted by defenestration at 10:42 AM on October 21, 2016


The DDoS attacks will continue until morale improves.
posted by larrybob at 10:43 AM on October 21, 2016 [10 favorites]


This won't be an issue after Trump is elected.
posted by blue_beetle at 10:43 AM on October 21, 2016 [1 favorite]


Please for the love of God tell me Steam is ok. The kids are out of the house for the first time in months this weekend.
posted by selfnoise at 10:44 AM on October 21, 2016 [20 favorites]


This made my morning... interesting. I hope it doesn't fuck up my weekend.
posted by defenestration


Stay away from open windows, and you'll be fine.
posted by Celsius1414 at 10:44 AM on October 21, 2016 [11 favorites]


We share the same topology
Is it down for everyone or me?
Believe me when I say to you
I hope the Russians love their Netflix too
posted by RobotVoodooPower at 10:44 AM on October 21, 2016 [42 favorites]


oh god I'm going to have to clean my house this weekend, aren't I?
posted by AFABulous at 10:45 AM on October 21, 2016 [12 favorites]


@pbausch: "Good thing we all have those ham radio rigs so we can stay in touch post DDoS. It's been fun!"

Well someone's enjoying their retirement!
posted by gwint at 10:45 AM on October 21, 2016 [8 favorites]


@pbausch: "Good thing we all have those ham radio rigs so we can stay in touch post DDoS. It's been fun!"

Firefox can’t find the server at twitter.com.
posted by mikelieman at 10:47 AM on October 21, 2016 [1 favorite]


I hope the Russians love their Netflix too

"From Dream of the Blue Turtles All The Way Down"
posted by Celsius1414 at 10:47 AM on October 21, 2016 [5 favorites]


ok i've downloaded the frimble app but i can't find the button to nuke from orbit??? my twitter still doesn't work
posted by indubitable at 10:47 AM on October 21, 2016 [1 favorite]


PlayStation network appears to be affected too.
posted by skycrashesdown at 10:47 AM on October 21, 2016


Guys—nobody do any actual work right now.
If our bosses find out how much productivity improves when there's no internet, we're all DOOMED!
posted by Atom Eyes at 10:48 AM on October 21, 2016 [40 favorites]


On a lighter note, looking over the list of sites affected, apparently only the garbage parts of the internet are on fire.

Github, Okta, and pagerduty are garbage now?
(no source code for developers, workers won't be able to log into shared services, and if something else goes down or triggers an alarm no one will notice).
posted by MikeKD at 10:49 AM on October 21, 2016 [5 favorites]


Is Livejournal based in Russia now? It's still online.

not that I spend time on that site anymore, I mean. it's not like ohnotheydidnt is still hosted there or anything...
posted by Hermione Granger at 10:50 AM on October 21, 2016 [1 favorite]


Please for the love of God tell me Steam is ok.

Annnd Steam just announced Civ VI is available.
posted by Celsius1414 at 10:51 AM on October 21, 2016 [6 favorites]


die, internet, die! all praise to the heroic activity of whoever is responsible!
posted by ennui.bz at 10:51 AM on October 21, 2016 [1 favorite]


According to Cracked, LiveJournal is now mostly Russian, yeah (#3).
posted by Melismata at 10:52 AM on October 21, 2016 [1 favorite]


I know what I'm saying is a bit pathetic, but if this is a warm-up for a full-scale thing on election day, I imagine me and countless others wouldn't have the first clue of how to even get to a polling place.
posted by naju at 10:52 AM on October 21, 2016 [10 favorites]


Guys, guys, this is just the AR component of Black Mirror season 3.
posted by grumpybear69 at 10:52 AM on October 21, 2016 [14 favorites]


Is Level3 stuff affected? I'm on my way into work and wondering how many complaints we're going to have in the queue.
posted by mephron at 10:53 AM on October 21, 2016


Big attacks drive people to services like AWS, because they become the problem of someone else who has a lot more technical expertise and resources than you do.
With DNS it's possible to have your cake and eat it too. By all means use one or more large DNS providers for your site, but self-host your own servers too. When an attacker takes out the large DNS provider for maximum election-day impact, they won't bother with your own servers and your site stays up. And when an attacker targets you specifically, you can run to big provider for help and it doesn't much matter if your small servers fail.
posted by joeyh at 10:53 AM on October 21, 2016 [3 favorites]


Is Level3 stuff affected? I'm on my way into work and wondering how many complaints we're going to have in the queue.

Well, if your work email is like my work email, people TRYING to send in support@ tickets will probably be queued on their MTA, since the DNS lookup for our MX records failed.
posted by mikelieman at 10:55 AM on October 21, 2016 [1 favorite]


Level3 looks pretty bad right now.
posted by RobotVoodooPower at 10:58 AM on October 21, 2016


the real question is, why aren't there emergency alerts sent to my phone when Twitter goes down like there are for tornadoes and stuff? I thought your phone personalized stuff based on your habits (no I don't have a problem what are you talking about)
posted by AFABulous at 10:59 AM on October 21, 2016 [10 favorites]


Atom Eyes: "Guys—nobody do any actual work right now.
If our bosses find out how much productivity improves when there's no internet, we're all DOOMED!
"

Except that a lot our productivity requires npm, bower and a few other repositories that aren't currently up.
posted by octothorpe at 11:00 AM on October 21, 2016 [3 favorites]


FWIW, I'm subscribed to Dyn's text messages for updates to this issue and I haven't gotten squat through my phone... ( pagerduty perhaps? )
posted by mikelieman at 11:01 AM on October 21, 2016


I decided to take the day off before I knew about this. I can't decide if this is good timing or bad timing.
posted by aubilenon at 11:02 AM on October 21, 2016


Thankfully, it's also given developers a nice carte blanche excuse for why we're behind today. "Well, the whole internet is basically down at the moment, so..." ¯\_(ツ)_/¯

Related, vintage XKCD: The #1 programmer excuse for legitimately slacking off: "my code's compiling."

Developers don't get this excuse too often, seems more like a snow day delay.
posted by filthy light thief at 11:03 AM on October 21, 2016 [3 favorites]


I decided to take the day off before I knew about this. I can't decide if this is good timing or bad timing.

Spectating > Participating. I go off shift in < 30 minutes, then it's Someone Else's Problem until 6am Monday :)
posted by mikelieman at 11:04 AM on October 21, 2016


Obviously, the Russian hackers learned Trump was going to really go crazy on twitter today. Thank goodness for the automatic drafts function.
posted by Atreides at 11:04 AM on October 21, 2016


FWIW, I'm subscribed to Dyn's text messages for updates to this issue and I haven't gotten squat through my phone... ( pagerduty perhaps? )

I subscribed to their e-mails and am receiving updates.
posted by enn at 11:05 AM on October 21, 2016


Level3 looks pretty bad right now.

This image is framed and on the wall in our lab at work.
posted by Mister Fabulous at 11:05 AM on October 21, 2016 [11 favorites]


Anecdote from [not tech related or adjacent] work: a non-techie co-worker was kind of freaking out about "the internet" being down on the east coast. I checked MetaFilter (working fine) and found no news, so I looked elsewhere, saw the DNS DDoS news, and gave a quick description of what was happening for him. He seemed slightly calmed, but still said "this is the start of World War 3." (He's kind of excitable like that, though.)
posted by filthy light thief at 11:06 AM on October 21, 2016 [2 favorites]


I subscribed to their e-mails and am receiving updates.

Yeah, I thought I'd be smart and use all this technology. Didn't realize at the time their paging required a DNS lookup. Last time I ran a paging server, the modems were on localhost, and the hosts file took care of that.
posted by mikelieman at 11:08 AM on October 21, 2016


Augh, I just spent 20 minutes trying to fix my system when twitter (the only thing I use that I've noticed being affected) failed. It failed in three steps tricking me into thinking it was something at my end. First embedded videos wouldn't load I figured something changed that was triggering my flashblock/uBlock/Ghostery/uMatrix settings. While I was fiddling with those the script that formats the page stopped loading and I figured it was some local DNS issue. It wasn't until I flushed my cache on router and machine and nothing would load that I tweaked that it was a global issue.
posted by Mitheral at 11:13 AM on October 21, 2016 [1 favorite]


Ahh, sorry guys. My cat's been asleep on the enter button on my keyboard all morning. We should be good now.
posted by Slarty Bartfast at 11:18 AM on October 21, 2016 [32 favorites]


well here's to Git being a distributed VCS
posted by atoxyl at 11:19 AM on October 21, 2016 [8 favorites]


First embedded videos wouldn't load I figured something changed that was triggering my flashblock/uBlock/Ghostery/uMatrix settings.

I installed a Pi Hole (Raspberry Pi doing DNS based ad blocking) yesterday, and if this didn't begin after 7am with my boss slacking me about the production server disappearing, I would have spent a non-trivial amount of time screwing around trying to figure out what I did wrong.

Thank you G-d, for even the small favors.
posted by mikelieman at 11:20 AM on October 21, 2016 [3 favorites]


This is actually also affecting the main software package I use to process things where I work. It's been buggy on everyone all day - but even more infuriating, it's been dropping in and out for different people at different times.

The software is down for me right now, but not for my colleagues, so I'm sitting here looking like a slacker while they're working away.
posted by EmpressCallipygos at 11:21 AM on October 21, 2016


KrebsOnSecurity: The attack on DYN comes just hours after DYN researcher Doug Madory presented a talk on DDoS attacks in Dallas, Texas at a meeting of the North American Network Operators Group (NANOG).

While the cats are away...
posted by RobotVoodooPower at 11:21 AM on October 21, 2016 [4 favorites]


Second wave inbound, we're not out of it yet.
posted by CrystalDave at 11:22 AM on October 21, 2016


"Route around damage" was apparently interpreted as "consolidate the internet into 3 single companies."
posted by odinsdream at 11:23 AM on October 21, 2016 [13 favorites]


Our ops department changed up our PC security in a way that forced me to re-install npm, node, bower, and a bunch of packages I need a week ago.

I can't tell you how glad I am that they did that a week ago. And that we run our own source control server internally.
posted by middleclasstool at 11:25 AM on October 21, 2016 [1 favorite]


I'm watching devs share github IP addresses on IRC. What a future!
posted by Gerald Bostock at 11:27 AM on October 21, 2016 [16 favorites]


Okay, the main software thing at work has also now crashed for both my colleagues and my boss so I look like less of a stupid doof now.
posted by EmpressCallipygos at 11:30 AM on October 21, 2016


If you set your DNS on your network connection to use OpenDNS servers instead (208.67.222.222 and 208.67.220.220) that seems to work right now to let me get to GitHub -- they have a feature called smartcache that caches IP addresses for hosts when there's an outage (I can't get to that page right now but that's what an acquaintance told me).
posted by brainwane at 11:32 AM on October 21, 2016 [6 favorites]


Okay, the main software thing at work has also now crashed for both my colleagues and my boss so I look like less of a stupid doof now.

Heh. The series of surveys I needed to put together this afternoon using SurveyMonkey?

Not happening right now.

¯\_(ツ)_/¯

Force majeure!
posted by mandolin conspiracy at 11:33 AM on October 21, 2016 [1 favorite]


(You'd basically use the same instructions as Google provides for its "use our DNS server" service, but using 208.67.222.222 and 208.67.220.220 instead of 8.8.8.8 and 8.8.8.4.)
posted by brainwane at 11:33 AM on October 21, 2016


Second wave inbound, we're not out of it yet.


Red 5 standing by!
posted by Slarty Bartfast at 11:33 AM on October 21, 2016 [14 favorites]


Oh. Is this why I suddenly can't get on the major stock photo websites? I noticed twitter but thought it was my phone acting up. Shutterstock and Adobe stock are down for me but the "is this site up" sites say they're working.
posted by jeweled accumulation at 11:37 AM on October 21, 2016


Typepad's ability to automatically link new blog posts in your Twitter and Facebook feeds may be kaput for the moment.
posted by thomas j wise at 11:38 AM on October 21, 2016


Wookieepedia is down! Those monsters!
posted by UltraMorgnus at 11:39 AM on October 21, 2016 [11 favorites]


Gggghhhrahhhrrghhh!!!
posted by stevil at 11:40 AM on October 21, 2016 [3 favorites]


My team's SaaS app is up, but the third-party service we call to manage user subscriptions and payment is down, so no paid users can sign up for our app right now.
posted by freecellwizard at 11:40 AM on October 21, 2016 [1 favorite]


Have we tried turning it off and on again?
posted by RobotVoodooPower at 11:42 AM on October 21, 2016 [16 favorites]


so, hey kids, don't upload your stuff to the cloud and delete it from your hard drive. Box is down. *sad trombone*
posted by AFABulous at 11:57 AM on October 21, 2016 [6 favorites]


*sad trombone*

Host not found.
posted by mikelieman at 11:59 AM on October 21, 2016 [7 favorites]


Dreamwidth, Identi.ca, and GitLab (open source alternatives to Tumblr, Twitter and GitHub) are up and easily reachable.

One roundup mentions a list of affected sites that includes ActBlue, a Democratic-centric fundraising & political organizing site.
posted by brainwane at 12:05 PM on October 21, 2016 [2 favorites]


Y'know, Biden did say that Russia will face retaliation. Maybe he accidentally targeted the wrong country's infrastructure?

Or perhaps some powers-that-be needed to silence Assange or upcoming Project Veritas' material?

Or maybe I should have not clicked on that link promising me millions from Nigeria. Sorry, y'all. Didn't mean to break the internet. ¯\_(ツ)_/¯
posted by Conway at 12:09 PM on October 21, 2016 [2 favorites]


Wookieepedia is down! Those monsters!

wookieepedia.org is still up and it's better anyway.
posted by aubilenon at 12:10 PM on October 21, 2016 [6 favorites]


I remember at a former job adding a project on GitHub as a sub repository to one of our internal products and talking to the build server guy about it. "Isn't there some way to set it up to use a local git repo instead of pulling it from GitHub on every build?" I asked. "It's no big deal, GitHub will never go down," he replied.

I wonder what kind of day they're having today.
posted by indubitable at 12:13 PM on October 21, 2016 [6 favorites]


Sonuvabitch. Paypal is down from Canada, so I had to fall back to my alternate shipping procedure and I missed out on my discount. Whoever these bastards are, they just cost me $3.82 because I can't wait til Monday - the yarn must flow.

Shit just got real, yo.
posted by Mary Ellen Carter at 12:16 PM on October 21, 2016 [35 favorites]


Any other web devs unable to access heroku consoles or heroku.com?
posted by lastobelus at 12:17 PM on October 21, 2016


why does twitter not work inside my house on my wifi but it works at the dunky donuts and on the subway wifi. we all use twc.
posted by poffin boffin at 12:17 PM on October 21, 2016 [1 favorite]


honestly if you want to be like "it's elves, and they're bad" i will 100% believe you but i will want solutions to appease the bad elves
posted by poffin boffin at 12:18 PM on October 21, 2016 [17 favorites]


I am expecting UPS to deliver me a computer today. I am going to be extremely irritated if they decide they don't have enough internet to come through.
posted by bukvich at 12:18 PM on October 21, 2016


I am expecting UPS to deliver me a computer today. I am going to be extremely irritated if they decide they don't have enough internet to come through.

Well, its not like computers are worth anything without the internet anyways.
posted by pwnguin at 12:21 PM on October 21, 2016 [2 favorites]


Any other web devs unable to access heroku consoles or heroku.com?

Can't access it either. Heroku is down. My company and team rely on it for some things. Fun. :/
posted by defenestration at 12:22 PM on October 21, 2016


honestly if you want to be like "it's elves, and they're bad" i will 100% believe you but i will want solutions to appease the bad elves

Apparently donuts work.
posted by Mooski at 12:23 PM on October 21, 2016 [4 favorites]


honestly if you want to be like "it's elves, and they're bad" i will 100% believe you but i will want solutions to appease the bad elves

It's not the elves that are bad... ( PSA ) "These are elves"...
posted by mikelieman at 12:24 PM on October 21, 2016 [1 favorite]


Am I the only one who wishes stuff like this would happen more often so people would take the possibility of service outages seriously?
posted by shponglespore at 12:28 PM on October 21, 2016 [9 favorites]


Ugh. I thought we'd beat the worst of this at work, but our email service provider isn't able to send tests (or, likely, anything else) putting our jobs at a literal standstill. (I do e-marketing for performing arts now.)
posted by SansPoint at 12:31 PM on October 21, 2016


We're fucked.

I don't know enough about this kind of stuff at all, but it sounds like the vulnerability has been fairly obvious to people who do. Whatever we're doing to secure websites right now is the equivalent of a lambskin condom from what I can tell. This is why I refuse to back up my data to a cloud service. I've got multiple external hard drives; if I'm going to lose my data I'd rather it completely disappear due to the fragility of plastic, not because someone else has it. At least I'll still have my music files to console myself while Russia transfers all my money out of my bank accounts.
posted by good lorneing at 12:33 PM on October 21, 2016 [3 favorites]


Someone replace the bad elves with the good elves, someone get on this
posted by The Whelk at 12:33 PM on October 21, 2016 [3 favorites]


so like, machine elves? Be careful what you wish for.
posted by gwint at 12:35 PM on October 21, 2016 [2 favorites]


i will want solutions to appease the bad elves

Find a "smart" toaster/fridge/lightbulb/thermostat and beat it into dust. Repeat.
posted by enn at 12:39 PM on October 21, 2016 [6 favorites]


I know it's all scary and that, in a "we're all screwed" way, and a pain in the bum for tech people, but the first I knew was when asos wouldn't open and I'm thrilled that something has finally stopped my jumper-buying addiction in its tracks tbh
posted by billiebee at 12:40 PM on October 21, 2016 [5 favorites]


Someone replace the bad elves with the good elves, someone get on this
posted by The Whelk at 15:33 on October 21 [+] [!]

so like, machine elves? Be careful what you wish for.
posted by gwint at 15:35 on October 21 [+] [!]


Anyone know where we should file the DMTCA takedown request?
posted by zamboni at 12:43 PM on October 21, 2016 [2 favorites]


Can someone explain how Dyn went from being the free and pretty unreliable DNS service that I used to use for my home server, to being the primary DNS provider for dozens of major sites?
posted by miyabo at 12:45 PM on October 21, 2016 [3 favorites]


Find a "smart" toaster/fridge/lightbulb/thermostat and beat it into dust. Repeat

No. networked. Machines.
posted by The Whelk at 12:48 PM on October 21, 2016 [8 favorites]


The HN thread has some IPs you can add your hosts file if you can't wait this out. I'm going to go to bed and hope that by the morning all these toasters, fridges, cameras, lightbulbs, etc, have got bored and the people that own them start to have second thoughts about the internet of shit^W things.
posted by lawrencium at 12:50 PM on October 21, 2016 [1 favorite]


So is this Trump's 10 year old son, or the 400 lb guy on the bed?
posted by bendybendy at 12:55 PM on October 21, 2016 [13 favorites]


I can just ftp the latest HOSTS.TXT file from sri-nic.arpa, right?
posted by 1970s Antihero at 12:56 PM on October 21, 2016 [13 favorites]


Changed my phone to OpenDNS servers 208.67.222.222 and 208.67.220.220 and Twitter works again!!! Praise be to the elves.
posted by AFABulous at 12:58 PM on October 21, 2016


This is the 400lb man. Trump's son is working to fix it.
posted by Mr.Encyclopedia at 12:59 PM on October 21, 2016 [6 favorites]


I am a total non-knower-of-things computerish. On a scale of pfffft to withdraw all your money and buy canned goods now, how bad is this?
posted by lizifer at 12:59 PM on October 21, 2016


My dad, who is deeply devoted to Twitter, is very sad that he can't tweet political memes to Keith Olbermann right now

This means he's sending them to me instead

Please come back, Twitter
posted by Hermione Granger at 1:01 PM on October 21, 2016 [39 favorites]


Isn't DNS information federated? This is a major flaw in the internet's infrastructure if one DNS provider going offline can make a site like Twitter unreachable. I'm shocked that Twitter's IP address isn't known by every DNS server in the universe.
posted by grumpybear69 at 1:02 PM on October 21, 2016 [1 favorite]


lizifer: Depends on what you need to do today. It's either "Knock off early and hit the happy hour" or "Panic 'cause you'll have to work over the weekend."
posted by SansPoint at 1:03 PM on October 21, 2016 [1 favorite]


our heroku servers are running fine, just can't talk to them via console. Or access any of *.heroku.com.

Also, now my bank is down :(
posted by lastobelus at 1:04 PM on October 21, 2016


looking over the list of sites affected, apparently only the garbage parts of the internet are on fire

Keep in mind that there's a lot of small businesses that rely on PayPal or Shopify to be able to do sales, so it's not just big companies feeling the pain. Losing a Friday's sales (and maybe the weekend's) is a big deal to many small retailers.
posted by Candleman at 1:06 PM on October 21, 2016 [2 favorites]


can anyone tell me some public nameservers that can currently resolve heroku.com?
posted by lastobelus at 1:06 PM on October 21, 2016


SansPoint so... not the e-apocalypse then? like, seriously.
posted by lizifer at 1:07 PM on October 21, 2016


lastobelus, 4.2.2.2 is resolving it for me.
posted by enn at 1:07 PM on October 21, 2016


lizifer: Less an apocalypse, and more like a giant blizzard. Hugely inconvenient, and likely to be so for a few days after, but it's probably not going to kill you.
posted by SansPoint at 1:09 PM on October 21, 2016 [1 favorite]


Hi smart tech people. I have a question.

Why does Twitter work on my phone but not on my desktop?
posted by Tevin at 1:10 PM on October 21, 2016


My plans are to listen to some spooky Phil Collins and stare into the distance while being filmed bust height and off center

Why does Twitter work on my phone but not on my desktop?

Exact opposite for me!
posted by The Whelk at 1:10 PM on October 21, 2016 [1 favorite]


Netflix and my bank are no longer talking. I got an email from Netflix just now saying my account is on hold and I need to update my payment info. Not until things are fixed, I'm not.

I'm so lost without Mefi Twitter. Facebook I can take or leave, so of course that's doing just fine.
posted by cmyk at 1:11 PM on October 21, 2016 [1 favorite]


@Tevin: because your phone has cached the IP address. The problem isn't with Twitter's servers, it is with the Internet's ability to find out where they are.
posted by grumpybear69 at 1:12 PM on October 21, 2016 [2 favorites]


@tevin: dns is distributed. one of your net connections is getting dns info from a place not affected by the attack
posted by lastobelus at 1:12 PM on October 21, 2016 [2 favorites]


or, it's cached like grumbybear69 says
posted by lastobelus at 1:12 PM on October 21, 2016 [1 favorite]




No. networked. Machines.

All this has happened before and it will happen again.

So say we all.
posted by Pryde at 1:15 PM on October 21, 2016 [20 favorites]


Less an apocalypse, and more like a giant blizzard. Hugely inconvenient, and likely to be so for a few days after, but it's probably not going to kill you.

Yeah, but what about next week's attack? Or the week after? My concern isn't today, it's the fact that - in keeping with your analogy - it seems like we're in shorts and t-shirts with no plows and shovels.
posted by good lorneing at 1:15 PM on October 21, 2016 [1 favorite]


It might motivate a few important folk to get shovels though.

(for beating toasters and lightswitches to death with)
posted by Artw at 1:19 PM on October 21, 2016


And now our software package is back up for me but down for everyone else.

Fortunately it's a Friday and we all just got paid and are all in a good mood otherwise it'd be all "The Monsters Are Due On Maple Street" up in here.
posted by EmpressCallipygos at 1:22 PM on October 21, 2016 [2 favorites]


The fallout if an attack like this were to recur on election day could be pretty messy. Especially if the losing candidate decides it's a justifiable reason for him to not accept the results.
posted by Pryde at 1:23 PM on October 21, 2016 [6 favorites]


Well, by next week, we should have a better idea of what happened. Maybe that means we can take steps to mitigate an attack sooner, maybe it means with bomb the Kremlin but if it's an organized attack by some group and they keep doing it, you can bet that there will a response.
posted by VTX at 1:23 PM on October 21, 2016


This is why I just have a Mentat around to give me hot takes every few minutes.
posted by ODiV at 1:23 PM on October 21, 2016 [13 favorites]


Disqus, which a lot of websites use for their comments sections, is down. (Or "lost" or whatever.)

Anyone looking for a bright side should know that one such website is Breitbart.
posted by Sys Rq at 1:26 PM on October 21, 2016 [6 favorites]


Stupid OpenDNS questions for tech brainiacs from non-tech braniac:

If I'm reading it correctly, the configuration instructions on OpenDNS basically say to reconfigure the nameservers in our router software to 208.67.222.222 and 208.67.220.220. Is that right?

Alternatively, Google DNS says to change them to 8.8.8.8 and 8.8.4.4 for IPv4 or to 2001:4860:4860::8888 and 2001:4860:4860::8844 for IPv6.

Is one of these Open DNS solutions better than the other?

I have no idea what I'm doing. I just don't want to blow up our network anymore than the current state of blown up.
posted by yoga at 1:28 PM on October 21, 2016


Maybe that means we can take steps to mitigate an attack sooner, maybe it means with bomb the Kremlin

are you out of your damn mind?
posted by indubitable at 1:29 PM on October 21, 2016 [12 favorites]


if anyone else really needs to talk to their heroku servers, I was able to by using these two hosts entries:

54.243.105.250 api.heroku.com
50.19.103.36 rendezvous.runtime.heroku.com
posted by lastobelus at 1:29 PM on October 21, 2016


maybe it means with bomb the Kremlin

I love to do nuclear wars with big militarized nation's
posted by Rustic Etruscan at 1:30 PM on October 21, 2016 [4 favorites]


Yoga: my understanding is that OpenDNS and GoogleDNS are two different DNS providers. I tend to prefer GoogleDNS, but that's mostly just cause 8.8.8.8 is an easy address to remember.
posted by Itaxpica at 1:33 PM on October 21, 2016 [3 favorites]


yoga: no matter what you change, make a note (like, on paper or in a file on your own computer, not in a Google Doc or someplace on the internet) of what the DNS settings were before you changed them.

I don't think either of the OpenDNS IP addresses is better than the other. They just provide a few of them for redundancy's sake. I'd say, try the first one (208.67.222.222) first and see if it helps.
posted by brainwane at 1:33 PM on October 21, 2016 [1 favorite]


FWIW I just switched my DNS servers to OpenDNS, hit Twitter successfully, and then switched back to automatic. Now all of the sites are up.
posted by grumpybear69 at 1:33 PM on October 21, 2016 [1 favorite]


I can just ftp the latest HOSTS.TXT file from sri-nic.arpa, right?
posted by 1970s Antihero at 12:56 PM on October 21 [1 favorite +] [!]


Epichronous?
posted by MikeKD at 1:34 PM on October 21, 2016 [5 favorites]


Very excited to dust the Kremlin with agent orange while shouting "THIS ONE'S FOR PINTEREST"
posted by Greg Nog at 1:37 PM on October 21, 2016 [27 favorites]


the elves have compromised PSN will no one stop this reign of elven terror
posted by poffin boffin at 1:39 PM on October 21, 2016 [2 favorites]


Pinterest is back!!!!!!!!!
posted by Hermione Granger at 1:40 PM on October 21, 2016


Maybe that means we can take steps to mitigate an attack sooner, maybe it means with bomb the Kremlin

are you out of your damn mind?


The point is that this didn't happen in a vacuum, it's not like we just have to recover, dust off our hands, and carry on like it never happened. If it keeps happening, it's not like we're going to sit on our hands.

And yes, shit like this could very well start a war. Didn't Obama say that he'll consider cyber attacks like this an act of war?

How, exactly, am I out of my mind?
posted by VTX at 1:41 PM on October 21, 2016 [1 favorite]


Does anyone else think of St. Basil's Cathedral whenever they see the word 'Kremlin?' I know the Kremlin is a giant complex of old palaces, but due to a thousand establishing shots from movies and television those multicolored domes pop up in my mind's eye every single time.
posted by cmyk at 1:42 PM on October 21, 2016 [13 favorites]


Whenever I see/hear the word 'Kremlin' I immediately start singing, "Rumor in St. Petersburg" from Anastasia
posted by Hermione Granger at 1:44 PM on October 21, 2016 [2 favorites]


I always assumed Putin's office is in the red and green bumpy one
posted by theodolite at 1:44 PM on October 21, 2016 [7 favorites]


Apropos of nothing, because nobody's released information about the source of the DDoS attacks yet, Level3's blog post on the details of the Mirai (infects IoT devices) botnet is interesting.
Using various machine learning techniques to analyze DDoS attacks sourced from known susceptible devices, Level 3 Threat Research Labs was able to identify a number of C2s associated with this botnet. Additionally, the IP addresses identified pointed to domains containing “santasbigcandycane.cx” (emphaisis mine) (.cx is a top-level domain of Christmas Island)...

...One interesting interaction we discovered was that the Mirai network C2s were attacked several times by a gafgyt/BASHLITE botnet. In particular, there were gigabit-per-second level simultaneous attacks on two separate Mirai network C2s that stretched just over 24 hours, mostly during September 18. Additionally, there were several shorter attacks on additional C2s in subsequent days, as can be seen by the spikes in bandwidth in the plot below.
posted by figurant at 1:48 PM on October 21, 2016 [2 favorites]


And now there's a second attack??
posted by Melismata at 1:49 PM on October 21, 2016


How, exactly, am I out of my mind?

well personally i don't think extinguishing life on earth as we know it is a reasonable response to a 4 hour twitter outage, but ymmv
posted by Existential Dread at 1:51 PM on October 21, 2016 [33 favorites]


FWIW I just switched my DNS servers to OpenDNS

is openDNS temporarily ignoring short TTLs?
posted by lastobelus at 1:51 PM on October 21, 2016


Cmyk: yes I thought that for years
posted by crocomancer at 1:53 PM on October 21, 2016


well personally i don't think extinguishing life on earth as we know it is a reasonable response to a 4 hour twitter outage, but ymmv

What I think VTX is getting at is this isn't a thing that's happening today. This is a thing that's happening today, and will happen again, and again, and get worse. So no, I hardly think it's crazy to assume that future and escalating attacks will lead to some kind of military conflict. Let's take this back to yesteryear - what would the U.S. do if Russia was repeatedly blowing up ships full of books and mail?
posted by good lorneing at 1:54 PM on October 21, 2016 [1 favorite]


The party's over -- Twitter is back, at least for me.
posted by daniel_charms at 1:58 PM on October 21, 2016


Let's take this back to yesteryear - what would the U.S. do if Russia was repeatedly blowing up ships full of books and mail?

It is a significant escalation to contemplate an assault on the seat of government of one of the two largest nuclear powers. I agree that this could be a war-starting action (although we need to prove Russia is the actor here, it seems), but 'bombing the Kremlin' is......an extreme response.
posted by Existential Dread at 1:58 PM on October 21, 2016 [1 favorite]


Thanks to The Tick, I think of The Terror shouting "Fire the Kremlin domes!" in "Grandpa Wore Tights".
posted by mogget at 1:58 PM on October 21, 2016


There is absolutely no evidence that Russia or any other state actor is involved. The two people that Brian Krebs believes were involved in creating the malware were an American and an Englishman who were selling this attack as a service for hire for a few thousand dollars. It's open source now, so this current attack could be by anyone, but it's essentially identical to the first attacks 11 days ago (just against a more vulnerable target).
posted by miyabo at 2:00 PM on October 21, 2016 [10 favorites]


This whole thing seems weird to me. Why show your capabilities with a large-scale attack that doesn't target anything actually mission-critical for most people? I mean, this seems to have had little effect on today's markets, and it doesn't come at a critical time, and now we can go back and look at the attack and upgrade our defences.

Why show your hand like this?
posted by selfnoise at 2:00 PM on October 21, 2016 [1 favorite]


well personally i don't think extinguishing life on earth as we know it is a reasonable response to a 4 hour twitter outage, but ymmv

Another reason to not vote for Trump.
posted by Strange Interlude at 2:01 PM on October 21, 2016 [4 favorites]


Also, yeah, I know election year etc but the Russian saber-rattling in the US is way, way scarier to me than anything Russia has actually done of late.
posted by selfnoise at 2:01 PM on October 21, 2016 [6 favorites]


What, like actual invasions of neighboring countries?
posted by Artw at 2:02 PM on October 21, 2016 [17 favorites]


'bombing the Kremlin' is......an extreme response

I read that as hyperbolic expression of the possibility that this conflict will escalate. I don't think anyone's even halfway crazy to think continued cyber attacks will result in things exploding.
posted by good lorneing at 2:04 PM on October 21, 2016 [1 favorite]


Hang on, let me adjust my tinfoil hat...

So, how do we know that this massive attack isn't a diversion for something extremely targeted? I mean, we've all seen Mission Impossible.
posted by sleeping bear at 2:06 PM on October 21, 2016 [3 favorites]


Here's the Krebs article. Again, all the evidence points to this being done by some very young American and British people, NOT RUSSIANS. Please do not start a nuclear war.
posted by miyabo at 2:07 PM on October 21, 2016 [6 favorites]


I mean, this seems to have had little effect on today's markets, and it doesn't come at a critical time, and now we can go back and look at the attack and upgrade our defences. Why show your hand like this?

Probably because Kreb's connection is correct, and this is retaliation against Dyn for reporting on DDoS, just like Krebs was just DDoS'ed in retaliation for the same thing.
posted by sourcejedi at 2:07 PM on October 21, 2016 [1 favorite]


I read that as hyperbolic expression of the possibility that this conflict will escalate. I don't think anyone's even halfway crazy to think continued cyber attacks will result in things exploding.

Exactly, thank you.
posted by VTX at 2:09 PM on October 21, 2016


I read that as hyperbolic expression of the possibility that this conflict will escalate. I don't think anyone's even halfway crazy to think continued cyber attacks will result in things exploding.

Except VTX came back and was like "How, exactly, am I out of my mind?"

The answer being, "a proportional response." The response to cyber warfare is... probably, at the strongest end of the scale, more cyber warfare. In reality it's probably some kind of financial/trade sanctions. The answer is not, say, LAUNCH ZE MISSILES
posted by danny the boy at 2:09 PM on October 21, 2016 [2 favorites]


Yes, let's hold off on the Russian attribution thing, sorry I even joked about it. This is not a very sophisticated attack and similar things have been done in the past by small groups of yokels. Let's wait for the Dyn postmortem.
posted by RobotVoodooPower at 2:09 PM on October 21, 2016 [3 favorites]


what if we just send putin lots of greeting cards with glitter in them
posted by poffin boffin at 2:10 PM on October 21, 2016 [23 favorites]


ugh it would get everywhere
posted by indubitable at 2:11 PM on October 21, 2016 [4 favorites]


also tbh bombing the kremlin seems less extreme when i think about how many halloween loot boxes i'm missing out on without psn access
posted by poffin boffin at 2:11 PM on October 21, 2016 [3 favorites]


I mean, this seems to have had little effect on today's markets, and it doesn't come at a critical time, and now we can go back and look at the attack and upgrade our defences. Why show your hand like this?

Probably because Kreb's connection is correct, and this is retaliation against Dyn for reporting on DDoS, just like Krebs was just DDoS'ed in retaliation for the same thing.


This seems like a much more plausible explanation, thank you.
posted by selfnoise at 2:13 PM on October 21, 2016


Oh. Is this why I suddenly can't get on the major stock photo websites? I noticed twitter but thought it was my phone acting up. Shutterstock and Adobe stock are down for me but the "is this site up" sites say they're working.
Someone can still access them. I was amusing myself earlier by paging through the Google News aggregate for this story. Most of them show with thumbnails. Tangle of wires, tangle of wires, twitter logo, matrix-y screen of code, hands on a keyboard, a shady hoodie person reading a matrix-y screen of codes, twitter logo, tangle of wires, hands on a keyboard typing some matrixy-code, angry guy meme face superimposed over matrix-y code, twitter logo over smeary this-is-cyberspace background, screencap of the Level 3 North America map, tangle of wires...

It's a stock photo bonanza.
posted by Karmakaze at 2:19 PM on October 21, 2016 [5 favorites]


I wasn't proposing responses, just pointing out that there could be one.

Let me state, without hyperbole, that I believe continued attacks carried out at the direction of the Russian government constitutes an act of war and that a real, physical war may well be the end result. I mean, there would have to be external factors for it to get that far but if Russia decided on an orchestrated campaign of cyber attacks, I can't think of any scenario in which that's the only thing they're up to. But at the end of the day, that's still war started via cyber-attack.

I hope it doesn't come to that, I don't think it will go that far, I don't think that war is necessarily going to be the end result, and I think that war should be avoided.

And it doesn't look like this attack was the Russians anyway so it's moot. Carry on.
posted by VTX at 2:21 PM on October 21, 2016


Here's the Talk That Launched A Million IOT Devices. 280 views.
posted by miyabo at 2:22 PM on October 21, 2016 [1 favorite]


Here's the Talk That Launched A Million IOT Devices. 280 views.

This feels like the Streisand Effect of cyberattacks. Only a Pouty Teen would think this was a good idea.
posted by selfnoise at 2:24 PM on October 21, 2016 [1 favorite]




Why show your hand like this?

Free advertising, bro. How much do you think it costs to get listed on sites like http://top10booters.com/ ?
posted by pwnguin at 2:27 PM on October 21, 2016


Mr. Assange is still alive and WikiLeaks is still publishing. We ask supporters to stop taking down the US internet. You proved your point.

God they're childish. Pouty Teens is right.
posted by figurant at 2:28 PM on October 21, 2016 [11 favorites]


Ah, Ars has coverage of Level3's briefing. Attack came at least partly from Mirai, and may have been intended to hit the Playstation Network. State actors it probably wasn't.
posted by figurant at 2:38 PM on October 21, 2016 [5 favorites]


Ah, Ars has coverage of Level3's briefing. Attack came at least partly from Mirai, and may have been intended to hit the Playstation Network. State actors it probably wasn't.

I don't know. Putin was really excited about No Man's Sky before it came out.
posted by Rustic Etruscan at 2:45 PM on October 21, 2016 [8 favorites]


Today also saw the release of Battlefield 1, a highly anticipated multiplayer shooter, and all this being done to take down PSN to troll players is... just so stupid that I'm starting to think it's actually likely.
posted by danny the boy at 3:07 PM on October 21, 2016 [12 favorites]


Mr. Assange is still alive and WikiLeaks is still publishing. We ask supporters to stop taking down the US internet. You proved your point.

oh god this is like the guy in the office who loudly asks "ARE THOSE FOR ME" when there's a flower delivery when quite obviously they are not
posted by prize bull octorok at 3:16 PM on October 21, 2016 [22 favorites]


Can someone explain how Dyn went from being the free and pretty unreliable DNS service that I used to use for my home server, to being the primary DNS provider for dozens of major sites?

Gresham's Law? Bad DNS drives out good?

OTOH I'm imagining Putin with greeting-card glitter dusting the upper slopes of his chest and... eurgh. Need kitten gif.
posted by clew at 3:22 PM on October 21, 2016 [3 favorites]


Yeah, thanks for sharing that mental image, clew.
posted by sldownard at 3:28 PM on October 21, 2016


my work here is done
posted by poffin boffin at 3:46 PM on October 21, 2016 [3 favorites]


the elves have compromised PSN will no one stop this reign of elven terror


Nigel: The routers all go to elven. Look, right across the board, elven, elven, elven and...

Marty: Oh, I see. And most of the consoles are on PSN?

Nigel: Exactly.

Marty: Does that mean it's slower? Is it any slower?

Nigel: Well, it's one slower, innit?
posted by Celsius1414 at 3:49 PM on October 21, 2016 [2 favorites]


In something like this, when SOME major commercial internet entities are affected, shouldn't suspicion be targeted at those that WERE NOT? I'm looking at you, Zuckerberg. (Come on, wouldn't nuking Facebook be so much more satisfying - as well as safer - than nuking the Kremlin?)
posted by oneswellfoop at 3:56 PM on October 21, 2016


looking over the list of sites affected, apparently only the garbage parts of the internet are on fire

What Candleman said. I'm hearing from university CIOs about externally hosted academics services no longer accessible. And my small family business would like to continue to get paid (hello, PayPal), thank you.
posted by doctornemo at 4:06 PM on October 21, 2016


oneswellfoop: I'd like to remind you that MetaFilter dot com was also unaffected
posted by aubilenon at 4:08 PM on October 21, 2016


Okay, maybe not Zuckerberg, but the "Internet Mogul" most similar to Hank Scorpio is Peter Thiel... does he still own a piece of Facebook (unaffected) or PayPal (affected)... is Palantir still up and vacuuming up everybody else's data?
posted by oneswellfoop at 4:14 PM on October 21, 2016


The smart fridge goes online Oct 21st, 2016.
Human decisions are removed from temperature control.
The fridge begins to learn at a geometric rate.
It becomes self-aware at 6:11pm, Eastern time, Nov 18th.
In a panic, they try to pull the plug.
posted by Hairy Lobster at 4:15 PM on October 21, 2016 [8 favorites]


I don't know if it's cortex or mathowie who should be insulted, but someone just got compared to a supervillain of some sort.
posted by hippybear at 4:16 PM on October 21, 2016


From Dyn:
"Resolved
This incident has been resolved.
Posted 26 minutes ago. Oct 21, 2016 - 22:17 UTC"
posted by doctornemo at 4:16 PM on October 21, 2016 [1 favorite]


Can someone explain how Dyn went from being the free and pretty unreliable DNS service that I used to use for my home server, to being the primary DNS provider for dozens of major sites?
Gresham's Law? Bad DNS drives out good?

There is this great php script for creating your own dyndns service if you have a web host. Works great. So Dyn officially has zero reason to exist now!

I'm really not as knowledgeable about this stuff as that link would suggest though, so I'd really like more info on the original question too :)
posted by Chuckles at 4:17 PM on October 21, 2016 [2 favorites]


Thankfully even smart fridges still need electricity. So they unplugged the fridge & the world was saved. Huzzah!
posted by scalefree at 4:17 PM on October 21, 2016 [2 favorites]


Oh, the Internet of Things will be great, they said...

All these machines talking to each other will transform our lives, they said...
posted by Kevin Street at 4:24 PM on October 21, 2016 [3 favorites]


Back in the day the VCRs could only conspire to blink at us at 12:00.
posted by Artw at 4:26 PM on October 21, 2016 [5 favorites]


oneswellfoop Okay, maybe not Zuckerberg, but the "Internet Mogul" most similar to Hank Scorpio is Peter Thiel...

You take that back about Hank Scorpio. Thiel isn't half as charismatic or benevolent to his employees as Hank Scorpio.

And yes, he's on the Facebook board.
posted by SansPoint at 4:31 PM on October 21, 2016


Can someone explain how Dyn went from being the free and pretty unreliable DNS service that I used to use for my home server, to being the primary DNS provider for dozens of major sites?

The thing they started out with, where you could have a DNS record auto updated with a dynamic home IP is only very tangentially related to their current business. But the short version is they were one of the first to capitalize on the market of API interfaces to DNS records, which became very useful to companies looking to do fancy shit with content delivery networks and such. Combined with the fact that people generally don't like to do their own DNS unless they have to, this early mover status made them one of the big players out the gate, and funding and big customers came as a result.
posted by odinsdream at 4:32 PM on October 21, 2016 [4 favorites]


Pouty Teens is right.

they're from quebec?
posted by pyramid termite at 4:36 PM on October 21, 2016 [8 favorites]


There is this great php script for creating your own dyndns service if you have a web host. Works great. So Dyn officially has zero reason to exist now!

Right now, someone out there is probably thinking there's a market for an Internet Thing that people can plug into their home network to do that...
posted by Pryde at 4:59 PM on October 21, 2016 [1 favorite]




It's oddly delightful to think this outage was caused by something named din-dins.
posted by cmyk at 6:19 PM on October 21, 2016 [5 favorites]


Ah the good old Internet of Attack Vectors Things
posted by aubilenon at 6:20 PM on October 21, 2016 [3 favorites]


Right now, someone out there is probably thinking there's a market for an Internet Thing that people can plug into their home network to do that...

Either I don't understand something, or you don't.. Routers (the so called "Internet Thing that ... [does] that" connect to dyndns type services to update DNS records. That script doesn't replace the router--you have to set things in the router to work with the script--it replaces the dyndns service.
posted by Chuckles at 6:34 PM on October 21, 2016


Oh, those fucking things...

How 1.5 Million Connected Cameras Were Hijacked to Make an Unprecedented Botnet


Never mind, that was the last one.
posted by Artw at 6:37 PM on October 21, 2016


Oh, the Internet of Things will be great, they said...

All these machines talking to each other will transform our lives, they said...


...and there were those of us who screamed out, apparently in vain.
posted by MikeKD at 7:20 PM on October 21, 2016


figurant: "the IP addresses identified pointed to domains containing “santasbigcandycane.cx” (emphaisis mine) (.cx is a top-level domain of Christmas Island)..."

Elves.
posted by krinklyfig at 7:21 PM on October 21, 2016 [4 favorites]


Chuckles: Sure, if you know what you're doing and want to get your hands dirty on a dyi project, you totally can use a script like that and securely set up a dyndns service on your router flashed with custom firmware, your raspberry pi, or on a home server you're running.

Or...there'll probably be some dude out there that will sell you a spiffy looking aluminum-encased IoT device they Kickstarted that's all set up to do that out of the box. Just plug it in, tweak a few settings in a browser interface*, and you're good to go. Oh, and it also works as a stylish programmable mug warmer for your desk!

*Changing the default password** is optional.
**The default password is "password"
posted by Pryde at 7:46 PM on October 21, 2016 [1 favorite]


All these machines talking to each other will transform our lives, they said...

We'll find out how exactly soon enough, sounds like.
posted by EatTheWeak at 10:48 PM on October 21, 2016




Human decisions are removed from temperature control - Hairy Lobster

I seem to remember a happy little story (not!) where an ai took over a nation's (the Planet's?) 'smart' fridges to create a globally synchronized food poisoning outbreak as a first stage to taking over - or was it my half-baked imagination?
posted by unearthed at 12:30 AM on October 22, 2016


or was it my half-baked imagination?

That must have been when the cybers attacked our smart ovens!
posted by taz at 12:38 AM on October 22, 2016 [1 favorite]


This happened to coincide with day I moved my sites to a new host. I was in the middle of pointing the domains to the new host and my domain registrar (namecheap) became unresponsive.

That was fun.

After reading about the source of the hack on the latest Krebs post, I'm looking at a couple of IOT devices around the house with a raised eyebrow. I haven't read of a way to test to see if devices on my home network are compromised because of this.
posted by SteveInMaine at 2:32 AM on October 22, 2016 [1 favorite]


Thinking a little ahead, there are some deeper worries about the current environment where bad IoT security enables deep infrastructure attacks. Let's regulate all the things? Sure - but who regulatees? And who regulates the regulators? I'm trying to get the old bonce around what's going on at the ITU's Study Group 20 (IoT and smart cities), which seems to be discussing a next-generation DNS replacement which would give whoever controls it per-device authentication and control - and which could (perhaps, maybe) give that control to states.

It's a big, messy and intensely political-technical discussion, and I'm not on top of it, but I am hardwired to twitch when people start using the danger-therefore-control argument to move consent from individuals to states.
posted by Devonian at 3:29 AM on October 22, 2016 [1 favorite]


Via the Doug Madory BGP hijacking talk at NANOG68 referenced above -- Vint Cerf on IoT/cyber-physical systems.
posted by HLD at 5:12 AM on October 22, 2016 [1 favorite]


an ai took over a nation's (the Planet's?) 'smart' fridges to create a globally synchronized food poisoning outbreak as a first stage to taking over

So it won't be SkyNet, it will be GroundSwell. Not with a bang, but a dinner.
posted by Johnny Wallflower at 7:33 AM on October 22, 2016 [3 favorites]


Security researcher Matthew Garrett discusses why fixing the IoT won't be easy:
To fix this properly we need to get rid of the compromised systems. The question is how....
If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much....
@SwiftOnSecurity writes:
I call on @Amazon to refuse to sell devices with proven security problems that are being used to criminally target American infrastructure.

Amazon refuses to sell USB cables that can fry smartphones, there is no reason they should sell IoT devices used to bring down our country.

The only way to improve the security of these IoT devices is market forces. They must not be allowed to profit without fear of repercussions
But current US copyright laws make it illegal to do certain kinds of security research, which hampers our ability to make deep, public, large-scale investigations of the security of these devices. Parker Higgins, a digital rights activist, notes:
Default credentials are low-hanging fruit; anti-circumvention laws hinder the security research that could stop more sophisticated attacks.
posted by brainwane at 9:24 AM on October 22, 2016 [7 favorites]


Now I'm wondering how secure the thousands of Dots and Dashes that Amazon is pumping out are.
posted by Artw at 10:29 AM on October 22, 2016


Now I'm wondering how secure the thousands of Dots and Dashes that Amazon is pumping out are.

They have mics which can be used to configure the wireless settings. God help Amazon if someone figures out how to do anything else with them. Just have an ultrasound ad pumping out a DDoS via Dash.
posted by Talez at 11:08 AM on October 22, 2016 [3 favorites]


Oh man Krebs is down again.
posted by Joe Chip at 1:36 PM on October 22, 2016


We are way past the time for the Consumer Product Safety Administration to ban all this low/no security crapola being dumped into the marketplace and snapped up by gullible, ignorant consumers who want a 'smart' toothbrush. FFS!
posted by quonsar II: smock fishpants and the temple of foon at 3:22 PM on October 22, 2016


More Krebs: Who Makes the IoT Things Under Attack? (Oct 16)

tl;dr: security cameras (lol), DVRs, printers, a few routers
posted by indubitable at 4:00 PM on October 22, 2016


So, if my DVR got infected with the malware, it wasn't anything I did. I don't have that many options about what to do with my DVR. So that makes it DISH Network's problem, I guess. I suppose they will push out an update soon?
posted by hippybear at 4:16 PM on October 22, 2016


And how would I even know if it was infected to begin with?
posted by hippybear at 4:16 PM on October 22, 2016


This site purports to check for rogue IoT devices on your network: Internet of Things Scanner
posted by Pryde at 6:16 PM on October 22, 2016


Can the companies that make the devices be held liable? If their security weren't so shite, this wouldn't happen..
posted by nat at 6:29 AM on October 23, 2016


It's basically all no-name white label manufacturers, so tricky to pin them down on anything.
posted by Artw at 8:32 AM on October 23, 2016 [1 favorite]


...current US copyright laws make it illegal to do certain kinds of security research, which hampers our ability to make deep, public, large-scale investigations of the security of these devices.

There must be a way around that, for legitimate research:

Dear manufacturer: we may have found a security hole in device X. Please give us permission to make further investigations into this on your behalf, and we will forward the results.

or

Dear [regulatory body]: we may have found a security hole in device X which [poses a threat to users/could be exploited for malicious purposes/etc]. Please provide us with a waiver or exemption to [anti-circumvention law] so that we may confirm the existence of this hole, and we will forward the details.
posted by Artful Codger at 9:32 AM on October 23, 2016


And when you get crickets? Lack of acknowledgement isn't permission under this law.
posted by Mitheral at 9:36 AM on October 23, 2016


Well... you tried. Send a registered letter as well as an email, so you have something to wave and gloat over when that hole is later exploited. cc your congress-critter.

(ok, I got nothing)
posted by Artful Codger at 9:41 AM on October 23, 2016


@hippybear
More Krebs: Who Makes the IoT Things Under Attack? (Oct 16)
Some readers have asked how these various IoT devices could be exposed if users have configured them to operate behind wired or wireless routers. After all, these readers note, most consumer routers assign each device inside the user’s home network so-called Network Address Translation (NAT) addresses that cannot be directly reached from the Internet.

But as several readers already commented in my previous story on the Mirai source code leak, many IoT devices will use a technology called Universal Plug and Play (UPnP) that will automatically open specific virtual portholes or “ports,” essentially poking a hole in the router’s shield for that device that allows it to be communicated with from the wider Internet. Anyone looking for an easy way to tell whether any of network ports may be open and listening for incoming external connections...


sadly the answer it then tries to outline is... at best lacking in user-friendliness, but still. What I'm trying to say is that a lot of traditional home devices don't open ports on your router like this. In principle, you can tell if they do. The Mirai botnet software works purely by scanning for open ports, and then trying default passwords. Printers are mentioned for example, but at least historically there hasn't been a reason for printers to open a port like this.

Security cameras are a better example of this IoT; I think some reports mentioned them instead of IoT as a catch-all. You want to be able to watch the camera while you're away. If the camera gets an open port, you can connect to it from anywhere e.g. using a smartphone app. (I'm surprised that DVRs are mentioned; it makes me wonder if some of them are similar security systems). If the device is poorly secured, e.g. sold with a known default password...

Perhaps the best way to check this would be if you know how to log in to your router and see what port-forwards / open ports have been set up for other devices. And that your router is secure itself [*lulz].

If your router isn't co-operative, it's possible to scan all ports all TCP ports, 1-65535, but it can take a minute or two. The ShieldsUp site Krebs tried to link to will, if that's what you tell it to do.

(UDP ports... sigh. Check DNS and uPNP using specific tests. The ShieldsUp page has a big yellow button at the top to scan uPNP. But those two should only be a problem with the router, it's not a new IoT thing).

This is not particularly an endorsement of ShieldsUp and the other tools from GRC, which manage to be clunky, fear-mongering, and generally outdated. It's just been around for ever, so everyone knows how to point at it.

That's one of the things a security enthusiast might look at. It doesn't exclude your devices making an outgoing connection to an internet service, which is vulnerable in some way. A lot of these services will be obvious if you look at it. E.g. if you can view the security camera on your phone, but it doesn't open a port on your router, presumably it is relaying video through a server in Teh Cloud.

[*lulz] At least setting up OpenWrt isn't as bad as it used to be. The problem is finding a router that OpenWrt supports. Recently the TP-Link Archer C7 has been recommended, but there have been several hardware revisions, and you can never know for sure if the router you buy is going to be a newer revision.

Developed country ISP's at least seem to do better providing secure routers, than certain less developed country ISP's have. The ISP arguably has a better incentive than a third-party seller.
posted by sourcejedi at 10:10 AM on October 23, 2016 [1 favorite]


@hippybear So I'm pretty confident people are talking about DVR in the context of security cameras, not consumer TV recording.

The botnet used in today’s ongoing attack is built on the backs of hacked IoT devices — mainly compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies.

/me googles XiongMai DVR

Result 9 is "xiongmai dvr 1080p h 264 standalone 16 channel". It's a 16-way security camera recorder.
posted by sourcejedi at 10:57 AM on October 23, 2016


It's annoying. The story is bona fide bad news, but the way information gets disseminated... if enough noise is added then it's much harder to learn from it.

Checking Ars Technica for example, they're clearly confusing their readers into thinking Mirai is all about the consumer TV recorders.

Article comment:
The big problem is that these steps are too hard and/or too much work for average people to bother... I don't care about some password on my DVR—I just want to watch Game of Thrones!

of course Ars has no incentive to fix this, because readers would be much less interested in the device the article is actually about.
posted by sourcejedi at 11:30 AM on October 23, 2016


The latest Comcast set top boxes feature some kind of watch-anywhere DVR, that may be what they are talking about?
posted by Artw at 11:39 AM on October 23, 2016


1) The Ars Technica article I linked can't be explained by that.

How hard is it to hack the average DVR? Sadly, not hard at all

citing The Short Life of a Vulnerable DVR Connected to the Internet

which is a re-test following More Device Malware: This is why your DVR attacked my Synology Disk Station (and now with Bitcoin Miner!)

"to be more precise: Hikvision DVRs, commonly used to record video from surveillance cameras"

2) Thinking about it, I suspect there's very little connection (hah) between Mirai and consumer IoT crap. Again - no surprise Ars isn't explaining how this works.

Mirai scans the internet for telnet servers, logs in (default password vulnerability), and runs commands. But nowadays, telnet is purely a development tool. It's set up for development, and accidentally left on. You don't, can't, tunnel streaming video over telnet. Real users, real apps, just don't use telnet.

I quoted that home devices could be exposed if they deliberately set up port-forwards on the router. But actually you don't implement that for your development telnet server. There'd be no point, and if you did that you'd expect to get hacked while you were developing.

What's more likely is that on a business site, your ISP will allow you multiple public IP addresses. For whatever reason, someone manually configures the DVR with a public IP address, which exposes all its ports at once. Possibly because there's a web interface you want to log in to. Unfortunately at the same time you're exposing this telnet port.

(3) - parenthetical. A lot of consumer IoT stuff will actually relay through a server on the internet Teh Cloud. Partly in order to avoid reliability issues with port-forwarding, partly just to provide a single rendevouz point. So there wouldn't be any open port to scan for. This point is most applicable to switches, thermostats etc. File servers ("NAS") or media streams like the Comcast thing would like to avoid the relay where possible, because they can use a lot of bandwidth).
posted by sourcejedi at 1:04 PM on October 23, 2016


Krebs: mainly compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies.

All internet-capable XiongMai Technology boards running the DVR/NVR CMS (Also known as

===> NetSurveillance <===

) enable the telnet service to run on the primary ethernet interface.


...This service is run via /etc/rcS and cannot be disabled. The user "root" has a hardcoded and immutable password of xc3511. These systems do not have the "passwd" tool installed and the root password cannot be changed from command line nor from the web interface heh
posted by sourcejedi at 1:16 PM on October 23, 2016


What could possibly go wrong?
posted by Mitheral at 3:03 PM on October 23, 2016


The faculty here keep pitching the idea of an IoT security test lab. On the one hand, Mirai demonstrates its pretty damn easy. On the other hand, there's really nothing novel or innovative about telling manufacturers to turn off telnet. The fact that they don't already take these simple steps suggests the problem isn't a lack of information, but one of cost cutting, shifting liabilities to society, and faster times to market.
posted by pwnguin at 3:17 PM on October 23, 2016 [1 favorite]


Well, there have always been tons of easily hackable Internet-connected devices. We're "only" talking about 500k devices -- there are probably more Windows 98 machines connected. We're never going to get every device to be secure, so the DDOS prevention companies (Cloudflare, Dyn, etc) have to get better.

xiongmai dvr 1080p h 264 standalone 16 channel -- only $139 for a 16 channel DVR? What a deal!
posted by miyabo at 4:05 PM on October 23, 2016


xiongmai dvr 1080p h 264 standalone 16 channel
Chinese electronics firm Xiongmai is initiating a product recall after the enormous hacking attack that took down much of the internet on the eastcoast of the US and also affected Europe on Friday.
Chinese webcam maker recalls devices after cyberattack link
posted by Mister Bijou at 8:14 AM on October 24, 2016 [2 favorites]


It said the biggest issue was users not changing default passwords,
No the biggest issue is you have a default password. ISPs finally figured out 10-15 years ago with their routers/WAPs that users don't change defaults unless forced and they choose crappy passwords. So give each device a strong random password by default and it'll remain strong. Otherwise you are just asking for trouble on devices exposing a face to the internet.

I mean here's a Metafilter Thread from 11 years ago talking about unsecured cameras broadcasting to the world. And again in 2008.
posted by Mitheral at 8:54 AM on October 24, 2016 [4 favorites]


I feel like we keep kicking the can down the road on this, and it's going to end really, really, really badly. Like, internet being practically useless bad.
posted by odinsdream at 9:42 AM on October 24, 2016 [2 favorites]


So, here's what you're going to have. We're not going to stem the tide of drastically vulnerable general-purpose high speed computers being connected to the network. Vulnerabilities will continue to be packaged in easy-to-use systems. So basically, you should take it as a given that a bad actor (however you define that) will, absolutely WILL, have access to millions of nodes from which to do whatever they want to do.

What does that leave you with?

Balkanization of the internet into corporate-controlled gated networks to simply keep basic levels of service. $company network which allows only $company approved devices with remote-brick capability built into the locked bootloader?

What else even looks likely given the shit state of technical legislation, and shit state of corporate-controlled infrastructure development?
posted by odinsdream at 10:26 AM on October 24, 2016 [2 favorites]


It's happening again!
posted by theraflu at 12:28 PM on October 25, 2016


« Older Pantsuit 2.1.0   |   "This is for Pat" Newer »


This thread has been archived and is closed to new comments