[e-mail]: You're one of 1 countries pwned by Cozy Bear

December 17, 2020 5:57 PM

The computer networks of a large number of United States government agencies and private sector companies have been infiltrated in a massive cyberattack that has been attributed by US Government officials to the Russian advanced persistent threat actor APT29, also known as Cozy Bear.

The list of affected government agencies and companies has grown larger each day since the intrusion was first revealed, with the National Nuclear Security Administration, the city government network of Austin, Texas, and Microsoft among the latest to be revealed as victims of the attack.

The hack, which was facilitated by a compromise of an update server used by SolarWinds Orion monitoring software, now poses a "grave risk" to the United States according to the US Cybersecurity and Infrastructure Security Agency (CISA), who has published a detailed report on the incident, including the details of how the SolarWinds supply chain was compromised, how the malware enables lateral movement on the part of the attackers, and how to mitigate the compromise on affected networks. CISA's director was fired last month by outgoing US President Donald Trump in retaliation for CISA's conclusion that the 2020 US election was "the most secure in American history", undermining Trump's attempts to cast doubt on the result.

SolarWinds, who was using an update server with the password solarwinds123 as recently as last year, is in PR crisis mode while its many government and corporate customers attempt to survey the damage and eliminate the threat from their networks. A "killswitch" has been discovered has enabled the seizure and sinkholing of the command and control domain used by the SUNBURST malware propagated by SolarWinds, but given the "God-mode" access the attackers were granted, it seems likely that they found ways to move laterally and compromise additional resources on the affected networks.

While it remains to be seen if this is the Largest Espionage Attack in History, it certainly ranks highly on such a list given the number of affected agencies and private sector firms affected.

posted by tonycpsu (50 comments total)

Looks like planting Trump has started paying dividends.
posted by Thorzdad at 6:32 PM on December 17, 2020 [33 favorites]

SolarWinds, who was using an update server with the password solarwinds123 as recently as last year

What the absolute fsck. This is something for which not even 2-factor authentication should be considered good enough. And it's not even thought to have been the source of the current intrusion. So there were probably multiple comparable holes in the company's security.

I think someone should have explained that the "Swiss Cheese" risk model doesn't mean that your security should have holes in it.
posted by Joe in Australia at 6:37 PM on December 17, 2020 [30 favorites]

Password now changed to solarwinds456.
posted by benzenedream at 6:45 PM on December 17, 2020 [72 favorites]

By the time the scope of the damage becomes clear, Trump will have scarpered off to one of his boss's many dachas. I hope Republicans are made to pay for this: reputationally, but mostly punitively. Jail time would be good.
posted by They sucked his brains out! at 6:47 PM on December 17, 2020 [7 favorites]

Everything bad does not happen to involve Trump (yet). This was stupid stuff done at a software company. I haven't used SolarWinds in a few years, but most of these network tools just have a fancy GUI that pings IPs. Now I worry what software we have on our network that is compromised. That'd be all that I can say publicly, not that I know everything or most stuff on the network where I work. I just do network, not security.
posted by baegucb at 7:09 PM on December 17, 2020 [6 favorites]

SW offers an agent to be truly effective, which means it has some local credentials on every box where it runs.

Or, you know, you don't trust it so you only let it ping your host, or maybe use a tight SNMP read-only account at most.

Man, I miss my old pre-fork Nagios setup: it was so good.
posted by wenestvedt at 7:17 PM on December 17, 2020 [5 favorites]

Password now changed to solarwinds456.

Are you kidding? Their insanely strong new password complexity requirements also indicate a special character -- so it will be solarwinds123! from now on.
posted by wenestvedt at 7:20 PM on December 17, 2020 [25 favorites]

I hope Republicans are made to pay for this: reputationally, but mostly punitively.

They’ll conveniently ignore it all until after Biden takes office, then start lambasting him for not securing the networks immediately.
posted by Thorzdad at 7:21 PM on December 17, 2020 [30 favorites]

I was going post a pithy comment about how many places think security and monitoring software is good, so more security and monitoring software must be better, but frankly I'm just glad none of our customers are running this.

This feels like the Enterprise Edition of the CCleaner event from 2017.
posted by krisjohn at 7:22 PM on December 17, 2020 [1 favorite]

If this is the Russians, it was a pretty ballsy intrusion — one for which the risk/benefit analysis would have been quite robust. Given that Russia put literal bounties on the heads of American soldiers in Afghanistan with no consequences from the Trump administration, I’d say that their risk assessment was, shall we say, rosy. So yeah, while Trump’s not *technically* responsible for this breach, it happened on his watch (with all the shit that entails) and he owns it.
posted by Toecutter at 7:54 PM on December 17, 2020 [25 favorites]

Trump doesn't deserve to be saddled with some corporation's malfeasance. I think what he's being called to account for (deservedly) is not standing up to Russia and calling then out for the back.
posted by CheeseDigestsAll at 8:00 PM on December 17, 2020 [5 favorites]

I’m just a lay person but it seems like the Orion product is meant to monitor Microsoft/Windows platforms? In my dream world they would just reimage the effected servers with Linux / open source software.

I know open source software has vulnerabilities too and can be compromised as well but it seems like as good as a time as any to migrate.
posted by mundo at 8:16 PM on December 17, 2020

Rudy G was head of a cyber task force that briefed the feds right around the time the Russians started penetrating the total landscaping of America's untucked cybershirt.
posted by srboisvert at 8:21 PM on December 17, 2020 [15 favorites]

If this were a political thriller, Putin's congratulatory call to Biden -- that came in prior to and probably prompted McConnell's acknowledgement of the election results -- could have easily been read as a public signal that the Trump admin accomplished what they were tasked with and are now disposable because Putin now has the keys to everything he wanted. And it would have been a taunt to Biden that he's now just a figurehead because the entire governmental organization he's now in charge of has been deeply, irrevocably compromised. And then the chapter or season or novel would end and we'd all be waiting around to see what happens in the next installment.

Of course such ludicrous events would have to be buoyed by some equally ludicrous world building and prior plot points. The president would have to have been someone who was in deep financial debt to Russian powers and he would have had to have pretensions of becoming a lifelong dictator just like Putin. But he would have had to have been very incompetent -- and yet, despite his utter incompetence, he would have had to have had a whole political party of lapdogs willing to sell the country out in hopes they'd get to be oligarchs someday and maybe get their piece of the kingdom to rule over, with ghoulish cruelty masked as "commitment to God and traditional values".

Yeah, that would be dumb story. No one would buy it.
posted by treepour at 8:39 PM on December 17, 2020 [79 favorites]

"As far as the cyber, I agree to parts of what Secretary Clinton said. We should be better than anybody else, and perhaps we’re not. I don’t think anybody knows that it was Russia that broke into the DNC. She’s saying Russia, Russia, Russia—I don't, maybe it was. I mean, it could be Russia, but it could also be China. It could also be lots of other people. It also could be somebody sitting on their bed that weighs 400 pounds, okay? ... We came in with the Internet. We came up with the Internet. And I think Secretary Clinton and myself would agree very much, when you look at what ISIS is doing with the Internet, they’re beating us at our own game. ISIS. So we had to get very, very tough on cyber and cyber warfare. It is a huge problem. I have a son—he’s 10 years old. He has computers. He is so good with these computers. It’s unbelievable. The security aspect of cyber is very, very tough. And maybe, it's hardly doable. But I will say, we are not doing the job we should be doing. But that’s true throughout our whole governmental society. We have so many things that we have to do better, Lester. And certainly cyber is one of them."
posted by Multicellular Exothermic at 9:48 PM on December 17, 2020 [19 favorites]

Yeah folks, but her emails.
posted by PenDevil at 9:50 PM on December 17, 2020 [13 favorites]

More Hacking Attacks Found as Officials Warn of ‘Grave Risk’ to U.S. Government.
posted by blue shadows at 10:12 PM on December 17, 2020

Yeah folks, but her emails.

This is a reminder that the reason Cheatin' Donald was so focused on Clinton's home email server was because it was one of the servers Trump's Russian Hackers could not breach.
posted by mikelieman at 12:07 AM on December 18, 2020 [27 favorites]

treepour: "Yeah, that would be dumb story. No one would buy it."

It would only really work anyway if there were some kind of horrific incident that distracted everyone from the plot while the final gears ground into place, something like a global pandemic that disrupted all normal activity and engaged the governments, military and medical establishments in trying to save people from dying.

But that would stretch credibility too far.
posted by chavenet at 1:31 AM on December 18, 2020 [11 favorites]

As I've said elsewhere, I really hope the unclassified networks are still airgapped from the ones that store the high level shit at NNSA and the national labs.

There probably isn't a lot Russia doesn't know about our nuclear weapons already, but it isn't exactly unheard of for one group's backdoors to be made use of by others and the last thing we need is detailed design information and actual PAL codes floating around.

I'm not really surprised that they'd go after FERC, but it is mildly surprising that they'd go after nuclear secrets. That could be very destabilizing in a way that won't end well for anyone, not merely the US. It's also one of the few things that has a serious chance of getting people mad enough to retaliate.
posted by wierdo at 1:31 AM on December 18, 2020 [1 favorite]

It's worth noting that the attackers managed to compromise the two-factor secrets in this case, rendering any extra protection from a second-factor, moot.
posted by fragmede at 2:20 AM on December 18, 2020 [2 favorites]

There probably isn't a lot Russia doesn't know about our nuclear weapons already, but it isn't exactly unheard of for one group's backdoors to be made use of by others and the last thing we need is detailed design information and actual PAL codes floating around.

I'm not really surprised that they'd go after FERC, but it is mildly surprising that they'd go after nuclear secrets. That could be very destabilizing in a way that won't end well for anyone, not merely the US. It's also one of the few things that has a serious chance of getting people mad enough to retaliate.

Remember last week (month?) when Trump was ordering the dismantling airplanes so that the US and Russia couldn't restart their nuclear treaty related flights, called Open Skies, over each other's nuclear arms facilities? File this under 'Things that make you go "Hmmmm" '.
posted by srboisvert at 2:45 AM on December 18, 2020 [13 favorites]

I hope this can be a catalyst towards more insourcing of critical IT. The US Federal government is huge, and its IT needs are unique. It would completely make sense for the GSA to build open source tools for things like network monitoring and security, that could be implemented across the government and by anyone else for whom they worked. You can see the success of the US Web design system as an example of how this can work.
posted by rockindata at 4:36 AM on December 18, 2020 [10 favorites]

I know open source software has vulnerabilities too and can be compromised as well but it seems like as good as a time as any to migrate.

I think some of the heaviest users of Solarwinds are in the public sector - government/defense, universities, etc. In a lot of these places, a paid support contract with a Well Known Vendor is worth way more to the decision makers. It's even tougher at the US state level, where some of the chief decision makers are appointees who may rotate in and out with the turning fortunes of the election cycle. But agreed.
posted by jquinby at 6:45 AM on December 18, 2020

It seems tragic and inevitable now that it will take something on par with a nuclear disaster before humans understand the implications of building a networked world without a clear picture of the consequences for failing to secure and maintain code properly.

It would be great if we could just rewind back to the conversation about whether or not to privatize the post office and all watch Friends together.

Instead, we're all putting our heads in the sand about the problem of when modern cryptography breaks.
posted by abuckamoon at 7:14 AM on December 18, 2020 [3 favorites]

This kind of event is terrifying to me because I’m working in software in a company that has customers, nothing special, and so I get through the days thinking I am probably not letting in Russian hackers to all kinds of customer networks. But the reality seems to be that some slip up somewhere may not have gone unnoticed, and I’m going to find out in a few months that I left a stepping stone for one of the “lateral movements” in the next of these. I mean I want to snigger at solarwinds123 and yet I’m pretty sure it’s a case of “let he who who is without vulnerabilities cast the first stone”.
posted by pulposus at 7:33 AM on December 18, 2020 [5 favorites]

Friendly reminder from the depressing world of cybersecurity: there isn't a lot of realistic defense against a determined attacker. If that attacker has nation-state access to resources (previously unknown exploits, which includes the humint kind to play a long game of compromising not just one but many supply chains so you can afford to burn one) then the likelihood that for every compromise you notice there are two you don't goes up.

Even air-gapped. (See the classic high frequency audio exfil achieved by modulating fan speeds).

Everyone who can read this message has some account somewhere with poor front end security, some IoT or legacy mobile device or smart TV with unpatched flaws... Or someone they know does and they've spent time within Bluetooth range or on the same network as such a device.

Sure, SolarWinds needed better security at that point in their supply chain, but so does every build server and firmware host in the supply chain before them and in the deployment and update infrastructure after.

I heard a great interview on On the Media where an expert described cyberweapons as "perfectly selectable." You can use exactly as much as the adversary will tolerate without escalating to open conflict, but at a scale and frequency unavailable in a hot-threat situation.

So what can be done about it? As long as capital drives technology this kind of compromise remains a cost of doing business. Regulation drives the bottom up slightly but way too slowly to make much difference. I'm not convinced that true software liability actually does anything but drive development out of regulated markets and self-inflict enormous cost (even if I think it's the ethically correct path). In short: look to your own situation but be compassionate about others' because your security is also only as good as its weakest link.
posted by abulafa at 8:35 AM on December 18, 2020 [8 favorites]

from the NYT link

Minutes after the government statement, President-elect Joseph R. Biden Jr. warned that his administration would impose “substantial costs” on those responsible. President Trump has been silent on the hacking.

It struck me that GCHQ's quoted response was so po-faced that I wondered if they were actually the ones squirrelling their way in before brexit.
posted by infini at 10:30 AM on December 18, 2020

Further, I wouldn't be surprised if all eyes already had "We're russian hackers" sockpuppets ready to roll.
posted by infini at 10:32 AM on December 18, 2020

Given that Twitter seems to think I'm an uppity black woman in inner city Philly*, I wouldn't be surprised at what else begins to emerge from this hack scandal even before pumpkinhead is dragged out of his shapely office.

*currently one of the suspended, yet again, rinse repeat
posted by infini at 10:34 AM on December 18, 2020

“Governments have long spied on each other but there is a growing and critical recognition that there needs to be a clear set of rules that put certain techniques off limits,” Mr. Smith said. “One of the things that needs to be off limits is a broad supply chain attack that creates a vulnerability for the world that other forms of traditional espionage do not.”

Let me translate this from Microsoft for you - "You can't do to us what we've been doing to billions of you for years, that's not fair play, after all, we're the mighty might and only we get to do these things. Bill said so"

etc.

/whiny voice
posted by infini at 10:38 AM on December 18, 2020 [1 favorite]

Yeah, that would be dumb story. No one would buy it.

You know what would be even dumber? What if Putin had convinced Trump to low-key sabotage American cyber defenses so the GRU could use their l33t hax0r skillz to rig the election for Trump. Putin knows that probably wouldn't work, but Trump jumps at the offer of help. Instead, Cozy Bear goes hog wild for 9 months, and Trump loses the election anyway. But Trump won't admit he lost, because he can't believe Putin would have double crossed him, and still expects him to pull a last-minute ace out of his sleeve. But Putin has want he wants now, and cuts Trump loose. Because unlike the Republican party, the old KBG hand recognizes when an operative has turned from as asset to a liability.
posted by vibrotronica at 10:41 AM on December 18, 2020 [14 favorites]

I Was the Homeland Security Adviser to Trump. We’re Being Hacked.



sniggers at headline
posted by infini at 10:48 AM on December 18, 2020

"You know what would be even dumber? What if Putin had convinced Trump to low-key sabotage American cyber defenses so the GRU could use their l33t hax0r skillz to rig the election for Trump. Putin knows that probably wouldn't work, but Trump jumps at the offer of help. Instead, Cozy Bear goes hog wild for 9 months, and Trump loses the election anyway. But Trump won't admit he lost, because he can't believe Putin would have double crossed him, and still expects him to pull a last-minute ace out of his sleeve. But Putin has want he wants now, and cuts Trump loose. Because unlike the Republican party, the old KBG hand recognizes when an operative has turned from as asset to a liability.
posted by vibrotronica at 10:41 AM on December 18 [2 favorites +] [!] "

If you believe that, I've got an andrenochrome farm for sale that you're going to LOVE.
posted by youthenrage at 10:59 AM on December 18, 2020

Hmm, from over a year ago. Sure, code is gonna have vulnerabilities - developers try to make sure things work, they don't think about how to break them. But a good way to find them and plug them is proper Pen-testing. I would argue that in 2020, this is more important than getting new fighter planes, etc.
posted by pyro979 at 10:59 AM on December 18, 2020 [4 favorites]

Mod note: Couple comments deleted. Please avoid asserting conspiracy stuff as fact without some support, but also please skip insisting that Putin is harmless or whatever; forcing people to focus on weird/glib/contrarian takes doesn't help with an already fraught and confusing subject.
posted by LobsterMitten (staff) at 11:14 AM on December 18, 2020 [7 favorites]

Billions Spent on U.S. Defenses Failed to Detect Giant Russian Hack

Now if this had been a developing country in the global South, it would be obvious where the money went.
posted by infini at 11:18 AM on December 18, 2020 [1 favorite]

So what can be done about it?

Teach the computers to fly airplanes

bring the fight to 'em!
posted by From Bklyn at 11:39 AM on December 18, 2020

“How is this not a massive intelligence failure, particularly since we were supposedly all over Russian threat actors ahead of the election,” Robert K. Knake, a senior Obama administration cybersecurity official, asked on Twitter on Wednesday. “Did the N.S.A. fall in a giant honey pot while the S.V.R.” — Russia’s most sophisticated spying agency — “quietly pillaged” the government and private industry?
posted by vibrotronica at 11:41 AM on December 18, 2020 [3 favorites]

The always terrific historian Heather Cox Richardson notes that the current anti-American, wreck-everything administration (my words, not hers) has been systematically gutting the nation's cybersecurity expertise:

...administration officials have deliberately forced out of CISA key cybersecurity officials. The destruction was so widespread, according to Dr. Josephine Wolff, a professor of cybersecurity policy at Tufts University’s Fletcher School who holds her PhD from the Massachusetts Institute of Technology (MIT), “they signify the systematic decimation of the personnel most directly responsible for protecting critical infrastructure, shielding our elections from interference and guarding the White House’s data, devices and networks.”

Almost exactly a year ago, on December 19, 2019, Wolff warned in the New York Times that “As we head into 2020, worrying about the integrity of our elections, the growing scourge of ransomware and the increasingly sophisticated forms of cyberespionage and cybersabotage being developed by our adversaries, it’s disconcerting to feel that many of our government’s best cybersecurity minds are walking out the front door and leaving behind too few people to monitor what’s coming in our back doors.”

It's all too common among the general populace of the country to ignore or denigrate the expertise and hard work of government employees, but they are protecting the country - or have been.

I hope President Biden can bring lots of them back.
posted by kristi at 11:45 AM on December 18, 2020 [15 favorites]

I was going post a pithy comment about how many places think security and monitoring software is good, so more security and monitoring software must be better

Agree. This is one of the most frustrating parts of the hack to me. The vector of the attack was the expensive security monitoring software. If this were the Middle Ages, the U.S. would be paying a very expensive flea to give us the Bubonic plague.

I agree that software development and cybersecurity is an extremely complicated endeavor, that a determined state actor is incredibly hard to defend against, but there needs to be serious systemic changes to avoid too many of these hacks occurring in the future.

I hope this can be a catalyst towards more insourcing of critical IT. The US Federal government is huge, and its IT needs are unique. It would completely make sense for the GSA to build open source tools for things like network monitoring and security, that could be implemented across the government and by anyone else for whom they worked.

I agree that these are some of the systemic changes needed and hopefully President Biden moves at least partially in this direction.
posted by mundo at 11:54 AM on December 18, 2020 [2 favorites]

What if Putin had convinced Trump

With Donald Trump unable to pay back the 40 billion rubles he owes Russian "bankers", Putin doesn't ask -- or convince -- Putin orders, and Donald Trump obeys like any good little debtor does.
posted by mikelieman at 1:52 PM on December 18, 2020 [1 favorite]

Mod note: Deleted Hilary Clinton derail
posted by Eyebrows McGee (staff) at 2:56 PM on December 18, 2020 [3 favorites]

Billions Spent on U.S. Defenses Failed to Detect Giant Russian Hack

In fairness, it would be difficult for the best captain to steer an aircraft carrier into a server room.
posted by They sucked his brains out! at 7:13 PM on December 18, 2020 [4 favorites]

Re-imaging the servers with open source OSs would be nice, except for the software that runs only under Windows. We're phasing in software that runs on top of Linux, but it's taken eight years already, with at least a year to go, to get our proprietary Windows software rewritten. Changing critical infrastructure isn't a fast or inexpensive job.
posted by lhauser at 8:15 PM on December 18, 2020 [1 favorite]

Password now changed to solarwinds456.

I've got the same combination on my luggage!
posted by zardoz at 10:23 PM on December 18, 2020 [6 favorites]

Trump, contradicting Pompeo, downplays gravity of massive cyberattack against U.S. government, as well as Russia’s role

President Trump on Saturday morning appeared to contradict his top diplomat, tweeting that he has been “fully briefed” and suggesting that “it may be China” that’s responsible for the breaches.

“The Cyber Hack is far greater in the Fake News Media than in actuality,” Trump tweeted, tagging Pompeo. “I have been fully briefed and everything is well under control.”

Trump also speculated with no evidence that the hacks may also have included “a hit on our ridiculous voting machines during the election, which is now obvious that I won big.” Twitter flagged that assertion, saying that “multiple sources called this election differently.”
posted by vibrotronica at 12:19 PM on December 19, 2020 [3 favorites]

Bruce Schneier: The US has suffered a massive cyberbreach. It's hard to overstate how bad it is

We need to adopt a defense-dominant strategy. As computers and the internet become increasingly essential to society, cyber-attacks are likely to be the precursor to actual war. We are simply too vulnerable when we prioritize offense, even if we have to give up the advantage of using those insecurities to spy on others.

Our vulnerability is magnified as eavesdropping may bleed into a direct attack. The SVR’s access allows them not only to eavesdrop, but also to modify data, degrade network performance, or erase entire networks. The first might be normal spying, but the second certainly could be considered an act of war. Russia is almost certainly laying the groundwork for future attack.

This preparation would not be unprecedented. There’s a lot of attack going on in the world. In 2010, the US and Israel attacked the Iranian nuclear program. In 2012, Iran attacked the Saudi national oil company. North Korea attacked Sony in 2014. Russia attacked the Ukrainian power grid in 2015 and 2016. Russia is hacking the US power grid, and the US is hacking Russia’s power grid – just in case the capability is needed someday. All of these attacks began as a spying operation. Security vulnerabilities have real-world consequences.

We’re not going to be able to secure our networks and systems in this no-rules, free-for-all every-network-for-itself world. The US needs to willingly give up part of its offensive advantage in cyberspace in exchange for a vastly more secure global cyberspace. We need to invest in securing the world’s supply chains from this type of attack, and to press for international norms and agreements prioritizing cybersecurity, like the 2018 Paris Call for Trust and Security in Cyberspace or the Global Commission on the Stability of Cyberspace. Hardening widely used software like Orion (or the core internet protocols) helps everyone. We need to dampen this offensive arms race rather than exacerbate it, and work towards cyber peace. Otherwise, hypocritically criticizing the Russians for doing the same thing we do every day won’t help create the safer world in which we all want to live.

posted by tonycpsu at 3:29 PM on December 24, 2020 [7 favorites]

yeah, this Schneier narrative has been echoed in the Dispatch too

The larger context here is that for many reasons—the Snowden revelations, the infamous digital attack on Iranian centrifuges (and other warlike uses of digital weapons), the U.S. “internet freedom” program (which subsidizes tools to circumvent constraints in authoritarian networks), Defend Forward, and more—the United States is widely viewed abroad as the most fearsome global cyber bully. From our adversaries’ perspective, the United States uses its prodigious digital tools, short of war, to achieve whatever advantage it can, and so adversaries feel justified in doing whatever they can as well, often with fewer scruples. We can tell ourselves that our digital exploits in foreign governmental systems serve good ends, and that our adversaries’ exploits in our systems do not, and often that is true. But this moral judgment, and the norms we push around it, have had no apparent influence in tamping down our adversaries’ harmful attacks on our networks—especially since the U.S. approach to norms has been to give up nothing that it wants to do in the digital realm, but at the same time to try to cajole, coerce, or shame our adversaries into not engaging in digital practices that harm the United States.

I don't recall if we had an FPP about it, but earlier in the spring, PPE hijackings were correlated to the use of these advanced tools to identify and pinpoint shipments for intervention in transit. That may have something to do with the increasing feeling of a runaway train manned by Trumpian trolls.
posted by infini at 10:18 AM on December 25, 2020 [1 favorite]

Finland’s Parliament was the target of a cyber attack during the past autumn, according to a statement released by the parliamentary press office on Monday.

general consensus uptil now is that it wasn't the russians next door
posted by infini at 1:52 AM on December 29, 2020