How Target Blew It
March 13, 2014 8:27 AM Subscribe
"The breach could have been stopped there without human intervention. The system has an option to automatically delete malware as it’s detected. But according to two people who audited FireEye's performance after the breach, Target's security team turned that function off." Bloomberg reports today on "Missed Alarms and 40 Million Stolen Credit Card Numbers: How Target Blew It." (The Target breach, previously.)
So many catastrophes depend on the failure of humans to do their jobs.
posted by Ruthless Bunny at 8:48 AM on March 13, 2014 [3 favorites]
posted by Ruthless Bunny at 8:48 AM on March 13, 2014 [3 favorites]
The mistake would seem to be not also outsourcing the operations in Minneapolis.
But if they did that, who would walk around in the skyways having heated discussions about turf wars over obscure metrics?
posted by COBRA! at 8:49 AM on March 13, 2014 [3 favorites]
But if they did that, who would walk around in the skyways having heated discussions about turf wars over obscure metrics?
posted by COBRA! at 8:49 AM on March 13, 2014 [3 favorites]
But if they did that, who would walk around in the skyways having heated discussions about turf wars over obscure metrics?
Mid-level management. We could call it "downsourcing."
posted by GenjiandProust at 8:51 AM on March 13, 2014 [3 favorites]
Mid-level management. We could call it "downsourcing."
posted by GenjiandProust at 8:51 AM on March 13, 2014 [3 favorites]
But, but, the free market! And Government is always failure. Businesses are always the most efficient!
In testimony before Congress, Target has said that it was only after the U.S. Department of Justice notified the retailer about the breach in mid-December that company investigators went back to figure out what happened.
Heh.
posted by Pogo_Fuzzybutt at 8:58 AM on March 13, 2014 [11 favorites]
In testimony before Congress, Target has said that it was only after the U.S. Department of Justice notified the retailer about the breach in mid-December that company investigators went back to figure out what happened.
Heh.
posted by Pogo_Fuzzybutt at 8:58 AM on March 13, 2014 [11 favorites]
Somewhere there is a pea-brained IT manager scared shitless because his mistrust of monitoring software led him to tell his techs not to do anything. "Just another false alarm out of India. Fucking outsourcing."
posted by sonic meat machine at 9:02 AM on March 13, 2014 [11 favorites]
posted by sonic meat machine at 9:02 AM on March 13, 2014 [11 favorites]
Why didn't they just ask the NSA to let them know when someone was logged into their system who shouldn't have been?
posted by blue_beetle at 9:04 AM on March 13, 2014 [4 favorites]
posted by blue_beetle at 9:04 AM on March 13, 2014 [4 favorites]
Had the company’s security team responded when it was supposed to, the theft that has since engulfed Target, touched as many as one in three American consumers, and led to an international manhunt for the hackers never would have happened at all.
I am trying to diagram this sentence and it hurts.
Had the company’s security team responded when it was supposed to, an international manhunt for the hackers never would have happened at all. The theft has since engulfed Target, and touched as many as one in three American consumers.
Okay, now it only hurts because they ignored the alarms from Bangalore, Fire Eye, Symantec, and disabled the effing malware killer.
Carry on.
posted by tilde at 9:07 AM on March 13, 2014 [2 favorites]
I am trying to diagram this sentence and it hurts.
Had the company’s security team responded when it was supposed to, an international manhunt for the hackers never would have happened at all. The theft has since engulfed Target, and touched as many as one in three American consumers.
Okay, now it only hurts because they ignored the alarms from Bangalore, Fire Eye, Symantec, and disabled the effing malware killer.
Carry on.
posted by tilde at 9:07 AM on March 13, 2014 [2 favorites]
I do wonder how many senior decision-makers will get fired over this, but then I think "probably none."
There is a museum in Stockholm that holds the Vasa, a ship that sank about a half hour into its maiden voyage in the early 17th C. By chance, it sank in an area that was deep, cold, salty, and oxygen-poor, so it was very well preserved. They brought it up in the 11950s, and it's a great artifact of the Age of Sail. Also nice is tyhe fact that only one person died in the disaster. Anyway, at the museum, there is a film about why the ship sank. It was built to be the largest gunship in Europe. While under construction, the Dutch started a larger one. So the King of Sweden said "Add more guns." Which increased the weight enough to make it unseaworthy. Afterwards, there was an inquest where the government tried to pin the blame on six different guys. One after another, they were able to produce documents showing that their response to the King's request was "Your Majesty, if we do that, the ship will sink" and his reply "I don't care; add more guns." No one was found responsible for the disaster. These things just happen sometimes.
posted by GenjiandProust at 9:08 AM on March 13, 2014 [50 favorites]
There is a museum in Stockholm that holds the Vasa, a ship that sank about a half hour into its maiden voyage in the early 17th C. By chance, it sank in an area that was deep, cold, salty, and oxygen-poor, so it was very well preserved. They brought it up in the 11950s, and it's a great artifact of the Age of Sail. Also nice is tyhe fact that only one person died in the disaster. Anyway, at the museum, there is a film about why the ship sank. It was built to be the largest gunship in Europe. While under construction, the Dutch started a larger one. So the King of Sweden said "Add more guns." Which increased the weight enough to make it unseaworthy. Afterwards, there was an inquest where the government tried to pin the blame on six different guys. One after another, they were able to produce documents showing that their response to the King's request was "Your Majesty, if we do that, the ship will sink" and his reply "I don't care; add more guns." No one was found responsible for the disaster. These things just happen sometimes.
posted by GenjiandProust at 9:08 AM on March 13, 2014 [50 favorites]
I've never worked with Target, but I have a large retail customer who have a very similar environment and they initially had a problem where FireEye was reporting a very high number of false positives because the VMs running under FireEye were not patched to the same level as the production servers. This led the SOC to pretty much ignore FireEye alerts, because they had learned that the alerts were always bogus. It is possible that Target was also having some learning curve issues with their FireEye install and that the real alert was lost in a sea of bogus ones. This would also explain why the auto-delete feature was not used -- it probably deleted legitimate code that caused all sorts of problems for their applications.
posted by Lame_username at 9:09 AM on March 13, 2014 [14 favorites]
posted by Lame_username at 9:09 AM on March 13, 2014 [14 favorites]
Yeah, the first thing that jumped out at me when reading the article was "of course they didn't set it to auto-delete, because false positives always happen." And, though I don't have specific experience with FireEye, I do develop software that can emit a ton of alerts if not configured correctly, and rarely do users spend enough time curating their whitelisting/blacklisting configuration enough to make the volume of alerts small enough that humans can be expected to respond to every one.
It's very hard to get right. I'm not excusing Target for their mistakes, but the fine article implies that there are appliances out there that have an "Easy button" that makes it trivial to filter out malware without side effects, and that's not even close to being true.
posted by tonycpsu at 9:13 AM on March 13, 2014 [13 favorites]
It's very hard to get right. I'm not excusing Target for their mistakes, but the fine article implies that there are appliances out there that have an "Easy button" that makes it trivial to filter out malware without side effects, and that's not even close to being true.
posted by tonycpsu at 9:13 AM on March 13, 2014 [13 favorites]
I am trying to diagram this sentence and it hurts.
Had the company’s security team responded when it was supposed to, an international manhunt for the hackers never would have happened at all. The theft has since engulfed Target, and touched as many as one in three American consumers.
I don't think that's right.
Had the company’s security team responded when it was supposed to, the theft that has since engulfed Target, touched as many as one in three American consumers, and led to an international manhunt for the hackers never would have happened at all.
The basic sentence here is: "Had the company’s security team responded when it was supposed to the theft never would have happened at all." All of this stuff--"that has since engulfed Target, touched as many as one in three American consumers, and led to an international manhunt for the hackers"--just modifies "theft."
posted by yoink at 9:15 AM on March 13, 2014 [4 favorites]
Had the company’s security team responded when it was supposed to, an international manhunt for the hackers never would have happened at all. The theft has since engulfed Target, and touched as many as one in three American consumers.
I don't think that's right.
Had the company’s security team responded when it was supposed to, the theft that has since engulfed Target, touched as many as one in three American consumers, and led to an international manhunt for the hackers never would have happened at all.
The basic sentence here is: "Had the company’s security team responded when it was supposed to the theft never would have happened at all." All of this stuff--"that has since engulfed Target, touched as many as one in three American consumers, and led to an international manhunt for the hackers"--just modifies "theft."
posted by yoink at 9:15 AM on March 13, 2014 [4 favorites]
Gah! Come on! (I can't be the only person disgusted by that gif?)
posted by Auden at 9:17 AM on March 13, 2014 [3 favorites]
posted by Auden at 9:17 AM on March 13, 2014 [3 favorites]
One after another, they were able to produce documents showing that their response to the King's request was "Your Majesty, if we do that, the ship will sink" and his reply "I don't care; add more guns." No one was found responsible for the disaster. These things just happen sometimes.
I think it would be truer to say "the King was found responsible for the disaster, but the King cannot be held responsible for the disaster." Which, perhaps, was your point.
posted by yoink at 9:17 AM on March 13, 2014 [4 favorites]
I think it would be truer to say "the King was found responsible for the disaster, but the King cannot be held responsible for the disaster." Which, perhaps, was your point.
posted by yoink at 9:17 AM on March 13, 2014 [4 favorites]
I do wonder how many senior decision-makers will get fired over this, but then I think "probably none."
The CIO has already resigned with some speculation that she was pushed out.
posted by Area Man at 9:18 AM on March 13, 2014 [3 favorites]
The CIO has already resigned with some speculation that she was pushed out.
posted by Area Man at 9:18 AM on March 13, 2014 [3 favorites]
I heard from my guy inside the card industry that the rumor going around is one of the admins screwed up and used the same password for Facebook as he did for his network login. Laughable, of course, but that's what people on the non-technical side of that business have heard was the cause.
posted by ob1quixote at 9:18 AM on March 13, 2014 [1 favorite]
posted by ob1quixote at 9:18 AM on March 13, 2014 [1 favorite]
No one was found responsible for the disaster. These things kings just happen sometimes.
posted by Kirth Gerson at 9:21 AM on March 13, 2014 [8 favorites]
posted by Kirth Gerson at 9:21 AM on March 13, 2014 [8 favorites]
I heard from my guy inside the card industry that the rumor going around is one of the admins screwed up and used the same password for Facebook as he did for his network login.
If Target was seriously using passwords for authentication into their network, they deserve to fail. The companies I've worked at all use SSH keys, plus passwords, plus 2FA for particularly sensitive areas, and usually connections have to go through a bastion host anyway, which has whitelisted IPs.
The idea that Target was doing less than that is frightening.
posted by sonic meat machine at 9:23 AM on March 13, 2014 [4 favorites]
If Target was seriously using passwords for authentication into their network, they deserve to fail. The companies I've worked at all use SSH keys, plus passwords, plus 2FA for particularly sensitive areas, and usually connections have to go through a bastion host anyway, which has whitelisted IPs.
The idea that Target was doing less than that is frightening.
posted by sonic meat machine at 9:23 AM on March 13, 2014 [4 favorites]
The CIO has already resigned with some speculation that she was pushed out.
I wonder how big her firing bonus was.
posted by dirigibleman at 9:34 AM on March 13, 2014 [6 favorites]
I wonder how big her firing bonus was.
posted by dirigibleman at 9:34 AM on March 13, 2014 [6 favorites]
The CIO has already resigned with some speculation that she was pushed out.
The CIO with the degree in merchandising and MBA and absolutely no technical background or experience...
posted by gyc at 9:37 AM on March 13, 2014 [1 favorite]
The CIO with the degree in merchandising and MBA and absolutely no technical background or experience...
posted by gyc at 9:37 AM on March 13, 2014 [1 favorite]
My credit card was compromised and I have been following this. I would be interested in a summary of how this works. The best I have seen is Krebs on security here and here. In my case the end user criminal was somebody in my city who used the fake credit card about five times for around 250 dollars before Visa's weird usage big brother program shut down the card. That seems like a huge risk to take for small rewards. Committing felonies in Texas is about the stupidest money making scheme I could think of.
The bank didn't even ask me to call the police. I wonder if the computer network has an alert that the customer is trying to use a stolen card, not just a card that might be over its limit or whatever, and they call the police and send them to arrest the guy. How hard could it be to do this?
posted by bukvich at 9:47 AM on March 13, 2014 [1 favorite]
The bank didn't even ask me to call the police. I wonder if the computer network has an alert that the customer is trying to use a stolen card, not just a card that might be over its limit or whatever, and they call the police and send them to arrest the guy. How hard could it be to do this?
posted by bukvich at 9:47 AM on March 13, 2014 [1 favorite]
I'm not saying I believe it, sonic meat machine. At the very least I would expect them to be using two-factor authentication. I just thought it was interesting that the sales and marketing side of the card industry doesn't hear that cheaping out on IT is what caused this problem. It was, "The nerds screwed up. As usual." Just like that deposit problem from last year.
posted by ob1quixote at 9:52 AM on March 13, 2014 [1 favorite]
posted by ob1quixote at 9:52 AM on March 13, 2014 [1 favorite]
Ten years ago or so, I worked in Information Security for a bank based just a couple of blocks from Target's HQ.
It was almost impossible to get anyone to take security seriously.
- The IDS system was set up by one guy, not configured very well, and not backed up.
- No one paid any attention to the alerts from the IDS; there wasn't enough network staff, so no one had the time to really learn how to investigate alerts.
- Tellers were granted access by faxing requests to IS, with their (four digit!) password plain on the page.
- Branch managers freely shared their passwords with their staff, which we discovered anew every time we did an audit, but no one cared.
- SFTP passwords for inter-bank file transfers were in a plain text file stored in the mainframe, and contained clever passwords like BANKNAME1.
The list goes on and on.
I'm not at all surprised that breaches like this happen. Security is hard, and it doesn't make money, so it's not prioritized.
posted by Ickster at 9:54 AM on March 13, 2014 [5 favorites]
It was almost impossible to get anyone to take security seriously.
- The IDS system was set up by one guy, not configured very well, and not backed up.
- No one paid any attention to the alerts from the IDS; there wasn't enough network staff, so no one had the time to really learn how to investigate alerts.
- Tellers were granted access by faxing requests to IS, with their (four digit!) password plain on the page.
- Branch managers freely shared their passwords with their staff, which we discovered anew every time we did an audit, but no one cared.
- SFTP passwords for inter-bank file transfers were in a plain text file stored in the mainframe, and contained clever passwords like BANKNAME1.
The list goes on and on.
I'm not at all surprised that breaches like this happen. Security is hard, and it doesn't make money, so it's not prioritized.
posted by Ickster at 9:54 AM on March 13, 2014 [5 favorites]
The idea that Target was doing less than that is frightening.
Fun ads, though.
posted by Blazecock Pileon at 9:55 AM on March 13, 2014
Fun ads, though.
posted by Blazecock Pileon at 9:55 AM on March 13, 2014
The best I have seen is Krebs on security here and here.
Thanks for that. My card was compromised at Target (the one time I have shopped at a Target in three years!) and I have been curious about this as well. It's interesting how widely varied the cards can be about fraud prevention. I had an old credit card via a credit union that would basically shut my card down if I traveled and forgot to tell them I was traveling (my bad, but the first time was sure a surprise) whereas I've had other cards where I didn't even get notified when someone did that small purchase/large purchase thing at a state I hadn't traveled to. Add to this you sort of wish the algorithms were smarter. Like "Hey I just bought a plane ticket to Alaska three weeks ago, maybe you can not shut my credit card down for trying to check into an Alaskan hotel today..." Like I get why fraud protection is important, I'm just always surprised how dumb it is and there appears to be little incentive for it to get better.
Someone on MeFi suggested the book Kingpin: How One Hacker Took Over the Billion-Dollar Cybercrime Underground which talks a lot, an awful lot, about the sort of backchannel black market business that is selling stolen CC numbers. Good reading.
posted by jessamyn at 9:56 AM on March 13, 2014 [6 favorites]
Thanks for that. My card was compromised at Target (the one time I have shopped at a Target in three years!) and I have been curious about this as well. It's interesting how widely varied the cards can be about fraud prevention. I had an old credit card via a credit union that would basically shut my card down if I traveled and forgot to tell them I was traveling (my bad, but the first time was sure a surprise) whereas I've had other cards where I didn't even get notified when someone did that small purchase/large purchase thing at a state I hadn't traveled to. Add to this you sort of wish the algorithms were smarter. Like "Hey I just bought a plane ticket to Alaska three weeks ago, maybe you can not shut my credit card down for trying to check into an Alaskan hotel today..." Like I get why fraud protection is important, I'm just always surprised how dumb it is and there appears to be little incentive for it to get better.
Someone on MeFi suggested the book Kingpin: How One Hacker Took Over the Billion-Dollar Cybercrime Underground which talks a lot, an awful lot, about the sort of backchannel black market business that is selling stolen CC numbers. Good reading.
posted by jessamyn at 9:56 AM on March 13, 2014 [6 favorites]
While not a complete solution, if the USA were using smart chip + pin cards like everyone else in the world this heist would have been far more difficult to pull off, even with a bungling retailer. I know from experience that some retailers abroad no longer accept dumb American cards because the risk of fraud is too high. Instead of fixing the problem, your bank's response will be "please report the retailer." OK dude, I report Belgium. Now what?
posted by 1adam12 at 10:04 AM on March 13, 2014 [4 favorites]
posted by 1adam12 at 10:04 AM on March 13, 2014 [4 favorites]
Like "Hey I just bought a plane ticket to Alaska three weeks ago, maybe you can not shut my credit card down for trying to check into an Alaskan hotel today..."
Seriously. I pre-paid for two weeks of hotels in Spain, airfare, a rental car, and an Alhambra ticket months in advance of my actual trip there, but my card got shut down 1.5 weeks into a two-week trip when the company noticed me actually buying things in Spain.
posted by LionIndex at 10:17 AM on March 13, 2014 [1 favorite]
Seriously. I pre-paid for two weeks of hotels in Spain, airfare, a rental car, and an Alhambra ticket months in advance of my actual trip there, but my card got shut down 1.5 weeks into a two-week trip when the company noticed me actually buying things in Spain.
posted by LionIndex at 10:17 AM on March 13, 2014 [1 favorite]
Echoing tonycpsu on "there is no easy button". Multiple times in the past few years, the security software stack at the company I work for has suddenly blocked packaged software from major vendors as "malware" (including business critical financial, order, and manufacturing execution applications). The downtime until the Info Security team was able to unblock the apps was bad enough. I can't even imagine the chaos if they had also been autoremoved, bringing the company to a halt until we could get everything reinstalled on thousands of computers across the US and Canada.
posted by superna at 10:22 AM on March 13, 2014 [2 favorites]
posted by superna at 10:22 AM on March 13, 2014 [2 favorites]
Chip and Pin is Broken (pdf). Please be careful what you wish for. Chip and pin would prevent this, however the original point of chip and pin was to make end users responsible for theft. The system we have right now is obviously non-optimal, but has the really nice benefit of not holding end users responsible for thefts. This might not be the case with chip and pin.
The rumour I heard was a contractor Target employed unknowingly violated their airgaps by plugging stuff into the wrong switch. Just a rumour.
A lot of this can be seen as a failing of computer engineering. If we had patches which we had faith in their ability not to break things by accident, or low enough false positive rates (which admittedly is a very, very hard problem) we could in theory deploy more automatic updates. So yeah, as others have pointed out while they maybe dumb, they're not criminally dumb.
It's also interesting to see how trends in information security shift over time. The current rage is data exfiltration detection. While this partially makes sense, I'm very uneasy with it as a primary means of compromise detection.
posted by yeahwhatever at 10:33 AM on March 13, 2014 [2 favorites]
The rumour I heard was a contractor Target employed unknowingly violated their airgaps by plugging stuff into the wrong switch. Just a rumour.
A lot of this can be seen as a failing of computer engineering. If we had patches which we had faith in their ability not to break things by accident, or low enough false positive rates (which admittedly is a very, very hard problem) we could in theory deploy more automatic updates. So yeah, as others have pointed out while they maybe dumb, they're not criminally dumb.
It's also interesting to see how trends in information security shift over time. The current rage is data exfiltration detection. While this partially makes sense, I'm very uneasy with it as a primary means of compromise detection.
posted by yeahwhatever at 10:33 AM on March 13, 2014 [2 favorites]
Great. Now what retail outlet am I going to turn my paycheck over to in order to keep the economy moving?
posted by Mental Wimp at 10:35 AM on March 13, 2014 [1 favorite]
posted by Mental Wimp at 10:35 AM on March 13, 2014 [1 favorite]
Like "Hey I just bought a plane ticket to Alaska three weeks ago, maybe you can not shut my credit card down for trying to check into an Alaskan hotel today..."
AmEx does a pretty good job.
posted by Mental Wimp at 10:38 AM on March 13, 2014
AmEx does a pretty good job.
posted by Mental Wimp at 10:38 AM on March 13, 2014
Bangalore got an alert and flagged the security team in Minneapolis. And then … nothing happened.
Without knowing the false-positive rate it's hard to pin the blame on Minneapolis based on this statement. Did they send the one valid alert along with 10,000 invalid ones, so that it was a needle in a haystack? I've seen server monitoring systems that churned out so many alerts that everyone became inured to receiving them (mostly creating rules to send the alerts directly to the trash), which is worse than useless.
posted by Kadin2048 at 10:53 AM on March 13, 2014 [1 favorite]
Without knowing the false-positive rate it's hard to pin the blame on Minneapolis based on this statement. Did they send the one valid alert along with 10,000 invalid ones, so that it was a needle in a haystack? I've seen server monitoring systems that churned out so many alerts that everyone became inured to receiving them (mostly creating rules to send the alerts directly to the trash), which is worse than useless.
posted by Kadin2048 at 10:53 AM on March 13, 2014 [1 favorite]
AmEx does a pretty good job
Eh, not for me they didn't. A few years back I bought a series of plane tickets, OK, fine. Then proceeded to make a hotel reservation for the first stop ... seemingly fine.
...but not actually fine at all, because they decided to lock down the card a couple of days later with no intervening purchases (ie., either the tickets or the hotel set off their alarms, but they did nothing about it until a few days had elapsed?).
Sure, lock down the card immediately & let me know -- that's reasonable enough. Waiting three days to say/do anything? Not cool, and I still don't trust them. Maybe they've improved?
posted by aramaic at 10:54 AM on March 13, 2014
Eh, not for me they didn't. A few years back I bought a series of plane tickets, OK, fine. Then proceeded to make a hotel reservation for the first stop ... seemingly fine.
...but not actually fine at all, because they decided to lock down the card a couple of days later with no intervening purchases (ie., either the tickets or the hotel set off their alarms, but they did nothing about it until a few days had elapsed?).
Sure, lock down the card immediately & let me know -- that's reasonable enough. Waiting three days to say/do anything? Not cool, and I still don't trust them. Maybe they've improved?
posted by aramaic at 10:54 AM on March 13, 2014
This whole thing is gold. Just from top to bottom. I struggle to see even one additional way Target could have made a mistake in their security beyond storing passwords in plain text.I don't want to be an apologist for Target here, because they obviously blew it, but the simple truth is that Target was doing a way better job than 90% of retailers. They have a dedicated security team of several hundred guys, they run a 24/7 intrusion detection center and have a dedicated SOC as well apparently a commitment to running state of the art tools. I can name dozens of retailers whose idea of network security is that they have a firewall and open source virus protection and who have one guy who does all the functions of a SOC.
posted by Lame_username at 10:58 AM on March 13, 2014 [3 favorites]
Did they send the one valid alert along with 10,000 invalid ones, so that it was a needle in a haystack?
Every fourth time I go to Target the shoplifting alarm goes off when I leave and I have never paused and I have never been questioned.
posted by bukvich at 11:17 AM on March 13, 2014
Every fourth time I go to Target the shoplifting alarm goes off when I leave and I have never paused and I have never been questioned.
posted by bukvich at 11:17 AM on March 13, 2014
Deterring shoplifters and preventing information security breaches are apples and oranges.
Also, you probably have a magnetic tag hidden in your jeans that are triggering the alarm.
posted by VTX at 11:43 AM on March 13, 2014 [2 favorites]
Also, you probably have a magnetic tag hidden in your jeans that are triggering the alarm.
posted by VTX at 11:43 AM on March 13, 2014 [2 favorites]
sonic meat machine: "Somewhere there is a pea-brained IT manager scared shitless because his mistrust of monitoring software led him to tell his techs not to do anything. "Just another false alarm out of India. Fucking outsourcing.""
There is a thing called alert fatigue. When every fucking failed ssh login and high server load causes an alert, it's a very short mental trip to having no alerts at all.
However, there is a way to ameliorate some of the above cited problems of automating responses to malware alerts. You could, for example, be assured that removing a file does not break production because you know files were not present on the continuous integration / testing server, and all new files would have to be put through first, right? And you could have some measure of certainty that changes to the monitoring system continue to work by running a test instance that monitors production in read only mode, and making sure it doesn't mark anything for deletion after the upgrade.
posted by pwnguin at 11:55 AM on March 13, 2014 [2 favorites]
There is a thing called alert fatigue. When every fucking failed ssh login and high server load causes an alert, it's a very short mental trip to having no alerts at all.
However, there is a way to ameliorate some of the above cited problems of automating responses to malware alerts. You could, for example, be assured that removing a file does not break production because you know files were not present on the continuous integration / testing server, and all new files would have to be put through first, right? And you could have some measure of certainty that changes to the monitoring system continue to work by running a test instance that monitors production in read only mode, and making sure it doesn't mark anything for deletion after the upgrade.
posted by pwnguin at 11:55 AM on March 13, 2014 [2 favorites]
> There is a thing called alert fatigue. When every fucking failed ssh login and high server load causes an alert, it's a very short mental trip to having no alerts at all.
The idea is that you shake these out in testing, though. Basically, each alert should either result in "We have an actual attack," or, "How do we prevent a false positive from happening again?"
If you alert on a single failed ssh login, your security system is broken. And it's pretty easy to set that threshold high without issues - because either the attacker has the right ssh credentials, in which case you'll never see a failed ssh, or they are using some variety of trial and error, when you'll see thousands if not millions or billions of ssh failures.
High server loads should always result in an operator warning, however - it's always a sign of something going wrong, though it's usually a software issue and not an intrusion. You should provision your machines so that you never ever get high server loads - because it's rare in a correctly configured commercial server situation that CPU is a limiting factor, it's nearly always RAM.
posted by lupus_yonderboy at 12:04 PM on March 13, 2014 [3 favorites]
The idea is that you shake these out in testing, though. Basically, each alert should either result in "We have an actual attack," or, "How do we prevent a false positive from happening again?"
If you alert on a single failed ssh login, your security system is broken. And it's pretty easy to set that threshold high without issues - because either the attacker has the right ssh credentials, in which case you'll never see a failed ssh, or they are using some variety of trial and error, when you'll see thousands if not millions or billions of ssh failures.
High server loads should always result in an operator warning, however - it's always a sign of something going wrong, though it's usually a software issue and not an intrusion. You should provision your machines so that you never ever get high server loads - because it's rare in a correctly configured commercial server situation that CPU is a limiting factor, it's nearly always RAM.
posted by lupus_yonderboy at 12:04 PM on March 13, 2014 [3 favorites]
One method that I've seen in virtualized environments is to have the staging servers "step into" a production role. Thus, you stage the servers, test them, and so on, then move them into the production rota. While they're still in staging, you take memory profiles and checksums of various aspects of the system, and then you can check that periodically after they're production machines. That will lead to a pretty robust monitoring system tailored to the VM itself.
posted by sonic meat machine at 12:38 PM on March 13, 2014 [1 favorite]
posted by sonic meat machine at 12:38 PM on March 13, 2014 [1 favorite]
lupus_yonderboy: " And it's pretty easy to set that threshold high without issues - because either the attacker has the right ssh credentials, in which case you'll never see a failed ssh, or they are using some variety of trial and error, when you'll see thousands if not millions or billions of ssh failures."
And this just makes the case for automation, which is why my servers run either fail2ban or DenyHosts. And PasswordAuth is disabled anyways, so this is just an extra layer of defense.
lupus_yonderboy: "High server loads should always result in an operator warning, however - it's always a sign of something going wrong, though it's usually a software issue and not an intrusion."
High server load is the wrong thing to measure. CPUs are not an insurance policy that you bought but never hope to use. If anything, you should be alerting on low CPU utilization. Monitor the reason you bought new / more computers in the first place: service response times. Because 'the website takes a minute to load' is a hell of a lot more relevant than CPU load when answering the question "should I wake up the on-call tech?"
posted by pwnguin at 12:53 PM on March 13, 2014 [1 favorite]
And this just makes the case for automation, which is why my servers run either fail2ban or DenyHosts. And PasswordAuth is disabled anyways, so this is just an extra layer of defense.
lupus_yonderboy: "High server loads should always result in an operator warning, however - it's always a sign of something going wrong, though it's usually a software issue and not an intrusion."
High server load is the wrong thing to measure. CPUs are not an insurance policy that you bought but never hope to use. If anything, you should be alerting on low CPU utilization. Monitor the reason you bought new / more computers in the first place: service response times. Because 'the website takes a minute to load' is a hell of a lot more relevant than CPU load when answering the question "should I wake up the on-call tech?"
posted by pwnguin at 12:53 PM on March 13, 2014 [1 favorite]
> CPUs are not an insurance policy that you bought but never hope to use.
Not at all, CPUs are vehicles for deploying RAM onto the Internet. A correctly-designed e-commerce site makes few demands on the CPU, so excessive CPU load is a red flag.
> If anything, you should be alerting on low CPU utilization.
That seems a bit crazy. You need to be provisioned for peak usage and then some, so there will be plenty of times when your overall usage will be an order of magnitude less than that - unless your load balancers are perfectly tuned for low-traffic conditions, it's likely that some of your servers will be getting almost no traffic (and why spend any time tuning for low-traffic conditions anyway?)
> Monitor the reason you bought new / more computers in the first place: service response times.
You should monitor that too - even more so!
posted by lupus_yonderboy at 1:25 PM on March 13, 2014
Not at all, CPUs are vehicles for deploying RAM onto the Internet. A correctly-designed e-commerce site makes few demands on the CPU, so excessive CPU load is a red flag.
> If anything, you should be alerting on low CPU utilization.
That seems a bit crazy. You need to be provisioned for peak usage and then some, so there will be plenty of times when your overall usage will be an order of magnitude less than that - unless your load balancers are perfectly tuned for low-traffic conditions, it's likely that some of your servers will be getting almost no traffic (and why spend any time tuning for low-traffic conditions anyway?)
> Monitor the reason you bought new / more computers in the first place: service response times.
You should monitor that too - even more so!
posted by lupus_yonderboy at 1:25 PM on March 13, 2014
And to reiterate, you should be monitoring pretty well any condition that affects your end users and either alerting or at least aggregating for a daily report.
If more people than usual, say, open shopping carts and never check out, you need to know this fact - you need to be somewhat creative in coming up with conditions that might indicate "badness" and alerting on them (another example might be "many searches where the user never actually clicks on a result").
posted by lupus_yonderboy at 1:30 PM on March 13, 2014 [1 favorite]
If more people than usual, say, open shopping carts and never check out, you need to know this fact - you need to be somewhat creative in coming up with conditions that might indicate "badness" and alerting on them (another example might be "many searches where the user never actually clicks on a result").
posted by lupus_yonderboy at 1:30 PM on March 13, 2014 [1 favorite]
You need to be provisioned for peak usage and then some...
This depends on your provisioning solution. If you're using cloud infrastructure (AWS et al.), you provision for only slightly more than is currently being used, and when it starts to hit a threshold you autoscale. This is one of the key selling points of that type of infrastructure, and it can even be used in-house if you don't trust Amazon (which Target shouldn't, obviously).
posted by sonic meat machine at 1:36 PM on March 13, 2014
This depends on your provisioning solution. If you're using cloud infrastructure (AWS et al.), you provision for only slightly more than is currently being used, and when it starts to hit a threshold you autoscale. This is one of the key selling points of that type of infrastructure, and it can even be used in-house if you don't trust Amazon (which Target shouldn't, obviously).
posted by sonic meat machine at 1:36 PM on March 13, 2014
Waiting three days to say/do anything? Not cool, and I still don't trust them. Maybe they've improved?
Yeah, maybe. They notified me when I was in Japan, not about my Japanese purchases but because someone had tried to buy something in Kansas City using the card. They knew I was in Japan because I had bought the ticket and had a string of contiguous charges from there.
So, so far, so good for me. Sorry to hear they didn't do well by you.
posted by Mental Wimp at 2:24 PM on March 13, 2014
Yeah, maybe. They notified me when I was in Japan, not about my Japanese purchases but because someone had tried to buy something in Kansas City using the card. They knew I was in Japan because I had bought the ticket and had a string of contiguous charges from there.
So, so far, so good for me. Sorry to hear they didn't do well by you.
posted by Mental Wimp at 2:24 PM on March 13, 2014
Same thing...I take my Strida 5.0 folding bicycle into my local small-ish town library on a Saturday morning and leave it in the foyer--directly under the surveilance cameras. Its stolen and the head of IT says the video recorder was turned off! Dammit!
Oh what were we talking about... :P
posted by xtian at 2:32 PM on March 13, 2014
Oh what were we talking about... :P
posted by xtian at 2:32 PM on March 13, 2014
I deployed Fire Eye here over a year ago (admittedly, in a much smaller environment). After a little tuning on the FE side, a little more tuning on the log management side, we *never* ignore a FireEye alert that bubbles all the way up. Probably the best money I've spent on Infosec technology in the last 2 years. Blaming technology for what is very clearly a failure of internal process and procedure, as well as oversight, is an all to common for teams that are bad at their job.
posted by kjs3 at 2:35 PM on March 13, 2014 [3 favorites]
posted by kjs3 at 2:35 PM on March 13, 2014 [3 favorites]
Have the nitwits who DEFRAUDED their credit card processor and demonstrably lied on their multiple, annual PCI Compliance Audits been arrested yet?
posted by mikelieman at 3:39 PM on March 13, 2014
posted by mikelieman at 3:39 PM on March 13, 2014
I'm sure they'll just ask their E&O insurer to cover that particular bill :P
posted by pwnguin at 5:08 PM on March 13, 2014 [1 favorite]
posted by pwnguin at 5:08 PM on March 13, 2014 [1 favorite]
Have the nitwits who DEFRAUDED their credit card processor and demonstrably lied on their multiple, annual PCI Compliance Audits been arrested yet?
Nitwit frauds usually have immunity.
posted by homunculus at 9:35 PM on March 13, 2014
Nitwit frauds usually have immunity.
posted by homunculus at 9:35 PM on March 13, 2014
Trustwave sued in conjuction with Target breach
Trustwave was also the PCI auditor for Heartland, who suffered an at-the-time large breach in 2009. ("Don't Hire a QSA by Seeking the Lowest Bid, Warns Heartland's Carr")
posted by These Premises Are Alarmed at 1:32 PM on March 25, 2014 [1 favorite]
Trustwave was also the PCI auditor for Heartland, who suffered an at-the-time large breach in 2009. ("Don't Hire a QSA by Seeking the Lowest Bid, Warns Heartland's Carr")
posted by These Premises Are Alarmed at 1:32 PM on March 25, 2014 [1 favorite]
Blogger who broke Target breach to be subject of movie
posted by KatlaDragon at 4:58 AM on March 26, 2014
posted by KatlaDragon at 4:58 AM on March 26, 2014
Pretty sure Krebs has done more than blog about this Target thing. But I guess it's about the only thing the average sfgate reader would know about.
posted by pwnguin at 4:09 PM on March 26, 2014
posted by pwnguin at 4:09 PM on March 26, 2014
« Older Trains Are Wonderful And People…? | found mike Newer »
This thread has been archived and is closed to new comments
The mistake would seem to be not also outsourcing the operations in Minneapolis.
posted by three blind mice at 8:41 AM on March 13, 2014 [18 favorites]