Total Protection ... from a usable computer
April 21, 2010 12:26 PM   Subscribe

McAfee's latest DAT update quarantined the svchost.exe file on millions (or maybe 800,000) of corporate Windows XP systems, rendering them inoperable (sort of).
posted by mrgrimm (120 comments total) 14 users marked this as a favorite
 
On nearly every computer I've owned or operated since 1993, virus scanners have proven far more harmful than the viruses themselves. ESPECIALLY on servers.
posted by Afroblanco at 12:27 PM on April 21, 2010 [14 favorites]


That's hilariously awesome (as long as you neither work at McAfee nor as a corporate tech support bot).
posted by kavasa at 12:28 PM on April 21, 2010


Funny, we were just talking about this. Lots of corp support uses mcafee (for whatever reason). Manual fix to get it back up and running again; or so it's reported by the fixers. Have to find a good copy of svchost and replace.
posted by LD Feral at 12:30 PM on April 21, 2010


I do support for a document management software suite, and we include a full-text search component. I cannot tell you how many times I have ad to get customers to exempt our software's directories from their antivirus scans because they tend to corrupt the full-text indexes. I always end up describing it like this: "Your antivirus is like an over-zealous mall security guard. It'l try to stop you from entering your own store."

And the rep McAfee has earned is that is waaaay worse about this than Symantec is.
posted by grubi at 12:37 PM on April 21, 2010 [4 favorites]


I am of the distinct opinion that Antivirus programs are simply extra-strong viruses that you knowingly install on your computer to bully out all of the other weaker viruses.

I may have taken this opinion wholesale from a source I cannot remember. It may be mine. Can't remember.
posted by The Esteemed Doctor Bunsen Honeydew at 12:38 PM on April 21, 2010 [38 favorites]


On nearly every computer I've owned or operated since 1993, virus scanners have proven far more harmful than the viruses themselves. ESPECIALLY on servers.

I dunno, if XP gets good and hosed by Malware, particularly malware that downloads other malware, you'll be picking bits out of it for so long that you might as well just reinstall.

Of course, I'm not actually sure that McAfee actually does anything to prevent that.
posted by Artw at 12:41 PM on April 21, 2010


I am of the distinct opinion that Antivirus programs are simply extra-strong viruses that you knowingly install on your computer to bully out all of the other weaker viruses.

Back in the Amiga days there were boot sector "anti-viruses" just like this. They were of course just as bad as regular viruses.
posted by Artw at 12:42 PM on April 21, 2010


The most secure computer in the world is one that won't boot.

Which I recommend McAfee Anti-Virus Suite.

shamelessly stolen
posted by There's No I In Meme at 12:42 PM on April 21, 2010


Okay, on a more serious note: While this is pretty much the worst-case false positive I can think of, and I'll certainly think twice about installing McAfee on any box in the future, I think claiming this incident justifies the non-use of antivirus software is a little bit silly.
posted by The Lurkers Support Me in Email at 12:43 PM on April 21, 2010 [1 favorite]


I've nuked it with every computer purchase, so I had to look it up on wikipedia, where I was amused by what will likely be a short-lived bit o' vandalism regarding this event.
posted by M.C. Lo-Carb! at 12:47 PM on April 21, 2010


"University of Michigan officials are working to find a major computer fix after nearly a third of Health System and Medical School computers went down around 11 a.m. on Wednesday.... about 8,000 of 25,000 computers in the system were taken offline by the problem, she said."

That sucks pretty hard. They're claiming the patients are unaffected, at least.
posted by ardgedee at 12:48 PM on April 21, 2010 [1 favorite]


My company shut down the update after it went out to some percentage of the company (people were getting endless reboots without warning). Does McAfee not even test their updates before sending them out? Hilariously inept.
posted by haveanicesummer at 12:48 PM on April 21, 2010


> on wikipedia, where I was amused by what will likely be a short-lived bit o' vandalism regarding this event.

For posterity:
"On April 21, 2010, beginning approximately at 2 PM GMT, millions of computers worldwide running Windows XP Service Pack 3 were affected by an erroneous virus definition file update by McAfee, resulting in the destruction of a Windows system file (svchost.exe) on those machines. Global chaos ensued. In 2067, this day was remembered in the feature movie The Day the Earth Stood Still. The movie became the biggest box office hit of that year, with revenues of 428.5 trillion UWD (united world dollars)."
posted by ardgedee at 12:51 PM on April 21, 2010 [15 favorites]


Is there a cool name for software that "destroys that village in order to save it" along the lines of malware, adware, ransomware, etc.?
posted by tommasz at 12:52 PM on April 21, 2010


In fairness svchost,exe is always doing all kinds of mysterious shit on my POC and I often an suspicious of it.
posted by GuyZero at 12:54 PM on April 21, 2010 [5 favorites]


Honestly, I can't get too worked up about this in terms of anti-virus in general (I would be pretty worked up if I were using McAfee, though). This sort of anti-virus will be necessary as long as you have users who are afraid of computers. I don't think there will ever be a satisfactory solution short of requiring computer operation licenses.
posted by charred husk at 12:58 PM on April 21, 2010


The Lurkers Support Me in Email: "Okay, on a more serious note: While this is pretty much the worst-case false positive I can think of, and I'll certainly think twice about installing McAfee on any box in the future, I think claiming this incident justifies the non-use of antivirus software is a little bit silly."

It certainly is a vindicating moment for the viewpoint. Though this is a one-off thing, the other disadvantages of anti-viruses aren't hard to look for. Boot time, system resources, having to constantly maintain the thing that is supposed to be maintaining you computer (would you like to schedule updates? It has been 2 days since your last full system scan! I have detected that you are attempting to use a macro in Office. Should I go ahead and block that for you?). Oh, I also forgot the most important part: subscription fees! Without paying the monthly or yearly fee or whatever, the software won't be updated and people will keep making new viruses and forms of malware.

This could turn into an AskMe thread, but here's my foolproof solution for not inflicting upon yourself either a virus or an anti-virus: Do not download anything you even remotely suspect to be unsafe. If you are in the habit of going to possibly unsafe parts of the internet, install NoScript. Clickjacking is something you should look out for. Connect to the internet through a router or a modem that has a built-in firewall. Are you doing these things? Good! You are now safe from viruses. And by the way: there are enough YouTube-like porn websites out there now that your personal habits should not be affecting your computer. What is this, 2004?
posted by battlebison at 1:00 PM on April 21, 2010 [3 favorites]


My wife works for a very large company that have been clobbered by this. She called this morning to say that they were being hit by a virus and then a couple of hours later from the house, where she was doing work on our PC, protected by free software. I suspect McAfee are going to lose quite a bit of business over this one.
posted by IanMorr at 1:01 PM on April 21, 2010


battlebison: "but here's my foolproof solution for not inflicting upon yourself either a virus or an anti-virus:"

Yes, most of us know all this. Down the hall I've got to support a guy who doesn't understand where e-mail comes from. I'm not about to spend 8 hours a day explaining what they need to allow in their NoScript settings to get their bank web page to work.
posted by charred husk at 1:09 PM on April 21, 2010 [4 favorites]


Is there a cool name for software that "destroys that village in order to save it" along the lines of malware, adware, ransomware, etc.?

BenTreWare.
posted by mrgrimm at 1:10 PM on April 21, 2010


I'm not sure how McAfee has managed to stay in business for so long, considering they are regularly rated worse than the free anti-virus programs out there.
posted by Talanvor at 1:11 PM on April 21, 2010


Mod note: A few comments removed. I swear to god, it's really okay not to drop PC/Mac OS War bait into threads. Please avoid it in the future. Thank you.
posted by cortex (staff) at 1:12 PM on April 21, 2010 [3 favorites]


What's considered best these days? My old XP box had WinClam, but the updates were a pain in the ass.
posted by Artw at 1:14 PM on April 21, 2010


Entire Kentucky K12 school system runs on McAfee corporate. My friend had one hell of a day at his school.
posted by deezil at 1:15 PM on April 21, 2010


I just got home from my local (state) government job where all the computers were decimated by this.
posted by OmieWise at 1:15 PM on April 21, 2010


I've had a very good experience with MSE recently. Had to pull some of the other freeware antivirals off the netbook because they were sucking too much processor/memory (as well as just plain sucking). Microsoft with a good security product. Who knew?
posted by bonehead at 1:17 PM on April 21, 2010 [3 favorites]


*computer security anecdote naively expanded to wide generalization*
posted by BeerFilter at 1:17 PM on April 21, 2010


Microsoft Security Essentials is actually pretty good these days, by all accounts.
posted by Sebmojo at 1:18 PM on April 21, 2010 [2 favorites]


8,000 out of 25,000 machines are down where I work. I am glad I have nothing to do with IT anymore. The irony of this causing us more downtime than any virus ever has is kinda awesome.
posted by paanta at 1:18 PM on April 21, 2010 [4 favorites]


As was already pointed out. Reading comprehension fail.
posted by paanta at 1:19 PM on April 21, 2010


This is not a first for McAfee and high-impact false positives. Software developed by the company I work for was quarantined incorrectly by McAfee last year, and it was a headache for all involved.
posted by aught at 1:20 PM on April 21, 2010


We just finished switching from McAfee to Sophos here. Whew.
posted by tracknode at 1:21 PM on April 21, 2010


@battlebison: I totally agree that most anti-virus software sucks for the reasons you provide, but I submit the counter-examples of Microsoft Security Essentials and Symantec Corporate Edition. Both stay well out of the way, and the former doesn't require a subscription.

NoScript seems like a poor solution to me. Most JavaScript is benign and useful. Isn't constantly whitelisting sites just another form of annoyance?
posted by The Lurkers Support Me in Email at 1:21 PM on April 21, 2010 [1 favorite]


I work for the University of Michigan in IT.

It started this morning with one phone call. Someone going, "Hey, I think my computer has a virus. It reboots every 60 seconds."

This was soon followed by a second phone call. And then a third. And right about then I realized maybe something was up.

This has been a nightmarish day from hell, my friends. I feel like one of those people in a war movie who is like 18 and fresh out of boot camp and hungry for war and blood and glory but then I experience the ravages of combat first hand and instead of being bloodthirsty I instead have dull, glassy eyes and my face is a gaunt expression of fear and doubt. I have seen things I cannot unsee this day.

I have seen things no man can unsee.

And I swear to God I will fucking punch someone from McAfee in the cock if it's the last thing I ever do.
posted by kbanas at 1:21 PM on April 21, 2010 [114 favorites]


Yep, I am constantly astonished the Microsoft Security Essentials works as well as it does. I keep expecting something to go horribly wrong with it, but so far, so good.
posted by IanMorr at 1:22 PM on April 21, 2010


Also, yeah, another vote for Security Essentials. I have never actually seen it combat a virus, but I like it because it has a tiny footprint and it doesn't bug me about something every 30 fucking seconds.
posted by kbanas at 1:23 PM on April 21, 2010


I'm not sure how McAfee has managed to stay in business for so long, considering they are regularly rated worse than the free anti-virus programs out there.

I was product manager for a direct competitor to the McAfee product that shat all over itself today. Here's the deal with why free AV hasn't taken over the market yet:

The free programs may test better (and even that's arguable) but they are targeted towards the consumer market, and don't have centrally management, which is a total dealbreaker. For enterprises larger than 100 protected computers or so, management and reporting is the more important feature, as your ability to protect is limited by your ability to manage that protection.

That doesn't address the question of why they're still in business, though - that's mostly a function of the fact that they have VERY credible Network IPS and anti-spam product lines, and are bundling the hell out of their endpoint protection suites, and are killing all but their largest competitors (Symantec and Trend are really the only two others left in the US, with Sophos, Kaspersky and F-Secure only really competitive in Europe) based on the strength of those bundled offerings.
posted by deadmessenger at 1:24 PM on April 21, 2010 [7 favorites]


(nb: This comes from a tech-savvy computer operator. There's no way I'd advise a complete neophyte to use something like NoScript, because so much of the webapps people use today simply break without JS. Most people wouldn't even understand what the hell JavaScript is, never mind understand why nothing is happening when they click on their FaceBook with the left mouse to get to their web news.)
posted by The Lurkers Support Me in Email at 1:25 PM on April 21, 2010 [1 favorite]


I've had a very good experience with MSE recently. Had to pull some of the other freeware antivirals off the netbook because they were sucking too much processor/memory (as well as just plain sucking). Microsoft with a good security product. Who knew?

heh, that's like buying a house with no door and after years of having your place robbed and vandalised the real-estate agency offers you a door for free.
posted by canned polar bear at 1:25 PM on April 21, 2010 [6 favorites]


I have this theory that anti-virus companies get worse and worse as time goes on. McAfee was originally in the forefront - I remember when it was freeware back in the 90's - and now its shit. Norton was big early on too, and now its considered worse than the viruses it protects you from. The company I was hired into was using CA when I arrived and apparently it was the best at the time it was selected, but by the time I got there it sucked horribly. We switched to Kaspersky and worked wonderfully but the recent release is too buggy for me to deploy in good conscience. AVG was the free anti-virus of choice until it became so bloated that it collapsed under its own weight. Now Microsoft has a light, effective and free anti-virus product. How long until it becomes too big for its britches and we all have to move to some other lightweight newcomer?
posted by charred husk at 1:26 PM on April 21, 2010 [1 favorite]


I suspect McAfee are going to lose quite a bit of business over this one.

Given the costs some companies and state agencies will incur because of this, I would think they'll likely face a few lawsuits as well. I'm not counting on it, but it wouldn't surprise me if McAfee got blown out of the anti-virus business by this.
posted by fatbird at 1:28 PM on April 21, 2010


I loath McAfee with the fiery passion of a million white hot suns.
posted by BrotherCaine at 1:31 PM on April 21, 2010


Apparently all the Windows machines in our department were hit by this. Luckily, scientists aren't big Windows fans, so relatively few people were affected. This is why, when the IT guy said an anti-virus program was required for net access on my laptop and that McAfee was free for University people, I paid the $60 for NOD32, and later switched to MSE.
posted by dirigibleman at 1:31 PM on April 21, 2010


Hmmmm...Would this, perhaps, be why Network Solutions' FTP servers went belly-up today?
posted by Thorzdad at 1:32 PM on April 21, 2010


I would think they'll likely face a few lawsuits as well.

What about all the fine print you have to agree to before you install?
posted by pracowity at 1:34 PM on April 21, 2010


I suspect McAfee are going to lose quite a bit of business over this one.

They will, but not as much as you might think. Vendor lock-in in the enterprise antivirus space is a BEAST, and McAfee is one of the worst at it. The cost of doing a rip-and-replace the antivirus on a quarter-million endpoints might make some large enterprises overlook this clusterfuck when it comes to renewal time.
posted by deadmessenger at 1:34 PM on April 21, 2010 [1 favorite]


i'm surprised that nobody seems to be using Avira, works quietly in the background and only bugs me when there's a problem. they also get bonus points from me for donating to a foundation if you buy a premium product.
posted by canned polar bear at 1:35 PM on April 21, 2010 [3 favorites]


Given the costs some companies and state agencies will incur because of this, I would think they'll likely face a few lawsuits as well.

They will, but they won't pay a dime, I suspect. Software EULAs are a fine art of weaselry, and are designed to prevent lawsuits of this type.
posted by deadmessenger at 1:35 PM on April 21, 2010


At some point we really need to consider liability penalties for software companies. I hate to feed tort lawyers, but it's just ridiculous that McAfee could release such a terrible bug as this and the only penalty they'll suffer is a slightly worse reputation.
posted by Nelson at 1:36 PM on April 21, 2010 [1 favorite]


There's no way I'd advise a complete neophyte to use something like NoScript, because so much of the webapps people use today simply break without JS. Most people wouldn't even understand what the hell JavaScript is, never mind understand why nothing is happening when they click on their FaceBook with the left mouse to get to their web news.

I've had a remarkable amount of success with my dad using these rules:

1. If something on the Web site doesn't seem to work, click the "No Smoking" kinda sign down at the bottom.
2. If it pops up with a list of stuff it's "blocking," click on the CLOSEST NAME TO THE SITE YOU'RE ON to allow it.
3. If the site still isn't working, click on the next closest name.
3.5. Google APIs is usually okay.
4. If you have to do this more than three times, or wind up being asked to click on things that make no sense to you, this probably isn't a site that's going to enrich your life to the point that it's worth the risk and bother of continuing.
posted by Shepherd at 1:39 PM on April 21, 2010 [10 favorites]


Man, I knew McAfee was terrible, but this is a bit beyond the pale.
posted by kafziel at 1:42 PM on April 21, 2010



Here's the thing that gets me. XP is a 9 year old OS. Vista came out three years ago, and Win7 last year. And those OSes are not only *far* less vulnerable, but this defect did not affect them.

Now, I've worked for large corporations, and yeah, change is expensive, and hard, and it well and truly sucks. And MS really screwed the pooch on helping Vista get out the door. Ballmer couldn't possibly be fired fast enough or hard enough, IMO, for that screwup.

That being said, the definition of moronitude is relying on an OS that is almost a decade old for anything mission critical. There's little reason for it. It's been four years since the Vista betas were available!

I feel for those people in the trenches who have to clean up this mess. It aint thier fault - it's the CIO's and IT managers - Those idiots who didn't want to face the cost of upgrading and risked hours or days of end user downtime to save a few bucks on thier bottom line.
posted by Pogo_Fuzzybutt at 1:46 PM on April 21, 2010 [2 favorites]


Vista wasn't a value proposition, significantly better or safer, and needed somewhat better hardware anyway. It's easy to see why it didn't fly. It was a solution to none of the (important) problems that XP had.

Win7 is the one most big orgs are going to jump to, I think. Keep in mind that even corporate computers have a 4-5 year lifecycle. It's going to take at least that long to move the majority of people over.
posted by bonehead at 1:50 PM on April 21, 2010


This probably also explains the "unscheduled" computer/server outage one of our vendors had today. One of their reps told me it wasn't affecting everyone's computer, just some of them (including hers), and so she couldn't complete my order and had to transfer me.
posted by never used baby shoes at 1:51 PM on April 21, 2010


That being said, the definition of moronitude is relying on an OS that is almost a decade old for anything mission critical.

You almost -- ALMOST -- had the perfect troll, but this line just took the cake.

Wait-- you _are_ trolling, aren't you? Please tell me you're trolling. Otherwise, I'll tell you a little story about your bank, your airline, and your phone, and you'll cry yourself to sleep.
posted by mark242 at 1:51 PM on April 21, 2010 [29 favorites]


He's not trolling. For really mission-critical stuff I always rely on an OS that's more than 10 years old. Sometimes z/OS but ideally TOPS-20.
posted by GuyZero at 1:54 PM on April 21, 2010 [5 favorites]


Pogo_Fuzzybutt: "XP is a 9 year old OS. Vista came out three years ago, and Win7 last year."

"Never switch to an OS until it is at least a year old", or alternatively, "Wait until the first service pack." We waited on Vista, it sucked, so we waited more. Windows 7 looks good but it is still young, so we're waiting.
posted by charred husk at 2:01 PM on April 21, 2010 [3 favorites]


I feel for those people in the trenches who have to clean up this mess. It aint thier fault - it's the CIO's and IT managers - Those idiots who didn't want to face the cost of upgrading and risked hours or days of end user downtime to save a few bucks on thier bottom line.

Eh? You totally missed the point on compelling reasons to upgrade. XP brought a host of reasons: Much better Domain support with AD and all that jazz, better true 32-bit hardware support, generally very stable, etc. Vista, from a corporate standpoint brought virtually nothing to the drawing table in terms of upgrades. Why "upgrade" when it doesn't really upgrade anything?
posted by jmd82 at 2:03 PM on April 21, 2010 [4 favorites]


Right. I've only just begun to sandbox a potential Windows 7 implementation, for roll out into only a tiny portion of the enterprise, and that's slated for Q1 next year.
posted by cavalier at 2:03 PM on April 21, 2010


I'm really, really glad our shop uses symantec.

We still use XP, mostly because we run about a hundred public terminals and XP seems to have the best lockdown tools (and budget constraints restrict us from doing a central server that hosts an image for them to all boot from).

Updates are a constant pain, but we don't really have virus trouble.
posted by codacorolla at 2:11 PM on April 21, 2010


Meanwhile (via boingboing)
posted by sebastienbailard at 2:15 PM on April 21, 2010


Exactly. The leading edge is the bleeding edge. Computers in the company exist to do work, and if there's no advantage to an upgrade, why would you spend the cost of the upgrade.

You're right that it's odd to be on XP still, but Vista was such a nightmare that we would have been fools to try to upgrade. The moment we heard Win7 was happening, we gave up.

Once Win7SP1 is out, we'll start thinking about a systemwide. Not until. We have better things to do than to beta test software we are paying for.
posted by eriko at 2:15 PM on April 21, 2010 [2 favorites]


So, I go to graduate school at a large university in Texas. I decided to not go in today so I could stay home, clean up the apartment, get some work done, and write.

It was right after I checked my e-mail this morning that all of the panicked "OMG WE HAVE A WORM SHUT IT DOWN" messages hit the inbox. It's all figured out now, but I maintain that this is the kind of thing that happens when I don't show up.

With great power comes great responsibility.
posted by SNWidget at 2:16 PM on April 21, 2010 [4 favorites]


My last job used Mcafee on ~1000 systems. It's as bad as they say, even though EPO wasn't bad. After a bad update caused issues with false positives I re-scheduled all updates for a three day delay. I wonder if the other security admin changed it back after I left...

FWIW My vote for home anti-virus goes to clamwin.
posted by anti social order at 2:18 PM on April 21, 2010


Here in beautiful Kentucky, news outlets are abuzz with how the police department is shutting down all in-car laptops over this.
posted by jbickers at 2:19 PM on April 21, 2010


It will be 2012 before I move my networks away from XP. Compared to Windows XP Pro, Vista upgrades, and Windows 7 upgrades, to a much lesser extent, are almost crippleware if you don't buy the top-tier versions, in my case. It's not worth the trouble right now. If your XP is updated properly, and have a decent antivirus package, you are fairly safe unless a button-clicking fool goes on a tour of a bunch of malware and porn sites, and at least some of those can be blocked by a decent firewall policy.

Vista can be safer, but it's like that kid from a Christmas Story that can barely walk from all the winter clothes he has on: safer, but too much of a pain to actually implement.

I have hope for Windows 7, but I'll just be trading one set of known problems from another, unknown set of problems. Par for the course, I guess, but at least I still have time to test and retest.
posted by chambers at 2:20 PM on April 21, 2010


This took out nearly every computer in every classroom on my university's campus Monday.

-facepalm- This is why half dumb clients are a bad idea.
posted by strixus at 2:23 PM on April 21, 2010



Vista, from a corporate standpoint brought virtually nothing to the drawing table in terms of upgrades. Why "upgrade" when it doesn't really upgrade anything?


Well, not to be snarky, but there is the FPP from earlier today...

And well, I happen to disagree about Vista. I admit, Vista wasn't that great for a number of reasons, but it was a huge improvement over XP in a number of areas, particularly after SP1. And Win7 is the upgrade Vista should have been. In any event, it's been 4 years...

I moved all the windows machines I'm responsible for to Vista the year before last, and Win7 late last year and haven't looked back. There's been some difficulty, sure. But not like this.

Don't get me wrong - I've been doing IT stuff for long enough to have a well developed sense of "if it aint broke don't fix it". That said, I think many people's sense of risk exposure by clinging to XP is calibrated.
posted by Pogo_Fuzzybutt at 2:23 PM on April 21, 2010


MIScalibrated.

jeez.
posted by Pogo_Fuzzybutt at 2:25 PM on April 21, 2010


This problem only affected corporate (ePO) downloads, so I think some of the McAfee hate here is misplaced:

The whole point of the McAfee ePO update suite is that it allows corporates to *test* each update on a small number of machines that represent the software configurations used in their company. Thats why they get the update files several days ahead of general consumers.

Any company which has more than a handfull of PCs hit by this needs to take a long hard look at how their IT is being managed - see The Practice of System and Network Administration.

Anti-virus companies never seem to get much credit, but McAfee found and fixed this issue on the same day, a much faster response time than a few IT companies I could mention. I'm no great fan of McAfee (we don't use it) but most of the criticism levelled at them could apply equally to just about any other Anti-virus vendor.
posted by Lanark at 2:26 PM on April 21, 2010 [6 favorites]


Why "upgrade" when it doesn't really upgrade anything?


Well, not to be snarky, but there is the FPP from earlier today...


There's nothing to indicate this problem happened because of some innate flaw in XP. A different McAfee fuckup could have easily hosed only Vista boxes, or Win7 boxes, or all Windows boxes.
posted by kmz at 2:31 PM on April 21, 2010


The real answer, of course, is to turn off automatic updates, and instead do such things manually. You might be out of date longer, and it requires discipline, but odds are you'll hear of issues from the automatic updaters before you get around to installing.

That sounds a lot like work, though, and most computer users will never take this advice.
posted by davejay at 2:40 PM on April 21, 2010


Midnight here, still picking up the pieces in my office. I'm doing the PC updating night shift, and I have a second shift coming in at 06h00 to continue uninstalling the update. Mcafee has some serious 'splaining to do, and a ton of chocolate to deliver to make up for this shit.
posted by alvarete at 2:45 PM on April 21, 2010


For people running Vista or 7 machines: how do you lock them down from users installing whatever they want on them?
posted by codacorolla at 2:46 PM on April 21, 2010


canned polar bear: I use Avira, as well. Never had a problem with it, aside from the stupid, obnoxious pop-up. This whole situation doesn't make me want to go out and pay for an anti-virus software. I'll stick with my free software that doesn't randomly eat important files.
posted by Dreamcast at 2:48 PM on April 21, 2010


Okay, on a more serious note: While this is pretty much the worst-case false positive I can think of, and I'll certainly think twice about installing McAfee on any box in the future, I think claiming this incident justifies the non-use of antivirus software is a little bit silly.

Indeed, making that call based on this incident alone would be rediculously silly. Especially so when there are so many other good reasons not to use anti-virus software. Well.. not so many.. just one really, but it's a Duesy! Anti-virus software doesn't do anything useful.

Much, much better to learn how to reinstall your computer efficiently (unattended install scripts for the win!), and run a scan on demand virus detection routine like (TrendMicro's HouseCall, but don't let it repair anything, just reinstall with your script).
posted by Chuckles at 2:49 PM on April 21, 2010 [1 favorite]


Really? Blowing away your whole OS and reinstalling when something goes wrong is an alternative to using antivirus software?
posted by The Lurkers Support Me in Email at 2:56 PM on April 21, 2010 [1 favorite]


Sure. It's a bad alternative, just like swimming is a bad alternative to flying when going from New York to London. But it's an alternative.
posted by GuyZero at 3:01 PM on April 21, 2010


It is a great alternative, as far as I'm concerned. I'm constantly installing stuff I don't really want long term. Uninstall never really works well. Reinstalling is a short and painless process with unattended install. QED.

Anyway, the part that I forgot to add is that there is never a problem. It is just that it is better to check once in a while to make sure you aren't being the fool.. Seriously, I don't put any constraints on my use of the internet at all, I visit lots of these "questionable" sites, and installing "questionable" software, but I never seem to get hit with anything. Of course there are some truly questionable sites and some truly questionable software that I don't go for, but it is pretty rare.

Honestly, this "really, reinstall everything seems necessary, OMG how terrible" line of argument is pretty tired nowadays. There's just no reason at all why it should be seen as a big deal.
posted by Chuckles at 3:08 PM on April 21, 2010


Well, I guess you just solved the issue we're having at work. I have 4 PCs already down because of this ridiculous shit. Tomorrow will be a bit easier now that we know WTF is going on.
posted by splice at 3:11 PM on April 21, 2010


"The real answer, of course, is to turn off automatic updates, and instead do such things manually. You might be out of date longer, and it requires discipline, but odds are you'll hear of issues from the automatic updaters before you get around to installing.

That sounds a lot like work, though, and most computer users will never take this advice."

That's NOT the real answer. This wasn't an software update it was a virus definition. You have those set to download to clients usually once a day or once every couple hours. You don't test them, because new definitions come out hourly you want to make sure you have the latest def to protect against new virus's. This type of thing has never happened that I remember and I've been working in IT at a high level for about 15 years.


"Anti-virus companies never seem to get much credit, but McAfee found and fixed this issue on the same day, a much faster response time than a few IT companies I could mention. I'm no great fan of McAfee (we don't use it) but most of the criticism levelled at them could apply equally to just about any other Anti-virus vendor."


No it couldn't, I have NEVER had a machine become inoperable for me because of a virus definition update. If this was a virus it would be the worst virus in the history of virus's. I talked to 3 major colleges today all of which became completely inoperable because of this definition. This is the worst thing I have ever heard of in my history in IT. EACH ONE OF THOSE MACHINES NOW HAS TO BE VISITED MANUALLY.
posted by LouieLoco at 3:19 PM on April 21, 2010


Heard about this. How could they suspect trusty svchost? I love that guy.
posted by BlackLeotardFront at 3:19 PM on April 21, 2010 [3 favorites]


And LAN support wonders why we fight so hard to keep McAfee off the build VMs we run.
  1. It sucks resources. Hard.
  2. It's a build virtual machine. It can be disposed of and rebuilt from scratch with maybe three hours of minimal supervision.
  3. It's a VM on an internal network--if it serves as the infection point we have much bigger problems
  4. Did I mention the resource usage? Build times increase 300% when it run on a build server.
Best we've been able to do is limit its scope to not include the various drives used for the builds.

Oh, and tonight is VM update night. At least these aren't XP instances.
posted by Fezboy! at 3:21 PM on April 21, 2010


VDI?!?!? Whatever. nLlite and, the Unattended install forums at MSFN, and AutoIt will get you where you need to be--all for free.
View does look like an interesting product, but god what a horrible video!!!
posted by Chuckles at 3:23 PM on April 21, 2010 [4 favorites]


Guys, what's a three-letter word for HAHAHAHAHAHAHAHAHA? I am doing a crossword puzzle.
posted by Mister_A at 3:38 PM on April 21, 2010 [1 favorite]


"Honestly, this "really, reinstall everything seems necessary, OMG how terrible" line of argument is pretty tired nowadays. There's just no reason at all why it should be seen as a big deal."

Um.. you clearly don't use much software on your machine. Doing a reinstall for me is a full day process - installing visual studio and all of my various tools, getting a huge source tree back in sync, getting all of the zillions of windows and office and visual studio updates installed... is not a quick and painless process. It was worth it when I went to win7, for a variety of reasons, but I sure don't look forward to doing it again.
posted by flaterik at 3:43 PM on April 21, 2010 [1 favorite]


Utilities Install.bat
start "WinRAR" /D"M:\BooksMusicR\Applications\Utilities\WinRAR" /wait install.bat
start "WinRAR Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\WinRAR" /wait install.bat
start "WhoLockMe" /D"M:\BooksMusicR\Applications\Utilities\WhoLockMe104" /wait install.bat
start "True Launch Bar" /D"M:\BooksMusicR\Applications\Utilities\True Launch Bar" /wait install.bat
start "True Launch Bar Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\True Launch Bar" /wait install.bat
start "Startup Control Panel" /D"M:\BooksMusicR\Applications\Utilities\Startup Control Panel" /wait install.bat
start "Scintilla" /D"M:\BooksMusicR\Applications\Utilities\Scintilla Text Editor" /wait install.bat
start "QuickPar" /D"M:\BooksMusicR\Applications\Utilities\QuickPar" /wait install.bat
start "Purrint" /D"M:\BooksMusicR\Applications\Utilities\Purrint" /wait install.bat
start "Nero" /D"M:\BooksMusicR\Applications\Utilities\Nero 6.3.1.17" /wait install.bat
start "HoeKey" /D"M:\BooksMusicR\Applications\Utilities\HoeKey" /wait install.bat
start "HoeKey Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\HoeKey" /wait install.bat
start "Foxit PDF reader" /D"M:\BooksMusicR\Applications\Utilities\Foxit PDF reader" /wait install.bat
start "Dave's Quick Search" /D"M:\BooksMusicR\Applications\Utilities\Dave's Quick Search Deskbar" /wait install.bat
start "Dave's Quick Search Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\Dave's Quick Search" /wait install.bat
start "Daemon Tools" /D"M:\BooksMusicR\Applications\Utilities\Daemon Tools" /wait install.bat
start "Better File Rename" /D"M:\BooksMusicR\Applications\Utilities\Better File Rename" /wait install.bat
start "AutoIt" /D"M:\BooksMusicR\Applications\Utilities\AutoIt" /wait install.bat
start "AutoIt Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\AutoIt" /wait install.bat
start "Ditto" /D"M:\BooksMusicR\Applications\Utilities\Ditto" /wait install.bat
start "Ditto Settings" /D"M:\BooksMusicR\Chuck\Workstation_Settings\System Settings\Ditto" /wait install.bat

exit
I have four categories like this one, but it is the biggest. What part of unattended install didn't you understand?
posted by Chuckles at 4:08 PM on April 21, 2010 [2 favorites]


LouieLoco: I talked to 3 major colleges today all of which became completely inoperable because of this definition. This is the worst thing I have ever heard of in my history in IT. EACH ONE OF THOSE MACHINES NOW HAS TO BE VISITED MANUALLY.

Oh God, that might explain what happened today at the college where I work. (I am one of the few people who was not affected and I am knocking on every piece of fake wood I can see in my office.) To make matters worse, it's the end of the semester. Not the best time for everyone's computer to go down so you can't access your final exam files and grading spreadsheets....
posted by hurdy gurdy girl at 4:20 PM on April 21, 2010


Cripes, I deal with enough of that shit at work. I really don't want to go home and be a sysadmin as my 5-10 hobby too.
posted by GuyZero at 4:20 PM on April 21, 2010 [1 favorite]


Unattended visual studio + updates still takes several hours, during which I obviously cannot be working.
posted by flaterik at 4:21 PM on April 21, 2010 [1 favorite]


Here in beautiful Kentucky, news outlets are abuzz with how the police department is shutting down all in-car laptops over this.

Also:
The computer problem forced about a third of the hospitals in Rhode Island to postpone elective surgeries and stop treating patients without traumas in emergency rooms....The National Science Foundation headquarters in Virginia also lost computer access.

Intel Corp. appeared to be among the victims, according to employee posts on Twitter. Intel did not immediately return calls for comment.

Peter Juvinall, systems administrator at Illinois State University, said that when the first computer started rebooting it quickly became evident that it was a major problem, affecting dozens of computers at the College of Business alone."
posted by ericb at 4:24 PM on April 21, 2010


Chuckles, why not use disk cloning software?
posted by stavrogin at 4:44 PM on April 21, 2010


I just deleted Mcafee personal ed. on the 19th. They updated the engine a few weeks ago and its been hogging ever since. It seems every other year they figure out that it's best to run a small footprint and then forget about it and go large the next year.
posted by couchdive at 4:56 PM on April 21, 2010


It's economic stimulus for IT workers!
posted by delmoi at 5:09 PM on April 21, 2010 [1 favorite]


Here's the thing that gets me. XP is a 9 year old OS. Vista came out three years ago, and Win7 last year. And those OSes are not only *far* less vulnerable, but this defect did not affect them.
Eh, XP isn't very resource intensive. You still see new computers sold with it, particularly netbooks. For a lot of people "Vista sucks" was a mantra and people stuck with XP because of it, so it hasn't really been "replaced" until Win7 for a lot of people.

The irony is that most of these guys figured that the security stuff wouldn't affect them – because they had Antivirus. Hahah.

--

I went from no AV to Microsoft Security essentials recently. I noticed some slowdowns when I have a bunch of explorer windows open and I'm moving files around, but it might just be a placebo effect -- just something I didn't notice before. But other then that, it works great.

I'm kind of surprised to discover that McAfee sucks so hard, I was assuming that since people paid for it would be better, and MSE would just be the bare minimum, and I assumed that the difference would be in the performance hit. But I guess not.
For people running Vista or 7 machines: how do you lock them down from users installing whatever they want on them?
Huh? This is the default for non-admin users.
No it couldn't, I have NEVER had a machine become inoperable for me because of a virus definition update. If this was a virus it would be the worst virus in the history of virus's. I talked to 3 major colleges today all of which became completely inoperable because of this definition. This is the worst thing I have ever heard of in my history in IT. EACH ONE OF THOSE MACHINES NOW HAS TO BE VISITED MANUALLY.
Read the comment you replied to more closely. This was a "beta" definition that companies were supposed to test themselves before release, it was supposed to be self tested. The real release would come out later. So basically people were doing it wrong. But McAfee is still going to get burned here. It think people at most figured in the worst case this kind of thing would mess up their custom applications, no one would have thought it would hose svchost.

If Microsoft Security Essentials can be easily rolled out on a corporate network, I'd expect a lot of people to look into it. I mean it's free.
posted by delmoi at 5:42 PM on April 21, 2010


Chuckles, why not use disk cloning software?

My first answer is that the hardware changes too often. Of course you can do disk cloning plus Repair Install to get around that problem. Another aspect is that you can run only some of the unattended scripts initially, and run others whenever you want. Maybe you don't need Audio Video tools on a machine, so you leave them off. A few months later you decide to add them, hit the script and two minutes later all is done.

Visual Studio takes hours to install?!?! I like my Microsoft products just fine, but sometimes they do things with an inexplicable level of stupid...
posted by Chuckles at 6:00 PM on April 21, 2010


Chuckles, why not use disk cloning software?

Because cloning requires identical (or very similar) hardware.

A properly configured automated install (Diegostart plug) mimics a manual installation of the OS, including all the correct drivers.
posted by donpardo at 6:22 PM on April 21, 2010 [1 favorite]


Yeah, team suite + tfs client + service packs takes a few hours. SP1 in particular takes FOREVER.
posted by flaterik at 6:40 PM on April 21, 2010


The computer problem forced about a third of the hospitals in Rhode Island to postpone elective surgeries and stop treating patients without traumas in emergency rooms

If I was running those hospitals I would be firing everyone responsible for binding surgical procedures to their Windows-based computers. If we're talking failsafe stuff like MRIs or specialized equipment, it had better be running HP-UX, Unix, or some sort of proprietary software. If we're talking routine stuff like scheduling, it's gonna be someone's ass for not having a backup plan to run things off paper and pencil. If surgery is absolutely dependent on running a server cluster in a bunker somewhere, and it's not dealing with some sort of offsite conferencing, something is wrong.
posted by crapmatic at 7:16 PM on April 21, 2010


If Microsoft Security Essentials can be easily rolled out on a corporate network, I'd expect a lot of people to look into it. I mean it's free.

It can't. Microsoft did have an antivirus product (not free) for enterprise (as opposed to consumer) user desktops, but, ironically, MS announced earlier today that it was discontinuing it.
posted by deadmessenger at 8:25 PM on April 21, 2010


This is the worst thing I have ever heard of in my history in IT. EACH ONE OF THOSE MACHINES NOW HAS TO BE VISITED MANUALLY.

My spouse's workplace had this problem, they were instructed to call IT if they were affected, and basically they had workers start up in safe mode and restore from an earlier date.

Does this not work? Why does every machine have to be visited manually?
posted by misha at 8:56 PM on April 21, 2010


Gadzooks.
Egads.
Terrible news.

Awful.

Maybe there's an answer...
An alternative of some kind?
Can't think what it is though.
posted by namasaya at 9:49 PM on April 21, 2010 [5 favorites]


I suspect McAfee are going to lose quite a bit of business over this one.
They will, but not as much as you might think. Vendor lock-in in the enterprise antivirus space is a BEAST, and McAfee is one of the worst at it. The cost of doing a rip-and-replace the antivirus on a quarter-million endpoints might make some large enterprises overlook this clusterfuck when it comes to renewal time.


Against vendor lock-in and the cost of a rip-and-replace, you have the cost of lost productivity and IT time to repair that the CIO will be reporting to the board. A lot of state agencies will just chalk it up to paperwork being delayed for a couple weeks, but some companies will rate the cost of this at millions of dollars, and a lot of CIOs will be asked how they're going to prevent something like this in the future. Anyone looking for an excuse to switch to Microsoft's malware protection or a free alternative just found a huge opportunity to sell such a change. Not a few companies will look seriously at whether they can get some of that money back.

Software EULAs are a fine art of weaselry, and are designed to prevent lawsuits of this type.

There's a limit to how much an EULA will protect a company, in part because they're not well tested in court, but mainly because the losses this will incur will justify a long, expensive legal battle to void that particular EULA. I would think that McAfee will settle those as quietly as possible. If a company thinks it can win back the 10s of millions it spent cleaning up after this and covering its own downstream losses, it becomes worth it to spend a million dollars on lawyers getting McAfee to cough up a couple million in settlement money.

We're well acquainted with the idea that some plaintiffs and ambulance chasing lawyers are willing to gamble on big payoffs with nuisance suits. I don't know why we conceptually excuse businesses from the same venal (or righteous anger driven) motives.
posted by fatbird at 10:51 PM on April 21, 2010


Against vendor lock-in and the cost of a rip-and-replace, you have the cost of lost productivity and IT time to repair that the CIO will be reporting to the board.

And the fact that a lot of big orgs are starting to chart a path from XP to Win7 -- a transition that'll give them an opportunity to painlessly switch AV vendors at the same time.
posted by Lazlo at 11:11 PM on April 21, 2010


Changing vendors isn't really going to prevent this kind of issue as this kind of mistake can happen to anyone. Cripes Microsoft once released a patch (NT4SP6) that crippled Lotus Notes (and other WinSOCK application) installations across the world.

"The irony of this causing us more downtime than any virus ever has is kinda awesome."

The Morris worm took down about 10% of the internet. ILOVEYOU incurred billions of dollars in clean up costs and actually damaged data (and then Kournikova used the same vector to really highlight all the gullible "refuse to learn" users). Conflicker infected ~10 million machines (most variants of Windows) and at one time indirectly grounded French military planes and infected systems on British Navy vessels. Modern viruses are often the first step in creating the bot nets that are responsible for much of the spam that makes it's way across the internet everyday.

A few days replacing SVCHOST on assorted machines pales in comparison and these are just a few examples of the most successful of viruses and malware. The virus problem would be completely intractable, in no small part because of numerous Microsoft trade offs over the years, if it wasn't for the widespread adoption of anti virus software.
posted by Mitheral at 11:41 PM on April 21, 2010


For anyone who thinks this has never happened before:
BitDefender March 2010
Symantec 2007 (Millions of PCs affected)
I'm sure there are plenty of others if you search around.

If you want to be 100% sure a virus update won't go wrong on your specific software/regional settings/hardware drivers then you need to test it on some sample machines.
posted by Lanark at 2:01 AM on April 22, 2010


what are you guys talking about? Nobody tests virus definitions, they come out hourly, just like virus'. Best practice is to have new definitions installed on workstations every 4 hours. It would be impossible to test all of those.
posted by LouieLoco at 6:33 AM on April 22, 2010


Gadzooks.
Egads.
Terrible news.

Awful.

Maybe there's an answer...
An alternative of some kind?
Can't think what it is though.


Like I haven't been pushing that for years. However, a lot of places suffer from what I see here at my workplace: our software doesn't run on Macs. As much as I prefer Macs and Mac OS X (and I do -- I have owned close to thirty iDevices over the last 7 years), try to convince our developers to switch away from Active X components, IE6 (you heard me), and MS SQL (which as far as a db engine goes, is quite good), etc, after all this time (25 years). Macs simply don't enter the equation. Neither does Linux. There's a ridiculous but solid lock-in. Being flippant about that doesn't change the facts.

We are, however, apparently working on an iPhone/iPad client. So that's nice.
posted by grubi at 6:37 AM on April 22, 2010


It would be impossible to test all of those.
posted by LouieLoco at 2:33 PM on April 22 [+] [!]


All you have to do is setup 2 or 3 download directories, populate one with all updates, another with everything 12 hours old (or older) and a third with updates 1 day old (or older).

Then you might put 2 servers and 50 clients in the first group, 10 servers and 10,000 clients in the second and 200 servers and 80,000 clients in the third.

In my experience most infections are from things that have been floating around for weeks or months. A delay of 24 hours is unlikely to prove fatal.
posted by Lanark at 8:01 AM on April 22, 2010


McAfee has a direct response on a corporate blog: McAfee Response To Current False Positive Issue and A Long Day at McAfee. Both by Barry McPherson, Executive VP for Support.

Barry seems to have studied at the Catholic Church school of passive voice construction and non-apologies. The posts aren't terrible and contain useful info, but instead of "we pushed an untested update that breaks a lot of computers" we get "the error can result in moderate to significant issues". Classic spin to positive, too, "In our ongoing efforts to protect our customers from a seemingly endlessly multiplying variety and volume of attacks, today we released a update file that clearly did more harm than good. There was a legitimate threat and we wanted to protect our customers ..."
posted by Nelson at 8:06 AM on April 22, 2010


I totally agree that McAfee should have tested their updates. But it is NOT best practices to test virus definitions before being installed. It is also NOT good practice to manage a network with day old definitions.
posted by LouieLoco at 8:17 AM on April 22, 2010


We switched to Kaspersky and worked wonderfully but the recent release is too buggy for me to deploy in good conscience.

I do IT for a small company, and we've stuck with KAV - we don't use the suite. I prefer it to just about anything else, although I've had to tweak it a few times due to false positives, and it always seems to get more complicated to configure with each new release. Still, performs consistently better as far as detection and footprint than most of the alternatives, and I've never had to worry about its ability to clean whatever it may find.
posted by krinklyfig at 8:32 AM on April 22, 2010


The virus problem would be completely intractable, in no small part because of numerous Microsoft trade offs over the years, if it wasn't for the widespread adoption of anti virus software.

Hard to say. I think it's unlikely that scenario would play out like that, but if it did, we might have had better solutions earlier, because MS would have been forced to adapt or die. No business is going to pay for computer equipment, software and training which is ultimately worthless to them, and the utility is too great for anyone to just give up on the technology. MS did foist their bad security onto others, and the relationships they set going forward were not beneficial, but if the situation were entirely unworkable it would not remain that way for long.
posted by krinklyfig at 8:41 AM on April 22, 2010


If you want to be 100% sure a virus update won't go wrong on your specific software/regional settings/hardware drivers then you need to test it on some sample machines.

Ideally, yes. In practice, this only happens when you have full time IT and the resources to allow them to do this. Most small businesses cannot test to this extent, particularly those with contracted IT services.
posted by krinklyfig at 8:44 AM on April 22, 2010


I think it's reasonable to expect that if you're paying a lot of money for virus updates from a vendor that the vendor has taken elementary steps to verify their updates don't break Windows. I mean, it's not like svchost.exe is some obscure file unique to a particular install flavour. You're buying an auto-update service; what's the point if you have to manually test every day's update?
posted by Nelson at 8:50 AM on April 22, 2010



Anyone looking for an excuse to switch to Microsoft's malware protection or a free alternative


Any enterprise (when I use the term enterprise, I'm referring to a company with 500 computer users or more) IT Security manager who suggests that they switch to a free alternative, will likely be asked this question:

"That's great - will we still be able to provide the same reports for the auditors that we can now?"

The answer, with regard to EVERY free AV product out there, is an unequivocal "No." The reason is that simply protecting your enterprise is NOT enough. You have to be able to prove that you did during an audit, and the only way to do that is via centralized management and reporting.

Freebies only work when you have the staff enough to physically touch every machine in your enterprise, and that stops scaling at around 50-100 users. Beyond that point, you need an enterprise tool, and those aren't free.
posted by deadmessenger at 9:05 AM on April 22, 2010 [1 favorite]


I mean, it's not like svchost.exe is some obscure file unique to a particular install flavour. You're buying an auto-update service; what's the point if you have to manually test every day's update?
posted by Nelson at 4:50 PM on April 22 [+] [!]


Virus signatures are more complex than just filenames - there are lots of viruses that name their processes svchost.exe
McAfee may try to test the software thoroughly but they can't possibly know which hardware drivers, applications and services your PC has installed. Then theres the DLL issue where installing the same set of applications in a different order results in different versions of shared DLLs. All these things will produce different signatures which may or may not trigger a false positive virus alert.

I don't think testing virus updates needs to be some onerous manual process, you just need to think carefully about who gets the updates first and schedule things sensibly such that one bad update is not going to disable every machine simultaneously.
posted by Lanark at 10:08 AM on April 22, 2010


Ok, So I work for an integrator and we have 500 plus clients with about half of those not having dedicated IT staff. How are they supposed to set up a test lab for every new virus def?
posted by LouieLoco at 10:51 AM on April 22, 2010


I work for an "enterprise", and at least for those of us that are Developers and have opted in to managing our own machines, MSE is just fine.
posted by flaterik at 11:03 AM on April 22, 2010


Virus signatures are more complex than just filenames

Yes. But how many versions of the legitimate Windows svchost.exe are there out there? Why didn't McAfee test that their virus scanner didn't flag that crucial, exists-on-every-system file? The point isn't the filename, it's that it's a ubiquitous, essential file that McAfee didn't effectively test for. If they can't do that right, why pay them?
posted by Nelson at 11:27 AM on April 22, 2010


« Older Unsung Heroine   |   CSS and JS - so now you know Newer »


This thread has been archived and is closed to new comments