Ruby off the Rails
February 1, 2013 5:39 PM Subscribe
What The Rails Security Issue Means For Your Startup summarizes the impact of recent arbitrary-code-execution security vulnerabilities in Ruby on Rails: "What Do We Do When Apocalyptically Bad Things Happen On Our Framework of Choice?"
I don't like this ride, I want off
posted by thelonius at 6:11 PM on February 1, 2013 [2 favorites]
posted by thelonius at 6:11 PM on February 1, 2013 [2 favorites]
holy crap re: embedded analytics javascript
posted by grubby at 6:18 PM on February 1, 2013 [1 favorite]
posted by grubby at 6:18 PM on February 1, 2013 [1 favorite]
That's amazing. Here's users of some kind of Bitcoin deposit house discussing a breach in their system: this RoR exploit results in a "heist."
I wonder which of the parts of internet that I use are compromised.
posted by damehex at 6:28 PM on February 1, 2013
I wonder which of the parts of internet that I use are compromised.
posted by damehex at 6:28 PM on February 1, 2013
Danger Will Rubyson.
posted by a non e mouse at 6:28 PM on February 1, 2013
posted by a non e mouse at 6:28 PM on February 1, 2013
I have one of those "old Rails apps running in obscurity" on some shared hosting. I wouldn't say the whole thing was a big pile of cargo culted example Rails code, but I also wouldn't say I had perfect understanding of every part of every view or controller. It has been running quietly for years. I update it periodically. When this story broke, I was two patch releases behind.
Just two weeks ago, in some unrelated work, I finally got around to introducing the toolchain at my job to Bundler in a way that involved actually understanding it in a professional context vs. just doing what a Rails tutorial had told me to do when I first set up an old hobby app, and I remember guiltily thinking about my old app running in the corner of a shared hosting space.
So when I read about this, I logged in to the shared hosting super hoping I'd see a Gemfile sitting there in the app directory, but it wasn't to be. Fortunately, as it turns out, I had done what one used to do when trying to run Rails on crappy shared hosting years ago and just built my own Rails stack and had all my own gems. Not as elegant as being able to bundle update, but not that many more keystrokes to get everything into a safe place.
About three weeks ago a former employer was briefly blacklisted by Google (along with TechCrunch and a bunch of other people running the iSocket ad network) causing a former colleague whom I love dearly to go on a lengthy rant about how Windows made the Internet insecure and that's why perfectly law abiding servers can get knocked off the 'net for tripping imperfect malware distribution detectors.
In my experience, poorly attended, poorly understood web applications have caused their own share of grief. Sometimes a compromised site is distributing malware (and Windows is the preferred target), other times they're part of some SEO scam, and yet more times they're sitting around in a shared hosting environment with poor security policies just sort of levering themselves up out of one cracked account that got knocked over thanks to one exploit and burrowing down into another to pull something else.
This is a good example of the sort of trouble poorly attended, poorly understood web applications can cause. I wonder how many "built a Rails app over the weekend on a lark" apps there are, sitting around on shared hosting, quietly being pressed into service for all sorts of crazy shit.
I don't know if I'd argue that you should have to have a license to operate one on the open 'net, but the one click installers and "Famous Three-Step Installations" don't strike me as an unmitigated good:
Several years back, a client had his Joomla site knocked over (an exploit in a contributed module that we didn't even know to update because the maintainer had his site compromised, failed to report to everybody that this had happened and just quietly re-released an unexploited version under the same version number). In the process of trying to ferret out the exploit, I Googled a few fairly distinct lines of malicious code and discovered a cornucopia of cracked Joomla sites. Public libraries, elementary schools, hobby sites, "hello world" sites people never did anything more with, "dev.somedomain" sites, etc. etc. etc. All sitting there serving up malware and pharma spam, but in such a way that you just wouldn't know.
And just this past week, we all had a laugh at the office when we found we could search github for ssh keys, unencrypted htpasswd entries and other sensitive information that people had blithely checked in to public repositories. I thought about my colleague ranting about Windows lamers as I stared at private ssh keys and htpasswd entries for a prominent Linux distribution's web server codebase, all proudly hosted in public.
That is all a lot of only vaguely connected thinking going toward the theme "it is kind of a mess out there."
posted by mph at 6:36 PM on February 1, 2013 [45 favorites]
Just two weeks ago, in some unrelated work, I finally got around to introducing the toolchain at my job to Bundler in a way that involved actually understanding it in a professional context vs. just doing what a Rails tutorial had told me to do when I first set up an old hobby app, and I remember guiltily thinking about my old app running in the corner of a shared hosting space.
So when I read about this, I logged in to the shared hosting super hoping I'd see a Gemfile sitting there in the app directory, but it wasn't to be. Fortunately, as it turns out, I had done what one used to do when trying to run Rails on crappy shared hosting years ago and just built my own Rails stack and had all my own gems. Not as elegant as being able to bundle update, but not that many more keystrokes to get everything into a safe place.
About three weeks ago a former employer was briefly blacklisted by Google (along with TechCrunch and a bunch of other people running the iSocket ad network) causing a former colleague whom I love dearly to go on a lengthy rant about how Windows made the Internet insecure and that's why perfectly law abiding servers can get knocked off the 'net for tripping imperfect malware distribution detectors.
In my experience, poorly attended, poorly understood web applications have caused their own share of grief. Sometimes a compromised site is distributing malware (and Windows is the preferred target), other times they're part of some SEO scam, and yet more times they're sitting around in a shared hosting environment with poor security policies just sort of levering themselves up out of one cracked account that got knocked over thanks to one exploit and burrowing down into another to pull something else.
This is a good example of the sort of trouble poorly attended, poorly understood web applications can cause. I wonder how many "built a Rails app over the weekend on a lark" apps there are, sitting around on shared hosting, quietly being pressed into service for all sorts of crazy shit.
I don't know if I'd argue that you should have to have a license to operate one on the open 'net, but the one click installers and "Famous Three-Step Installations" don't strike me as an unmitigated good:
Several years back, a client had his Joomla site knocked over (an exploit in a contributed module that we didn't even know to update because the maintainer had his site compromised, failed to report to everybody that this had happened and just quietly re-released an unexploited version under the same version number). In the process of trying to ferret out the exploit, I Googled a few fairly distinct lines of malicious code and discovered a cornucopia of cracked Joomla sites. Public libraries, elementary schools, hobby sites, "hello world" sites people never did anything more with, "dev.somedomain" sites, etc. etc. etc. All sitting there serving up malware and pharma spam, but in such a way that you just wouldn't know.
And just this past week, we all had a laugh at the office when we found we could search github for ssh keys, unencrypted htpasswd entries and other sensitive information that people had blithely checked in to public repositories. I thought about my colleague ranting about Windows lamers as I stared at private ssh keys and htpasswd entries for a prominent Linux distribution's web server codebase, all proudly hosted in public.
That is all a lot of only vaguely connected thinking going toward the theme "it is kind of a mess out there."
posted by mph at 6:36 PM on February 1, 2013 [45 favorites]
Answer: It’s absolutely trivial to detect Rails applications in a scalable fashion, but why bother? Fire four HTTP requests at every server on the Internet: if the server is added to your botnet, it was running a vulnerable version of Ruby on Rails.)I have a few servers with hardly any traffic, and you can see lots of "hack attempts" trying to access goofy URLs and such.
I have to wonder, though - why is auto update not the default standard at this point? Obviously if you have a major website and a paid IT person who's job is to install patches you'll want to do it manually in case something goes wrong. But for desktop software, or small web projects doesn't it make more sense to have things just update automatically? Maybe make it easy for people to manually revert if they want too.
Requiring everyone to manually update everything, when there are lots of 'small' sites out there, servers running long forgotten software and the like it seems like just having things update themselves automatically would be for the best.
(So long as the central distribution point doesn't get hacked, of course....)
posted by delmoi at 6:36 PM on February 1, 2013
Incidentally, this is via (MeFi's own!) Ned Batchelder's post War is Peace, which takes a Python perspective.
posted by We had a deal, Kyle at 6:40 PM on February 1, 2013 [2 favorites]
posted by We had a deal, Kyle at 6:40 PM on February 1, 2013 [2 favorites]
(So long as the central distribution point doesn't get hacked, of course....)
Like rubygems.org was this past week? There was a brief window where, scrambling to patch your Rails app, you could have theoretically installed an updated version of the Rails gem that had malicious code in it.
posted by mph at 6:42 PM on February 1, 2013
Like rubygems.org was this past week? There was a brief window where, scrambling to patch your Rails app, you could have theoretically installed an updated version of the Rails gem that had malicious code in it.
posted by mph at 6:42 PM on February 1, 2013
So it was the YAML handling what done it? Man, that bastard son of an INI file format had it coming. When I first saw it creep into Perl a decade a go, I thought someone seriously didn't bloody understand how to do markup, and it would all end badly. Those who do not understand XML are doomed to repeat it, only this time with arbitrary code execution.
posted by scruss at 6:54 PM on February 1, 2013 [13 favorites]
posted by scruss at 6:54 PM on February 1, 2013 [13 favorites]
Some day I hope to see the words "MeFi's own" and "Samizdata" related to something cool.
posted by Samizdata at 6:55 PM on February 1, 2013 [1 favorite]
posted by Samizdata at 6:55 PM on February 1, 2013 [1 favorite]
So - being a pompous code-jock brogramming douchebag *doesn't* make your code more secure? Huh, whodathunk.
posted by symbioid at 6:56 PM on February 1, 2013 [2 favorites]
posted by symbioid at 6:56 PM on February 1, 2013 [2 favorites]
For non-programmers, the tl;dr explanation in the article:
"...which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution."
We have been compromised.
posted by cgk at 6:56 PM on February 1, 2013
"...which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution."
We have been compromised.
posted by cgk at 6:56 PM on February 1, 2013
As the complexity of interconnected systems increases, the growth rate of unpredictable security flaws asymptotically approaches the vertical. If somebody with a) brains and resources or b) dumb luck wants to p0wn you, you will be p0wn3d.
Delmoi: allowing auto-update is just as much a security hole as any other vulnerability. The central distribution point doesn't have to get hacked if it's being run by skeevy outfits to begin with. Can you trust Microsoft or Amazon not to use the auto-update hole to inflict stuff you don't want on you, delete things they don't think you should have, or generally spy on you? Heh, wasn't there some Large Company Of Excellent Reputation caught rootkiting peoples systems not long ago?
posted by jfuller at 6:58 PM on February 1, 2013 [1 favorite]
Delmoi: allowing auto-update is just as much a security hole as any other vulnerability. The central distribution point doesn't have to get hacked if it's being run by skeevy outfits to begin with. Can you trust Microsoft or Amazon not to use the auto-update hole to inflict stuff you don't want on you, delete things they don't think you should have, or generally spy on you? Heh, wasn't there some Large Company Of Excellent Reputation caught rootkiting peoples systems not long ago?
posted by jfuller at 6:58 PM on February 1, 2013 [1 favorite]
The article author talks about "new security technology", but it's basically a problem with YAML that allows a YAML expression to create an arbitrary object. And now everyone is finding vairious ways of getting YAML to do that. I'm sure that kind of mistake's been done before; find a function with a buffer overflow, then find various ways to get your data to that function. Not really a "new technology" like some sort of clever analysis tool.
posted by Monday, stony Monday at 7:00 PM on February 1, 2013
posted by Monday, stony Monday at 7:00 PM on February 1, 2013
There was a brief window where, scrambling to patch your Rails app, you could have theoretically installed an updated version of the Rails gem that had malicious code in it.
No, it's far worse than that.
At any point in time after Rubygems.org changed over to being a Rails application (~May '10), an attacker could have used the vulnerability to add malicious code into any gem.
Worse, because the gems themselves have no cryptographic signature to ensure integrity and authenticity, it's basically impossible to establish that any version of a gem has not been tampered with, short of a full audit of every version of every gem available.
It's a particularly bad shitstorm.
posted by Coda at 7:10 PM on February 1, 2013 [15 favorites]
No, it's far worse than that.
At any point in time after Rubygems.org changed over to being a Rails application (~May '10), an attacker could have used the vulnerability to add malicious code into any gem.
Worse, because the gems themselves have no cryptographic signature to ensure integrity and authenticity, it's basically impossible to establish that any version of a gem has not been tampered with, short of a full audit of every version of every gem available.
It's a particularly bad shitstorm.
posted by Coda at 7:10 PM on February 1, 2013 [15 favorites]
I don't currently have any Rails apps I'm working with, and I've already started being conscientious about duplicate credentials over the last few years, but I'm still glad I read this, because it's a good reminder on a couple of points:
* Anything running off a local dev machine with a predictable path is potentially exploitable.
* With the proliferation of online services you use directly or that services you use rely on comes a proliferation of attack surfaces, so being conscientious is important.
* Your favorite code sucks (i.e., even widely used stuff that's the fashion of the moment can have serious vulnerabilities).
I didn't understand this part, though:
which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution.
Did he mean that any page that accepts user-supplied photo *URLs* can do this (via specifying arbitrary attack URLs), or that it's possible to craft images themselves in a way that executes arbitrary code? I guess it's not unimaginable to me that someone could find a bug in a common image handling library and sneak in something, but as attack vectors go, that sounds like a pretty awkward one (much more so than arbitrary object deserialization).
posted by weston at 7:14 PM on February 1, 2013
* Anything running off a local dev machine with a predictable path is potentially exploitable.
localhost:3000 is only one predictable path* With the proliferation of online services you use directly or that services you use rely on comes a proliferation of attack surfaces, so being conscientious is important.
* Your favorite code sucks (i.e., even widely used stuff that's the fashion of the moment can have serious vulnerabilities).
I didn't understand this part, though:
which means any webpage that can contain a user-supplied cat photo could ALSO contain a user-supplied remote code execution.
Did he mean that any page that accepts user-supplied photo *URLs* can do this (via specifying arbitrary attack URLs), or that it's possible to craft images themselves in a way that executes arbitrary code? I guess it's not unimaginable to me that someone could find a bug in a common image handling library and sneak in something, but as attack vectors go, that sounds like a pretty awkward one (much more so than arbitrary object deserialization).
posted by weston at 7:14 PM on February 1, 2013
chortling about Ruby on Rails developers suffering at the moment would be about as well-advised as a classical Roman cursing the gods during a thunderstorm while tapdancing naked in the pool on top of a temple consecrated to Zeus while holding nothing but a bronze rod used for making obscene gestures towards the heavens.
OK, I take my previous snarky comment back.
Between the attacks on NYT and Twitter lately, the Java attacks and now this, how ultimately fucked are we.
Do we *have* to go running to the National Security State to end up providing secure systems? Because I'd really rather not. At the same time, we do have so many small cracks in the infrastructure, that I suppose, it's only a matter of time until *something* happens.
I even got a little conspiratorial minded today thinking about the "from China" claims of the hacks and wondered if it wasn't our own guys doing small hacks to push for stronger security measures that would put them in charge, but dear help me God if I turn into a raving, frothy Alex Jones freak.
Do we have enough eyeballs really working on this shit?
posted by symbioid at 7:17 PM on February 1, 2013
OK, I take my previous snarky comment back.
Between the attacks on NYT and Twitter lately, the Java attacks and now this, how ultimately fucked are we.
Do we *have* to go running to the National Security State to end up providing secure systems? Because I'd really rather not. At the same time, we do have so many small cracks in the infrastructure, that I suppose, it's only a matter of time until *something* happens.
I even got a little conspiratorial minded today thinking about the "from China" claims of the hacks and wondered if it wasn't our own guys doing small hacks to push for stronger security measures that would put them in charge, but dear help me God if I turn into a raving, frothy Alex Jones freak.
Do we have enough eyeballs really working on this shit?
posted by symbioid at 7:17 PM on February 1, 2013
To some extent this is why there is such a push for application firewalls, and why companies like F5 are doing such brisk business with their load balancer integrated app firewall.
posted by iamabot at 7:17 PM on February 1, 2013
posted by iamabot at 7:17 PM on February 1, 2013
Did he mean that any page that accepts user-supplied photo *URLs* can do this (via specifying arbitrary attack URLs), or that it's possible to craft images themselves in a way that executes arbitrary code?
The URLs. Rails does some pretty fancy footwork with query parameters and the like.
posted by Coda at 7:18 PM on February 1, 2013
The URLs. Rails does some pretty fancy footwork with query parameters and the like.
posted by Coda at 7:18 PM on February 1, 2013
And then ...http://blog.twitter.com/2013/02/keeping-our-users-secure.html
posted by iamabot at 7:23 PM on February 1, 2013 [1 favorite]
posted by iamabot at 7:23 PM on February 1, 2013 [1 favorite]
Rails does some pretty fancy footwork with query parameters and the like.
Automatically?
posted by benito.strauss at 7:23 PM on February 1, 2013
Automatically?
posted by benito.strauss at 7:23 PM on February 1, 2013
Defense in Depth, kids... the two vendors of Web Application Firewalls I currently work with had this covered from day zero, and an open source app firewall project has the how-to here.
If you don't have a firewall guy... get one. The intrusion-response and tiger-team guys are Teh Sexah, I know, but you really, really need to have someone on staff who knows how and when to turtle-up. Understanding your network boundaries and the traffic that passes through them on all seven layers* is bone-essential in this day and age.
(*OSI is an obsolete and useless model up until it isn't.)
posted by Slap*Happy at 7:24 PM on February 1, 2013 [2 favorites]
If you don't have a firewall guy... get one. The intrusion-response and tiger-team guys are Teh Sexah, I know, but you really, really need to have someone on staff who knows how and when to turtle-up. Understanding your network boundaries and the traffic that passes through them on all seven layers* is bone-essential in this day and age.
(*OSI is an obsolete and useless model up until it isn't.)
posted by Slap*Happy at 7:24 PM on February 1, 2013 [2 favorites]
Automatically?
Oh yes. Rails, you see, is omakase.
posted by Coda at 7:26 PM on February 1, 2013 [5 favorites]
Oh yes. Rails, you see, is omakase.
posted by Coda at 7:26 PM on February 1, 2013 [5 favorites]
I know Metafilter has a strong IT contingent, and I'm definitely picking up on how serious this is for software creators and internet services providers. Can someone with the smarts tell the rest of us unwashed masses (i.e. "internet = a place to look at cat pictures" type people) what we should be doing to protect ourselves? It would be greatly appreciated. (Can I offer you a cat picture in return?)
posted by Osrinith at 7:48 PM on February 1, 2013
posted by Osrinith at 7:48 PM on February 1, 2013
Osrinith: It isn't going to infect your desktop computer, and if you don't develop or run your own ruby on rails apps, you're not directly affected. However, it may have allowed some sites or services you use to get hacked. So you should make sure you're using different passwords on each site, and make sure they're reasonably good passwords. Those are the primary things users can do about security breaches in server software
Why? Because when a site gets hacked sometimes the hackers get ahold of the password database. The password DB is usually encrypted, so they just encrypt a bunch of passwords and see if any match. So if you have an easy to guess password it's likely to get found that way. If you use the same password on multiple sites, that could then give the hackers a way to either mess with you (spamming from your account or whatever) or a way to attack other sites they now can log into.
posted by aubilenon at 7:54 PM on February 1, 2013 [3 favorites]
Why? Because when a site gets hacked sometimes the hackers get ahold of the password database. The password DB is usually encrypted, so they just encrypt a bunch of passwords and see if any match. So if you have an easy to guess password it's likely to get found that way. If you use the same password on multiple sites, that could then give the hackers a way to either mess with you (spamming from your account or whatever) or a way to attack other sites they now can log into.
posted by aubilenon at 7:54 PM on February 1, 2013 [3 favorites]
If you reuse passwords, then a vulnerability in a new cat photo website (Pettr) can provide attackers with both your email address and the same password you use to check your email (or bank account or Apple account or anything else important).
If your passwords are compartmentalized to a site (using, for example 1Password or similar), then the damage of a disclosure is also compartmentalized: you sigh and generate a new password for them and them only.
The other thing I'd mention is that Google (among others) allow you to use 2-factor authentication for your account. This requires both something you know (a password) and something you have (usually your phone or a physical token) in order to log in. The folks who broke into Pettr won't have that second piece and won't be able to log into your account. Where 2-factor authentication is offered, use it.
posted by Coda at 8:02 PM on February 1, 2013 [2 favorites]
If your passwords are compartmentalized to a site (using, for example 1Password or similar), then the damage of a disclosure is also compartmentalized: you sigh and generate a new password for them and them only.
The other thing I'd mention is that Google (among others) allow you to use 2-factor authentication for your account. This requires both something you know (a password) and something you have (usually your phone or a physical token) in order to log in. The folks who broke into Pettr won't have that second piece and won't be able to log into your account. Where 2-factor authentication is offered, use it.
posted by Coda at 8:02 PM on February 1, 2013 [2 favorites]
To answer why many of us install and nix the auto update policy:
Lots of us figure the earlier adopters will detect vulnerabilities in the latest release, so we'll let it age a while.
==
Confession: it's embarrassing to find out your domain has harbored bots. Crap! My frickin wordpress code had a bot. Wordpress, fer crissakes.
posted by surplus at 8:12 PM on February 1, 2013
Lots of us figure the earlier adopters will detect vulnerabilities in the latest release, so we'll let it age a while.
==
Confession: it's embarrassing to find out your domain has harbored bots. Crap! My frickin wordpress code had a bot. Wordpress, fer crissakes.
posted by surplus at 8:12 PM on February 1, 2013
"WordPress is an unauthenticated remote shell that, as a useful side feature, also contains a blog."
posted by weston at 8:23 PM on February 1, 2013 [40 favorites]
posted by weston at 8:23 PM on February 1, 2013 [40 favorites]
See, this is the reason I never took the time to learn Ruby.
Nah, who am I kidding. It was just laziness.
posted by nowhere man at 8:32 PM on February 1, 2013 [3 favorites]
Nah, who am I kidding. It was just laziness.
posted by nowhere man at 8:32 PM on February 1, 2013 [3 favorites]
Oh yes. Rails, you see, is omakase.
This is a tangent, but DHH alone is the one reason that I disregarded most things Ruby and everything Rails for years. That guy needs to stop talking in public.
posted by device55 at 8:34 PM on February 1, 2013 [8 favorites]
This is a tangent, but DHH alone is the one reason that I disregarded most things Ruby and everything Rails for years. That guy needs to stop talking in public.
posted by device55 at 8:34 PM on February 1, 2013 [8 favorites]
Well, this sounds bad so I'll do what I usually do and just go on living my life and hoping that other people take care of the problem.
posted by Justinian at 8:42 PM on February 1, 2013 [9 favorites]
posted by Justinian at 8:42 PM on February 1, 2013 [9 favorites]
Delmoi: allowing auto-update is just as much a security hole as any other vulnerability. The central distribution point doesn't have to get hacked if it's being run by skeevy outfits to begin with.If the distribution center is skeevy, then you're fucked as soon as you run any code from them at all.
Can you trust Microsoft or Amazon not to use the auto-update hole to inflict stuff you don't want on youYes? I run auto-updates from Microsoft on my windows laptop. I have manual install setup for my desktop but I pretty much install all of the security updates.
And most of my servers are on Amazon's EC2 service, which means they could theoretically inject code into them if they wanted. But why would they? It seems like the damage to their reputation if they got caught would far outweigh the value to them of anything on my server.
If you don't trust someone, then you don't run software they provide. As I said, if they truly do want malicious software on your machine, they can add a secret backdoor. They don't need a "front door" of security updates.
Remember, all I said was that it should be the default. It's something you can disable if you want, but the default should be turned on so people who don't know what they're doing will be secure.
That is how it works for browsers like chrome and firefox, which are probably going to be the most targeted software out there.
The problem here is this glitch essentially leaves people wide open to remote code execution. That seems like a much more serious problem then "The people maintaining this system might maliciously abuse an auto-update system, but wouldn't include a back door in their software for some reason" It seems really unlikely that maintainers of RoR would somehow want to hack people.
Of course they could get hacked which is the main flaw here.
Like rubygems.org was this past week?Yup, like that. And that is the main problem with this. I would say that not only should you trust that people are not going to hack you yourself, but also that they're not idiots and are capable of securely running an update site. Mainly they should be cryptographically signing all their updates, on a secure computer not connected to the internet...
Worse, because the gems themselves have no cryptographic signature to ensure integrity and authenticity, it's basically impossible to establish that any version of a gem has not been tampered with, short of a full audit of every version of every gem available.Okay, wow that's pretty stunning. I didn't pay all that much attention to RoR but it's always given off a douchey, amateur-hour vibe. Still that's pretty amazing.
Oh yes. Rails, you see, is omakase.Wow. Yeah. That's pretty douchey. I mean, it's not a bad idea to standardize on certain design patterns but wow is that a douchey, full-of-yourself way to express that. And given the fact their "amazing chefs" just served up a dish of remote code execution for all their users it certainly makes them look pretty ridiculous for writing about how brilliant their choices are.
This is a tangent, but DHH alone is the one reason that I disregarded most things Ruby and everything Rails for years. That guy needs to stop talking in public.Apparently the language is pretty interesting, although I'm not sure if it's any better then python as a scripting language. Kind of sucks for Ruby that it's so closely associated with Rails. On the other it seems pretty similar to Python, which doesn't have any of the negative associations with Rails
___
Do we *have* to go running to the National Security State to end up providing secure systems? Because I'd really rather not. At the same time, we do have so many small cracks in the infrastructure, that I suppose, it's only a matter of time until *something* happens.I do think for the "average user", even for people running small websites, simply having auto-update be the default, along with decent security for the modules (i.e. digital signatures, keeping the private key offline, etc) would prevent a lot of this stuff.
I know Metafilter has a strong IT contingent, and I'm definitely picking up on how serious this is for software creators and internet services providers. Can someone with the smarts tell the rest of us unwashed masses (i.e. "internet = a place to look at cat pictures" type people) what we should be doing to protect ourselves? It would be greatly appreciated. (Can I offer you a cat picture in return?),This is a thing that's going to let people hack websites. If you're not running a website, using Ruby on Rails, it's not going to be a problem, so long as you always use different passwords on every site.
And, if you're not doing that at this point... start doing that. There are some good password managers out there you can use.
posted by delmoi at 8:49 PM on February 1, 2013 [1 favorite]
I went to a Bloch talk about one of his Java corner case books, and one of his examples was how Java code that is supposed to have, say, a singleton security manager class, can be duped via serialization-based attacks into running a spurious, attacker controlled second instance. I bet any web framework that works by serializing a bunch of object data and passing it to and fro is vulnerable to this kind of thing in principle. They are all hideous! But doing without them is even worse!
posted by thelonius at 8:56 PM on February 1, 2013 [1 favorite]
posted by thelonius at 8:56 PM on February 1, 2013 [1 favorite]
I always run professional maintained servers for my projects. Hosting companies, reputable ones anyway, may let a few things through (never happened to me, knock on wood) but they're a million times more competent than I am. Whenever I hear a friend or low-tech-knowledge customer tell me they're "just going to host our own box" I think, jesus, you can't even use your own computer properly.
posted by maxwelton at 9:14 PM on February 1, 2013 [4 favorites]
posted by maxwelton at 9:14 PM on February 1, 2013 [4 favorites]
Unpleasant situation! Very good comments all around, people. I just wanted to remark on this from way upthread:
why is auto update not the default standard at this point?
There is a deeply ingrained resistance to things like auto-update on production servers. From the perspective of an admin, it's a don't fix what's not broken mentality. Updates can deliver new bugs as well as fixes. And if you're using software from different non-standard repositories and running custom code with sensitive dependencies the last thing you need is some random update throwing a spanner in the works. Even distros that are supposed to be safe for this sort of thing, like Ubuntu LTS, can go wrong on basic auto-update. (And to add to the problem, the safer distros like LTS often don't get all the fixes they should because of the resistance to doing anything that resembles a backport.) If you use a derivative distro and are doing something even slightly esoteric then it's even worse; I've seen auto-updates wreck LTS based Mythbuntu boxes, for instance.
Now, that doesn't mean that you aren't still responsible for keeping a publicly exposed production box up to date as an admin. It just means that if you don't trust auto-update (for any number of valid reasons) then you really must be religious about keeping tabs on issues and doing your own updates (which could include some internal automation of deployment once an update is validated).
posted by snuffleupagus at 9:44 PM on February 1, 2013
why is auto update not the default standard at this point?
There is a deeply ingrained resistance to things like auto-update on production servers. From the perspective of an admin, it's a don't fix what's not broken mentality. Updates can deliver new bugs as well as fixes. And if you're using software from different non-standard repositories and running custom code with sensitive dependencies the last thing you need is some random update throwing a spanner in the works. Even distros that are supposed to be safe for this sort of thing, like Ubuntu LTS, can go wrong on basic auto-update. (And to add to the problem, the safer distros like LTS often don't get all the fixes they should because of the resistance to doing anything that resembles a backport.) If you use a derivative distro and are doing something even slightly esoteric then it's even worse; I've seen auto-updates wreck LTS based Mythbuntu boxes, for instance.
Now, that doesn't mean that you aren't still responsible for keeping a publicly exposed production box up to date as an admin. It just means that if you don't trust auto-update (for any number of valid reasons) then you really must be religious about keeping tabs on issues and doing your own updates (which could include some internal automation of deployment once an update is validated).
posted by snuffleupagus at 9:44 PM on February 1, 2013
Generally speaking, in the managed e-com space, we see critical patching done as needed for critical vulnerabilities in very tight timelines. The lag can be anywhere from 4 hours to many days depending on a number of things related to the business cycle and development cycle. It's not abnormal for things to be patched before public disclosure, and it's also not abnormal for things to remain unpatched until the next monthly or quarterly cycle depending on where development is and depending on what the overall security model for the environment is and what assets are in place to mitigate this kind of things.
There is a reason PCI-DSS now calls for application firewalls, partially it's because they tend to catch stuff like this ahead of time or in shortened windows so you're not waiting on a business cycles:
risk identified-> impact evaluated -> change created -> change approved -> change implemented -> business verification testing -> change released to production -> changes back released via same path to dev/qa, stage, prep-prod
But it's also so there is another tool available that can be adapted to, and that suits the purpose of, mitigating this type of vulnerability disclosure.
There is also a reason businesses are hesitant to move application tiers in to "the cloud", because it often does not allow you to segregate the environment in a manner that provides you with a robust enough model to mitigate the compromise of a logical application tier.
posted by iamabot at 10:09 PM on February 1, 2013 [4 favorites]
There is a reason PCI-DSS now calls for application firewalls, partially it's because they tend to catch stuff like this ahead of time or in shortened windows so you're not waiting on a business cycles:
risk identified-> impact evaluated -> change created -> change approved -> change implemented -> business verification testing -> change released to production -> changes back released via same path to dev/qa, stage, prep-prod
But it's also so there is another tool available that can be adapted to, and that suits the purpose of, mitigating this type of vulnerability disclosure.
There is also a reason businesses are hesitant to move application tiers in to "the cloud", because it often does not allow you to segregate the environment in a manner that provides you with a robust enough model to mitigate the compromise of a logical application tier.
posted by iamabot at 10:09 PM on February 1, 2013 [4 favorites]
From the perspective of an admin, it's a don't fix what's not broken mentality.
This user shares that perspective. Updates are as likely to break things as to improve them. Change means work, even when it's good change, and I reserve the right to decide when and whether I will do that work.
What's more, it's my goddamn machine, and I will control it. I will not allow some random company to install stuff on my machine for me just because they feel like it! It's not their machine. They don't get that privilege.
posted by Mars Saxman at 10:25 PM on February 1, 2013 [1 favorite]
This user shares that perspective. Updates are as likely to break things as to improve them. Change means work, even when it's good change, and I reserve the right to decide when and whether I will do that work.
What's more, it's my goddamn machine, and I will control it. I will not allow some random company to install stuff on my machine for me just because they feel like it! It's not their machine. They don't get that privilege.
posted by Mars Saxman at 10:25 PM on February 1, 2013 [1 favorite]
Putting the OOPS in object oriented programming.
Get these motherfucking objects off this motherfucking program!
posted by StickyCarpet at 1:31 AM on February 2, 2013
Get these motherfucking objects off this motherfucking program!
posted by StickyCarpet at 1:31 AM on February 2, 2013
So, I'm a Pointy Haired Boss. We use some shared services (those $20pcm web hosting services from 1&1 and DreamHost) to host some PHP/SQL things, some custom, mostly things like MediaWiki and OpenCommerce and Bugzilla and WordPress. Might we also have some Ruby things on there that were just uploaded and are running? Or is it a service provided by the server, like PHP, so if the server doesn't have support for it (which I can check fairly easily) then I can relax for now?
posted by alasdair at 2:03 AM on February 2, 2013
posted by alasdair at 2:03 AM on February 2, 2013
Wow. Feeling for the RoR people out there. 'Course... that crowd has always been a little smug around us boring PHP folks... but... yeah. There's a curmudgeon in our office who eschews all frameworks because of these kinds of concerns. Normally, we just blow him off -- like we adopted Zend on a lot of our projects and whatnot because it makes sharing components super easy -- but it's undeniable that incidents like this occur when a whole framework is compromised and suddenly everyone is vulnerable.
It's certainly an ongoing source of debate. Obviously, he represents one extreme. It took a lot of peer pressure to even get him to start using a standard javascript library like jQuery. It can be paralyzing, the fear that the rug will be pulled out from under you if you use some fancy solution that you don't fully understand. Some level of abstraction, good. Too much? Dangerous.
Alasdair -- you should definitely audit your stuff just to make sure. Of course, shared services are somewhat inherently insecure, so I wouldn't stress to much. Related, though, is the upstream threat. How many sites do you use that run RoR? Any passwords you or your staff were using there are now probably compromised.
posted by ph00dz at 5:32 AM on February 2, 2013 [2 favorites]
It's certainly an ongoing source of debate. Obviously, he represents one extreme. It took a lot of peer pressure to even get him to start using a standard javascript library like jQuery. It can be paralyzing, the fear that the rug will be pulled out from under you if you use some fancy solution that you don't fully understand. Some level of abstraction, good. Too much? Dangerous.
Alasdair -- you should definitely audit your stuff just to make sure. Of course, shared services are somewhat inherently insecure, so I wouldn't stress to much. Related, though, is the upstream threat. How many sites do you use that run RoR? Any passwords you or your staff were using there are now probably compromised.
posted by ph00dz at 5:32 AM on February 2, 2013 [2 favorites]
Frankly, if you're running any kind of code which accepts personally-identifying information, or which uses a framework primarily composed of magic, and you're not using some kind of application firewall in front of it all, then you're just asking for trouble.
mod_security for Apache (and nginx) isn't rocket science any more, and it's stopped several clear attempts to exploit this hole on our systems.
Even better is defense in depth; layers of security, solid segmentation of application components and services from each other via firewalling, and so on.
This is not always obvious (and the need for it is not always obvious) to most IT users; perhaps out of this latest set of fail, we'll have some traction to improve the state of affairs, just one tiny jot.
posted by nonspecialist at 6:58 AM on February 2, 2013 [3 favorites]
mod_security for Apache (and nginx) isn't rocket science any more, and it's stopped several clear attempts to exploit this hole on our systems.
Even better is defense in depth; layers of security, solid segmentation of application components and services from each other via firewalling, and so on.
This is not always obvious (and the need for it is not always obvious) to most IT users; perhaps out of this latest set of fail, we'll have some traction to improve the state of affairs, just one tiny jot.
posted by nonspecialist at 6:58 AM on February 2, 2013 [3 favorites]
I would have expected someone to do the "trivial" job of listing major websites (just the top 5000 or so) with Rails botnet exposure. Why has this not yet been published?
posted by surplus at 7:04 AM on February 2, 2013
posted by surplus at 7:04 AM on February 2, 2013
surplus: here's a starter (although I would hope all the ones listed here are now fixed) http://rubyonrails.org/applications. There's also a metric shedload of redmine installs out there.
It's be nice to know what site the twitter hack attempt was sourced from (having got the email from them about needing to change my password - fortunately I'm pretty sure that the old one was a 20 char alphanumber+symbols one generated by lastpass, but still).
posted by titus-g at 7:24 AM on February 2, 2013
It's be nice to know what site the twitter hack attempt was sourced from (having got the email from them about needing to change my password - fortunately I'm pretty sure that the old one was a 20 char alphanumber+symbols one generated by lastpass, but still).
posted by titus-g at 7:24 AM on February 2, 2013
The other thing about Redmine is that it integrates with versioning systems, so there could be quite a lot of source code (and any hardcoded access details) getting got as well.
posted by titus-g at 7:34 AM on February 2, 2013
posted by titus-g at 7:34 AM on February 2, 2013
Does anyone have any recommendations of where to look for security bulletins like this, in a way that can be easily filtered to the technologies/frameworks you use? I'm not a Ruby guy, but Java frameworks obviously have their share of issues from time to time (not to mention the platform itself). I'd love to be able to find out about them reliably, rather than hope I happen upon a blog post about them... but realistically I know that any source that's just a firehose of security bulletins for everything under the sun is something I won't check regularly.
posted by Riki tiki at 7:35 AM on February 2, 2013
posted by Riki tiki at 7:35 AM on February 2, 2013
It may be worth checking to see if there's anything like FreeBSD's portaudit for whatever system you are using. It checks the packages that are actually installed, which means it's less likely to go out of date when you do things like install new software and forget to add it to your bulletins filter.
posted by titus-g at 7:42 AM on February 2, 2013
posted by titus-g at 7:42 AM on February 2, 2013
device55: "DHH alone is the one reason that I disregarded most things Ruby and everything Rails for years. That guy needs to stop talking in public."
No, keeping people from using Ruby, and especially keeping them from using Rails, is God's work.
posted by idiopath at 8:30 AM on February 2, 2013 [9 favorites]
No, keeping people from using Ruby, and especially keeping them from using Rails, is God's work.
posted by idiopath at 8:30 AM on February 2, 2013 [9 favorites]
More seriously, the fundamental design policy behind ruby, that things just work, that objects are just a special kind of data structure that can be modified like any other data structure in your code (and classes etc.) is very friendly to exploritory coding - trying a bunch of things and making something work, kind of an evolutionary trial and error coding style. This is probably the biggest strength of ruby, its design that encourages exploration and experimentation.
Sadly, this is exactly the kind of programming style that someone attempting to find a novel way to break into your app uses. If your framework is poorly designed from a security standpoint, as rails fundamentally is, arbitrary interactions with the outside world lead to trivial execution of custom code. The article is wrong on at least one point: these are not new cutting edge kinds of bugs, they are the kind of bugs you always see when the application layer fucks around with the barrier between code and data (see also outlook viruses and viruses embedded in ms office macros).
So a hackable language (in the sense of exploratory development) becomes a hackable app (in the sense of easy to break into and exploit).
posted by idiopath at 9:03 AM on February 2, 2013 [10 favorites]
Sadly, this is exactly the kind of programming style that someone attempting to find a novel way to break into your app uses. If your framework is poorly designed from a security standpoint, as rails fundamentally is, arbitrary interactions with the outside world lead to trivial execution of custom code. The article is wrong on at least one point: these are not new cutting edge kinds of bugs, they are the kind of bugs you always see when the application layer fucks around with the barrier between code and data (see also outlook viruses and viruses embedded in ms office macros).
So a hackable language (in the sense of exploratory development) becomes a hackable app (in the sense of easy to break into and exploit).
posted by idiopath at 9:03 AM on February 2, 2013 [10 favorites]
Ugh. I'm going to be spending the next couple of months learning web development with Ruby on Rails. Although, I expect to use what I learn for other platforms and languages, this is alarming*. Although it's very embarrassing (to say the least) for the RoR community, as the article says, the communities supporting all other web platforms (Django, Drupal, what have you), should be examining and re-examining their product very carefully.
*I was wary of getting into web development to begin with, but that was mostly fearful ignorance. This does not help.
posted by wobh at 10:51 AM on February 2, 2013 [1 favorite]
*I was wary of getting into web development to begin with, but that was mostly fearful ignorance. This does not help.
posted by wobh at 10:51 AM on February 2, 2013 [1 favorite]
It sounds like this is a vulnerability that came out of Ruby's culture of deserializing executable data without thinking too carefully about security. I know another language whose users like to jump back and forth across the code/data divide. Can anyone with Lisp experience comment on this?
posted by d. z. wang at 10:53 AM on February 2, 2013
posted by d. z. wang at 10:53 AM on February 2, 2013
these are not new cutting edge kinds of bugs
No kidding. There have been giant warning signs posted all over python's pickle module for *years*. I can't remember a time when pythonistas weren't aware of this class of attack, and (hopefully) the framework designers have taken it into account.
posted by dcrewi at 11:14 AM on February 2, 2013
No kidding. There have been giant warning signs posted all over python's pickle module for *years*. I can't remember a time when pythonistas weren't aware of this class of attack, and (hopefully) the framework designers have taken it into account.
posted by dcrewi at 11:14 AM on February 2, 2013
Based on my experience with Common Lisp and Clojure, "the code/data divide" is really just a convention. In other languages, the convention is grammatical one, but a convention nonetheless. I expect most programmers know it's a false dichotomy but failing to code appropriately about it happens anyway. This isn't new.
posted by wobh at 11:21 AM on February 2, 2013
posted by wobh at 11:21 AM on February 2, 2013
It sounds like this is a vulnerability that came out of Ruby's culture of deserializing executable data without thinking too carefully about security. I know another language whose users like to jump back and forth across the code/data divide. Can anyone with Lisp experience comment on this?
Does Lisp even have a Web framework?
posted by ymgve at 11:30 AM on February 2, 2013
Does Lisp even have a Web framework?
posted by ymgve at 11:30 AM on February 2, 2013
Weblocks is the first thing a google search comes up with.
posted by wobh at 11:36 AM on February 2, 2013
posted by wobh at 11:36 AM on February 2, 2013
clojure has some infrastructure for such things, and my own company has an in-house open source clojure based webapp framework (which I have been working on quite a bit with) - though the full public release with fireworks and fanfare is not for a few months yet
Fundamentally, in clojure, as with most lisps, read and eval are separate actions. Though all executable code in a lisp can fundamentally be stored as a first class data object, there is not the same kind of emphasis on "magic", where some data structures cause unexpected changes in your execution environment.
These vulnerabilities in rails are coming about because in ruby objects and classes can be generated within the deserialization process.
Clojure has read (and the variant I would use more often in my code, read-string), but it explicitly never generates a class instance or definition - a token can resolve to a number, a symbol, a class, a static class method, etc. - but the stage where a data object is defined and (re)bound in a namespace cannot happen during the read stage (this is a fundamental principle of mainstream lisp since at least the 1970's). A list or hash map can be constructed (which hypothetically can cause a DOS if you read from a dodgy stream), but nothing is going to get redefined, and unless you explicitly store the objects they are immediately garbage collected.
As far as I know, nothing about ring (or our own webapp we have built on top of clojure's ring library) automagically calls read on data from a client (what happens quite a bit often something is parsed out of json or xml, using an underlying java lib, and then converted into static data structures). And even if it did, because of the way read is designed to work, the worst a malicious client could do is create a very large expensive to compute hash map to be generated (if we presume the worst about clojure's hash map implementation).
posted by idiopath at 12:10 PM on February 2, 2013 [3 favorites]
Fundamentally, in clojure, as with most lisps, read and eval are separate actions. Though all executable code in a lisp can fundamentally be stored as a first class data object, there is not the same kind of emphasis on "magic", where some data structures cause unexpected changes in your execution environment.
These vulnerabilities in rails are coming about because in ruby objects and classes can be generated within the deserialization process.
Clojure has read (and the variant I would use more often in my code, read-string), but it explicitly never generates a class instance or definition - a token can resolve to a number, a symbol, a class, a static class method, etc. - but the stage where a data object is defined and (re)bound in a namespace cannot happen during the read stage (this is a fundamental principle of mainstream lisp since at least the 1970's). A list or hash map can be constructed (which hypothetically can cause a DOS if you read from a dodgy stream), but nothing is going to get redefined, and unless you explicitly store the objects they are immediately garbage collected.
As far as I know, nothing about ring (or our own webapp we have built on top of clojure's ring library) automagically calls read on data from a client (what happens quite a bit often something is parsed out of json or xml, using an underlying java lib, and then converted into static data structures). And even if it did, because of the way read is designed to work, the worst a malicious client could do is create a very large expensive to compute hash map to be generated (if we presume the worst about clojure's hash map implementation).
posted by idiopath at 12:10 PM on February 2, 2013 [3 favorites]
It sounds like this is a vulnerability that came out of Ruby's culture of deserializing executable data without thinking too carefully about security. I know another language whose users like to jump back and forth across the code/data divide. Can anyone with Lisp experience comment on this?
Lisp really isn't really common enough to be a big target yet, but any language with an
Still, if Clojure ever catches on like Ruby, it wouldn't surprise me terribly if something similar occurs.
posted by Wemmick at 12:32 PM on February 2, 2013
Lisp really isn't really common enough to be a big target yet, but any language with an
eval function is potentially vulnerable to similar attacks. However, I think seeing this as partly a side effect of the Ruby culture of informal hacker-friendly dynamism is correct. The code-is-data paradigm that Lisp enables is only partly from eval — the other part comes from macros, but since macro expansion is almost always a "compile time" operation, they aren't a security risk in most cases. Plus, my impression of the Lisp community is that they're more disciplined about this kind of thing, and using eval without a good reason is frowned upon: for example.Still, if Clojure ever catches on like Ruby, it wouldn't surprise me terribly if something similar occurs.
posted by Wemmick at 12:32 PM on February 2, 2013
Wemmick: "the other part comes from macros, but since macro expansion is almost always a "compile time" operation, they aren't a security risk in most cases"
Except reader macros (which, like the name says, happen at read time - reader macros are actually how the details of read are implemented). We make those hard to implement for a reason (however nice they are as a feature). What we are saying about eval (that you pretty much never need it and should be suspicious of code that uses it) goes doubly so for reader macros.
Wemmick: " if Clojure ever catches on like Ruby, it wouldn't surprise me terribly if something similar occurs"
clojure does have a few bits of magic behavior hidden in it, but the design of the language and the culture around clojure are much more (rightfully) suspicious of magic language features in my experience. Sadly, I think that same avoidance of DWIM magic will also prevent clojure from being as popular as ruby. That and the sexps.
posted by idiopath at 12:52 PM on February 2, 2013 [1 favorite]
Except reader macros (which, like the name says, happen at read time - reader macros are actually how the details of read are implemented). We make those hard to implement for a reason (however nice they are as a feature). What we are saying about eval (that you pretty much never need it and should be suspicious of code that uses it) goes doubly so for reader macros.
Wemmick: " if Clojure ever catches on like Ruby, it wouldn't surprise me terribly if something similar occurs"
clojure does have a few bits of magic behavior hidden in it, but the design of the language and the culture around clojure are much more (rightfully) suspicious of magic language features in my experience. Sadly, I think that same avoidance of DWIM magic will also prevent clojure from being as popular as ruby. That and the sexps.
posted by idiopath at 12:52 PM on February 2, 2013 [1 favorite]
idiopath: "Fundamentally, in clojure, as with most lisps, read and eval are separate actions."
What I find most ironic about the rapid adoption of JSON is that the main selling point has always been that the syntax (or a subset of it) evaluates to data structures in at least two very popular web app development languages. This current yaml-to-method-dispatch howler is bad enough, but JSON is a timebomb waiting to go off, not least because there are people asking questions like this one. It frightens me.
posted by vanar sena at 1:00 PM on February 2, 2013
What I find most ironic about the rapid adoption of JSON is that the main selling point has always been that the syntax (or a subset of it) evaluates to data structures in at least two very popular web app development languages. This current yaml-to-method-dispatch howler is bad enough, but JSON is a timebomb waiting to go off, not least because there are people asking questions like this one. It frightens me.
posted by vanar sena at 1:00 PM on February 2, 2013
vanar sena, the difference to me is that the set of objects JSON creates are pretty straightforwardly delimited.
I mean, if you want to get pedantic about it, XML creates data structures on the machine loading it as well (even if we are just talking about the parser stack used to churn the token stream in expat). There is something to be said, I guess, for the fact that what data structure comes out the tail end of XML parsing is implementation dependent, rather than being part of the spec.
But creating hash tables and lists (not to mention numbers and strings) is hardly more intrinsically a security risk than reading packets from a remote host is. Not to mention something as complex and dangerous as loading an image (as the commit history of imagemagick, libpng, and the like will reveal) - which is hardly considered a controversial feature.
posted by idiopath at 1:17 PM on February 2, 2013
I mean, if you want to get pedantic about it, XML creates data structures on the machine loading it as well (even if we are just talking about the parser stack used to churn the token stream in expat). There is something to be said, I guess, for the fact that what data structure comes out the tail end of XML parsing is implementation dependent, rather than being part of the spec.
But creating hash tables and lists (not to mention numbers and strings) is hardly more intrinsically a security risk than reading packets from a remote host is. Not to mention something as complex and dangerous as loading an image (as the commit history of imagemagick, libpng, and the like will reveal) - which is hardly considered a controversial feature.
posted by idiopath at 1:17 PM on February 2, 2013
idiopath: Except reader macros (which, like the name says, happen at read time - reader macros are actually how the details of read are implemented).
Ah, thanks. I'm familiar more with the Scheme/Common Lisp side of things, and haven't examined reader macros much - which could be used for deserialization now that I think about it. Guess I should have drawn a distinction between them and defmacro-type forms, which is what I usually think of with the code vs. data divide.
I do think you're right that the very limitations which make it more secure may also make Clojure less likely to catch on. Those who fail to learn from the past, etc...
posted by Wemmick at 1:20 PM on February 2, 2013
Ah, thanks. I'm familiar more with the Scheme/Common Lisp side of things, and haven't examined reader macros much - which could be used for deserialization now that I think about it. Guess I should have drawn a distinction between them and defmacro-type forms, which is what I usually think of with the code vs. data divide.
I do think you're right that the very limitations which make it more secure may also make Clojure less likely to catch on. Those who fail to learn from the past, etc...
posted by Wemmick at 1:20 PM on February 2, 2013
Reader macros are a feature of common lisp (that is where I first found them), but are rarely delved into.
read is used in template parsing in our clojure web framework (you don't get templates from untrusted sources). But afaik we don't use read on anything coming from a client.
The leinengen project.clj format is clojure code, and I trust clojars is using read rather than eval to deserialize that data (to avoid a repeat of the gem hack).
posted by idiopath at 1:37 PM on February 2, 2013
read is used in template parsing in our clojure web framework (you don't get templates from untrusted sources). But afaik we don't use read on anything coming from a client.
The leinengen project.clj format is clojure code, and I trust clojars is using read rather than eval to deserialize that data (to avoid a repeat of the gem hack).
posted by idiopath at 1:37 PM on February 2, 2013
one last thing before I take a break from this derail:
https://github.com/ato/clojars-web/blob/master/src/clojars/main.clj
clojars is in fact using read, and turning off a bad feature in clojure that sneaks eval into read (major wtf there, I hope that is changed in future clojure versions).
http://clojuredocs.org/clojure_core/clojure.core/*read-eval*
posted by idiopath at 1:42 PM on February 2, 2013
https://github.com/ato/clojars-web/blob/master/src/clojars/main.clj
clojars is in fact using read, and turning off a bad feature in clojure that sneaks eval into read (major wtf there, I hope that is changed in future clojure versions).
http://clojuredocs.org/clojure_core/clojure.core/*read-eval*
posted by idiopath at 1:42 PM on February 2, 2013
JSON is a timebomb waiting to go off, not least because there are people asking questions like this one.
I don't think JSON itself is that risky. Parsing it with eval, sure, but I think it's becoming more common knowledge that this is unsafe, and if you combine that with the proliferation of easy and safe ways to deserialize it (arguably easier than eval, in fact), it would seem direct risk is relatively small.
I worry more about the indirect risks driven by programmer's desire for productivity and ease. If I'm understanding the attack under discussion correctly, the fundamental problem has less to do with any specific serialization format and more to do with the intersection of OOP + serialization -- having deserialization to an instance of an arbitrary class.
Because if your language has constructors, then every class instance created means the execution of some code. And your language has metaclass hacking of some sort, then reasoning about exactly what a constructor does (or other methods, for that matter) can become more than a casual exercise (heck, it's arguably not entirely casual if you're using inheritance or mixins). And that's before you start thinking about *native* objects that may involve eval or code execution.
But wanting OOP + easy serialization/deserialization is pretty natural once you've started using OOP at all. I've wanted this feature in other languages for convenience. I've gone looking for it without thinking about the potential consequences.
It's these kinds of indirect threats rather than a data format itself or eval that worry me, because I can see how they're easy to overlook.
posted by weston at 1:54 PM on February 2, 2013 [3 favorites]
I don't think JSON itself is that risky. Parsing it with eval, sure, but I think it's becoming more common knowledge that this is unsafe, and if you combine that with the proliferation of easy and safe ways to deserialize it (arguably easier than eval, in fact), it would seem direct risk is relatively small.
I worry more about the indirect risks driven by programmer's desire for productivity and ease. If I'm understanding the attack under discussion correctly, the fundamental problem has less to do with any specific serialization format and more to do with the intersection of OOP + serialization -- having deserialization to an instance of an arbitrary class.
Because if your language has constructors, then every class instance created means the execution of some code. And your language has metaclass hacking of some sort, then reasoning about exactly what a constructor does (or other methods, for that matter) can become more than a casual exercise (heck, it's arguably not entirely casual if you're using inheritance or mixins). And that's before you start thinking about *native* objects that may involve eval or code execution.
But wanting OOP + easy serialization/deserialization is pretty natural once you've started using OOP at all. I've wanted this feature in other languages for convenience. I've gone looking for it without thinking about the potential consequences.
It's these kinds of indirect threats rather than a data format itself or eval that worry me, because I can see how they're easy to overlook.
posted by weston at 1:54 PM on February 2, 2013 [3 favorites]
Easy serialization / deserialization is fine, just remember that you should never trust your serialized data with anyone who you wouldn't trust to rewrite your source code. For example if someone has authenticated to the CMS, you can probably go ahead and convert data their client sends you into changes in the content db. But you may want to make sure that doesn't have a way to touch the ORM db, so you probably shouldn't run arbitrary sql commands from the client if both dbs are using the same backend.
posted by idiopath at 2:09 PM on February 2, 2013
posted by idiopath at 2:09 PM on February 2, 2013
Shouldn't proper type checking eliminate this sort of thing, even with deserialization of completely arbitrary data? Provided that you have proper type checks in place, preferably in a language that at least strongly encourages it -- BTW how does Ruby fare in that regard?
posted by Anything at 2:46 PM on February 2, 2013
posted by Anything at 2:46 PM on February 2, 2013
(I'm asking that from the perspective of mainly a Scala programmer, for whom the sorts of errors on display here look very very alien.)
posted by Anything at 2:48 PM on February 2, 2013
posted by Anything at 2:48 PM on February 2, 2013
I went looking for a Java library to parse yaml for configuration data and was shocked at the complexity embedded in a seemingly simple grammar. All I wanted was something along the lines of org.json.JSONObject which parses back and forth to semi typed collections and maps (primitives and lists are kept, but objects are all implemented by a generic map interface)
Every yaml library I found wanted to push the data straight into 'real' objects. Then there is some truly crazy parts of the spec where you can declare the class to deserialize into. This is likely stranger to me as a Java programmer where a map and an object with methods have a much greater distinction. However, I thought it looked scary enough even then and pulled out our yaml for json. For what it's worth, this is the same reason why I always treat serialized Java objects as hostile.
I'm not trying to kick ruby when it's down here, but I think that there are clear 'magic' patterns that invite these bugs. (And I think there is a relationship with the current DRY fetish)
posted by PissOnYourParade at 3:17 PM on February 2, 2013 [1 favorite]
Every yaml library I found wanted to push the data straight into 'real' objects. Then there is some truly crazy parts of the spec where you can declare the class to deserialize into. This is likely stranger to me as a Java programmer where a map and an object with methods have a much greater distinction. However, I thought it looked scary enough even then and pulled out our yaml for json. For what it's worth, this is the same reason why I always treat serialized Java objects as hostile.
I'm not trying to kick ruby when it's down here, but I think that there are clear 'magic' patterns that invite these bugs. (And I think there is a relationship with the current DRY fetish)
posted by PissOnYourParade at 3:17 PM on February 2, 2013 [1 favorite]
I haven't been developing in Rails very long. Fixing this will be an educational experience.
posted by double block and bleed at 3:45 PM on February 2, 2013
posted by double block and bleed at 3:45 PM on February 2, 2013
clojars is in fact using read, and turning off a bad feature in clojure that sneaks eval into read (major wtf there, I hope that is changed in future clojure versions).
That strikes me as bizarre. Anything accepting data from who knows where shouldn't be using read to read it. Lisp is, famously, not that hard to parse, and a safe metacircular evaluator that does everything clojars needs can't be that hard. (I don't know how much clojars needs, actually, but I'd assume that the data it wants from project.clj or whatever is unlikely to be determined at runtime.)
posted by kenko at 4:47 PM on February 2, 2013
That strikes me as bizarre. Anything accepting data from who knows where shouldn't be using read to read it. Lisp is, famously, not that hard to parse, and a safe metacircular evaluator that does everything clojars needs can't be that hard. (I don't know how much clojars needs, actually, but I'd assume that the data it wants from project.clj or whatever is unlikely to be determined at runtime.)
posted by kenko at 4:47 PM on February 2, 2013
Ah, thanks. I'm familiar more with the Scheme/Common Lisp side of things, and haven't examined reader macros much - which could be used for deserialization now that I think about it.
Reader macros are much more used in Common Lisp than either Scheme or Clojure, I thought? Clojure doesn't even have the facilities for real reader macros.
posted by kenko at 4:48 PM on February 2, 2013
Reader macros are much more used in Common Lisp than either Scheme or Clojure, I thought? Clojure doesn't even have the facilities for real reader macros.
posted by kenko at 4:48 PM on February 2, 2013
Does Lisp even have a Web framework?
Several different Lisps have several Web frameworks each.
posted by kenko at 4:49 PM on February 2, 2013
Several different Lisps have several Web frameworks each.
posted by kenko at 4:49 PM on February 2, 2013
Reader macros are much more used in Common Lisp than either Scheme or Clojure, I thought?
Yeah, I think you're right - my CL knowledge only comes from playing around with it since there's little demand for it in industry. I never managed to explore reader macros much beyond the knowledge that they exist. My (cursory) understanding was that they're mostly a way to implement simple syntactic sugars.
Now that I've looked at it more closely, it seems a series of design weaknesses made this possible. Looks like you have an XML or JSON parser that's spitting out YAML, YAML allows the creation of arbitrary objects for existing classes, and then there's an existing class in Ruby that evaluates arbitrary code when its objects are initialized. At every stage, more capability than necessary. Deserialization from externally supplied data shouldn't be able to create anything other than plain old data types with trivial constructors.
(On an unrelated note, what will it take for industry to take security seriously and start talking about formal methods or Object Capability languages? Half of our security problems are solvable with better tools.)
posted by Wemmick at 5:48 PM on February 2, 2013 [1 favorite]
Yeah, I think you're right - my CL knowledge only comes from playing around with it since there's little demand for it in industry. I never managed to explore reader macros much beyond the knowledge that they exist. My (cursory) understanding was that they're mostly a way to implement simple syntactic sugars.
Now that I've looked at it more closely, it seems a series of design weaknesses made this possible. Looks like you have an XML or JSON parser that's spitting out YAML, YAML allows the creation of arbitrary objects for existing classes, and then there's an existing class in Ruby that evaluates arbitrary code when its objects are initialized. At every stage, more capability than necessary. Deserialization from externally supplied data shouldn't be able to create anything other than plain old data types with trivial constructors.
(On an unrelated note, what will it take for industry to take security seriously and start talking about formal methods or Object Capability languages? Half of our security problems are solvable with better tools.)
posted by Wemmick at 5:48 PM on February 2, 2013 [1 favorite]
Shouldn't proper type checking eliminate this sort of thing, even with deserialization of completely arbitrary data?
Eliminate? I don't think so. If you have classes in the language whose explicit approved behavior is to execute arbitrary code, then it doesn't matter how fussy the type system is. Deserialization to such a class will be a vulnerability.
Manifest typing would help -- the programmer would have to tell the compiler/interpreter what type/class they're expecting to come back from a de-serializer, and if it was something unexpected, there'd be an error.
Even this seems neither necessary nor sufficient to me, though: if the problem is that you want the programmer to specify what they want back, you could make it a required argument to your deserializing function. And AFAIK, most strictly typed languages have a Object/root/anything type that serves as a super type class to everything else, so a programmer (at the framework/library level or elsewhere) who decides that magic deserialization to anything is a virtue, and define a deserializing function that gives back something of the broadest fits-all type.
posted by weston at 10:59 PM on February 2, 2013
Eliminate? I don't think so. If you have classes in the language whose explicit approved behavior is to execute arbitrary code, then it doesn't matter how fussy the type system is. Deserialization to such a class will be a vulnerability.
Manifest typing would help -- the programmer would have to tell the compiler/interpreter what type/class they're expecting to come back from a de-serializer, and if it was something unexpected, there'd be an error.
Even this seems neither necessary nor sufficient to me, though: if the problem is that you want the programmer to specify what they want back, you could make it a required argument to your deserializing function. And AFAIK, most strictly typed languages have a Object/root/anything type that serves as a super type class to everything else, so a programmer (at the framework/library level or elsewhere) who decides that magic deserialization to anything is a virtue, and define a deserializing function that gives back something of the broadest fits-all type.
posted by weston at 10:59 PM on February 2, 2013
And AFAIK, most strictly typed languages have a Object/root/anything type that serves as a super type class to everything else, so a programmer (at the framework/library level or elsewhere) who decides that magic deserialization to anything is a virtue, and define a deserializing function that gives back something of the broadest fits-all type.
I think the distinction between library-level and more downstream levels is important here -- if the user of the library needs to explicitly declare at some point that he or she is happy with calling for absurdly broad types, then that should profoundly limit the number of leaky applications, unless, again in the case of Scala for example, the library developer has specifically gone out of his or her way to write insecure code (for example by having the deserializer accept a type argument, but then just casting an internal broad-typed result to that type, which in Scala would take more effort than doing something sane).
I'll add that in a language like Scala you wouldn't even need to declare the desired secure type at the site of the call to the deserializer -- if anywhere down the line your code expects the deserialized object to be an instance of your MyNiceType, the deserializer method won't ever get called if its return type is SkeevyBroadType. In fact the code won't even compile -- I suppose similar safety might be also possible in a dynamic type system, though I'm not sure.
I'm usually shy of hyping My Preferred Language, but if it's the case that use of a language like Scala would help avoid these sorts of disasters, which, it seems to me, it would, then I suppose I'll have to.
posted by Anything at 11:59 PM on February 2, 2013
I think the distinction between library-level and more downstream levels is important here -- if the user of the library needs to explicitly declare at some point that he or she is happy with calling for absurdly broad types, then that should profoundly limit the number of leaky applications, unless, again in the case of Scala for example, the library developer has specifically gone out of his or her way to write insecure code (for example by having the deserializer accept a type argument, but then just casting an internal broad-typed result to that type, which in Scala would take more effort than doing something sane).
I'll add that in a language like Scala you wouldn't even need to declare the desired secure type at the site of the call to the deserializer -- if anywhere down the line your code expects the deserialized object to be an instance of your MyNiceType, the deserializer method won't ever get called if its return type is SkeevyBroadType. In fact the code won't even compile -- I suppose similar safety might be also possible in a dynamic type system, though I'm not sure.
I'm usually shy of hyping My Preferred Language, but if it's the case that use of a language like Scala would help avoid these sorts of disasters, which, it seems to me, it would, then I suppose I'll have to.
posted by Anything at 11:59 PM on February 2, 2013
... Having a type declaration at the deserializer call site might be good practice in this sort of situation, though, even if the type would be checked elsewhere as well.
posted by Anything at 12:08 AM on February 3, 2013
posted by Anything at 12:08 AM on February 3, 2013
So a hackable language (in the sense of exploratory development) becomes a hackable app (in the sense of easy to break into and exploit).
I also feel that this aspect of its design leads to its reputation for inability to scale.
Things Just Working is great in some aspects of computing (hey, I develop on MBPs and love it). But when you start to get to Serious Amounts Of Data, you -really- need to be able to understand what's happening, and change it at very low levels. You also need your frameworks to not be doing a lot of "helpful" things - not only because that helpfulness takes resources, but also because it can have very unexpected side effects that can cause hair removingly subtle bugs.
I don't have examples for Ruby or Rails, because I've had less than zero reason to ever use them. And I'm torn, because I don't think programming should be an elite club that only those of us who enjoy writing serializers and using synchronization primitives. But Ruby/Rails, as interesting as they are from a language perspective... I am not left feeling like it's a net positive.
posted by flaterik at 12:26 AM on February 3, 2013 [1 favorite]
I also feel that this aspect of its design leads to its reputation for inability to scale.
Things Just Working is great in some aspects of computing (hey, I develop on MBPs and love it). But when you start to get to Serious Amounts Of Data, you -really- need to be able to understand what's happening, and change it at very low levels. You also need your frameworks to not be doing a lot of "helpful" things - not only because that helpfulness takes resources, but also because it can have very unexpected side effects that can cause hair removingly subtle bugs.
I don't have examples for Ruby or Rails, because I've had less than zero reason to ever use them. And I'm torn, because I don't think programming should be an elite club that only those of us who enjoy writing serializers and using synchronization primitives. But Ruby/Rails, as interesting as they are from a language perspective... I am not left feeling like it's a net positive.
posted by flaterik at 12:26 AM on February 3, 2013 [1 favorite]
I'll add that in a language like Scala you wouldn't even need to declare the desired secure type at the site of the call to the deserializer -- if anywhere down the line your code expects the deserialized object to be an instance of your MyNiceType, the deserializer method won't ever get called if its return type is SkeevyBroadType. In fact the code won't even compile -- I suppose similar safety might be also possible in a dynamic type system, though I'm not sure.
Strict originalism in data types, eh? Are you sure you're not programming in Scalia?
posted by snuffleupagus at 5:17 AM on February 3, 2013 [3 favorites]
Strict originalism in data types, eh? Are you sure you're not programming in Scalia?
posted by snuffleupagus at 5:17 AM on February 3, 2013 [3 favorites]
I'm not at all happy about the number of vulnerabilities that continue to appear in the world's deployed codebase (especially Java.... WTF, Oracle?), but I still feel entitled to emit one tiny crocodile tear of schadenfreude over these RoR issues.
I'm self taught and old-school; over the span of roughly 15 years I learned Perl for CGI, then Java (+ Oracle SQL), later C# + MS SQL Server, PHP + MySQL and currently Python + Mongo. And of course several frameworks in most of those languages. I had the good fortune of landing in a larger company where there were some good mentors and architects to teach me good design practices and to insist on code quality. With such help I got fairly good at db design, creating useful objects, proper separation of presentation, logic and model layers, performance and security, etc.
So, when something like Ruby on Rails came out and garnered all that hype (and DHH was cover-boy on Linux Journal, etc), I read up on it.... and my takeaway was that it solved a bunch of problems that frankly, I didn't have. I didn't need some app framework to completely take over the task of db design, I already had good patterns that were easy to reuse, and I had enough skills to design a configurable app to handle different forms, as opposed to rolling a new app for every new form. Also, I'd heard from people I respect that RoR was a fast start, but didn't scale up very well. I don't know whether that's changed.
So, I've stayed mostly away from "app-in-a-box" approaches like RoR, and since I have the luxury of mostly working on enterprise-level back-ends, I haven't done much with site frameworks like Drupal and such. In my most current contract, I'm helping re-host and clean up some Django sites (which were poorly built to begin with). I've had to learn Python, which is actually kind of cool, and I'm learning about the Django way of doing things... but these things have been a royal pain in the ass to troubleshoot and redeploy. Once again, the "speed" of the Django approach allowed someone to produce hard-to-maintain code. (Now I'm sure a good Django developer wouldn't have these problems... but can they lift the lid on the code and really customize it, or make changes to a table schema without the whole thing breaking?)
Since I expect to retire (or be forced out) from app development in maybe 5 years, perhaps I'm a little bit crotchety now, and more than a little jealous of the young'uns working with concepts and complexity that are mostly out of my grasp... still, I feel a little bit vindicated for the calls and choices I've made, and that while I'm not God's Gift to app development, the majority of projects I've put my hands to have not crashed spectacularly or risen up to bite me in the ass.
(omg, I'm a programming conservative. Get offa my server lawn!)
posted by Artful Codger at 10:24 AM on February 3, 2013 [3 favorites]
I'm self taught and old-school; over the span of roughly 15 years I learned Perl for CGI, then Java (+ Oracle SQL), later C# + MS SQL Server, PHP + MySQL and currently Python + Mongo. And of course several frameworks in most of those languages. I had the good fortune of landing in a larger company where there were some good mentors and architects to teach me good design practices and to insist on code quality. With such help I got fairly good at db design, creating useful objects, proper separation of presentation, logic and model layers, performance and security, etc.
So, when something like Ruby on Rails came out and garnered all that hype (and DHH was cover-boy on Linux Journal, etc), I read up on it.... and my takeaway was that it solved a bunch of problems that frankly, I didn't have. I didn't need some app framework to completely take over the task of db design, I already had good patterns that were easy to reuse, and I had enough skills to design a configurable app to handle different forms, as opposed to rolling a new app for every new form. Also, I'd heard from people I respect that RoR was a fast start, but didn't scale up very well. I don't know whether that's changed.
So, I've stayed mostly away from "app-in-a-box" approaches like RoR, and since I have the luxury of mostly working on enterprise-level back-ends, I haven't done much with site frameworks like Drupal and such. In my most current contract, I'm helping re-host and clean up some Django sites (which were poorly built to begin with). I've had to learn Python, which is actually kind of cool, and I'm learning about the Django way of doing things... but these things have been a royal pain in the ass to troubleshoot and redeploy. Once again, the "speed" of the Django approach allowed someone to produce hard-to-maintain code. (Now I'm sure a good Django developer wouldn't have these problems... but can they lift the lid on the code and really customize it, or make changes to a table schema without the whole thing breaking?)
Since I expect to retire (or be forced out) from app development in maybe 5 years, perhaps I'm a little bit crotchety now, and more than a little jealous of the young'uns working with concepts and complexity that are mostly out of my grasp... still, I feel a little bit vindicated for the calls and choices I've made, and that while I'm not God's Gift to app development, the majority of projects I've put my hands to have not crashed spectacularly or risen up to bite me in the ass.
(omg, I'm a programming conservative. Get offa my server lawn!)
posted by Artful Codger at 10:24 AM on February 3, 2013 [3 favorites]
Artful Codger: "I'd heard from people I respect that RoR was a fast start, but didn't scale up very well. I don't know whether that's changed."
"scaling" ruby consists of running it on more cores. It is pretty much dead last performance wise for backend implementation languages. IIRC in idiomatic usage, ruby is even slower than bash (where every non-trivial action requires spawning a new OS level process to do it in).
Artful Codger: "jealous of the young'uns working with concepts and complexity that are mostly out of my grasp..."
Don't worry, they don't grasp the concepts or the complexity either, which is what led to these problems in the first place.
posted by idiopath at 11:27 AM on February 3, 2013 [3 favorites]
"scaling" ruby consists of running it on more cores. It is pretty much dead last performance wise for backend implementation languages. IIRC in idiomatic usage, ruby is even slower than bash (where every non-trivial action requires spawning a new OS level process to do it in).
Artful Codger: "jealous of the young'uns working with concepts and complexity that are mostly out of my grasp..."
Don't worry, they don't grasp the concepts or the complexity either, which is what led to these problems in the first place.
posted by idiopath at 11:27 AM on February 3, 2013 [3 favorites]
kenko: "(I don't know how much clojars needs, actually, but I'd assume that the data it wants from project.clj or whatever is unlikely to be determined at runtime.)"
It is always loaded at runtime, project.clj is a file included in each clojure jar uploaded to clojars, that contains metadata about that project (version, dependencies, licensing, etc.). This is a direct analog to the ruby gems / yaml situation (or the closest that exists in the clojure world).
If you look at the standard set of reader macros in clojure, as long as *read-eval* is turned off and you haven't added anything wacky to the read table, read can't in principle do much the most commonly used json and xml parsers for clojure couldn't do in terms of causing arbitrary code to run.
But you are probably right, better to make a custom parser that can handle {:key "value}, ["vector" "of" "items"] and can read things like [2r100 8r10 8] and know they are all the same thing. My worry here being that by the time you are done with all the corner cases, you have something that does exactly what read did (if *read-eval* was off) and runs in n^2 complexity instead of read's linear time (this is hyperbole and not meant to be a true description of the algorithmic complexity of the implementations).
I was doing some performance testing recently, because a coworker thought maybe using read-string in our templates would introduce unwanted overhead, and we discovered to our surprise (read-string "abc") runs significantly faster than (string-match #"a" "abc"). This performance lead is maintained even if you precompute the #"a" regex instead of specifying it via a read macro as we do here (ie. (let [re #"a"] (dotimes [i 10000] (string-match re "abc"))) is slower than (dotimes [i 10000] (read-string "abc")). And then, mind you, in the read-string case you are done - the object is deserialized into the proper datatype, but string-match has only provided the match, and you next have to verify the match and either check the next matcher or construct the proper result. And of course this is close to the simplest regex one could match, and for one that actually matches the datatypes we want, we would be seeing even larger differences in performance.
posted by idiopath at 11:55 AM on February 3, 2013
It is always loaded at runtime, project.clj is a file included in each clojure jar uploaded to clojars, that contains metadata about that project (version, dependencies, licensing, etc.). This is a direct analog to the ruby gems / yaml situation (or the closest that exists in the clojure world).
If you look at the standard set of reader macros in clojure, as long as *read-eval* is turned off and you haven't added anything wacky to the read table, read can't in principle do much the most commonly used json and xml parsers for clojure couldn't do in terms of causing arbitrary code to run.
But you are probably right, better to make a custom parser that can handle {:key "value}, ["vector" "of" "items"] and can read things like [2r100 8r10 8] and know they are all the same thing. My worry here being that by the time you are done with all the corner cases, you have something that does exactly what read did (if *read-eval* was off) and runs in n^2 complexity instead of read's linear time (this is hyperbole and not meant to be a true description of the algorithmic complexity of the implementations).
I was doing some performance testing recently, because a coworker thought maybe using read-string in our templates would introduce unwanted overhead, and we discovered to our surprise (read-string "abc") runs significantly faster than (string-match #"a" "abc"). This performance lead is maintained even if you precompute the #"a" regex instead of specifying it via a read macro as we do here (ie. (let [re #"a"] (dotimes [i 10000] (string-match re "abc"))) is slower than (dotimes [i 10000] (read-string "abc")). And then, mind you, in the read-string case you are done - the object is deserialized into the proper datatype, but string-match has only provided the match, and you next have to verify the match and either check the next matcher or construct the proper result. And of course this is close to the simplest regex one could match, and for one that actually matches the datatypes we want, we would be seeing even larger differences in performance.
posted by idiopath at 11:55 AM on February 3, 2013
ph00dz: "'Course... that crowd has always been a little smug around us boring PHP folks... but... yeah. There's a curmudgeon in our office who eschews all frameworks because of these kinds of concerns."
Couldn't you argue that PHP's huge standard library makes it framework-esque by itself?
That said, for all of PHP's problems, at least the fucking JSON parser doesn't parse arbitrary code by default.
I'm not a Ruby person, but YAML's always struck me as one of those weirdly redundant inventions -- it's never seemed particularly necessary or convenient in any place that I've seen it applied. It also seems very surprising that Ruby's YAML parser could have had such dangerous functionality, and been in such widespread use, without anybody realizing it. I mean, these days, any use of
Shouldn't Ruby's object deserialization functions have been meticulously inspected and vetted, given that these functions make rather obvious attack vectors? Ned Batchelder's suggestion for providing separate
posted by schmod at 1:31 PM on February 3, 2013
Couldn't you argue that PHP's huge standard library makes it framework-esque by itself?
That said, for all of PHP's problems, at least the fucking JSON parser doesn't parse arbitrary code by default.
I'm not a Ruby person, but YAML's always struck me as one of those weirdly redundant inventions -- it's never seemed particularly necessary or convenient in any place that I've seen it applied. It also seems very surprising that Ruby's YAML parser could have had such dangerous functionality, and been in such widespread use, without anybody realizing it. I mean, these days, any use of
eval() in any language is avoided at all costs, and is meticulously scrutinized in cases where it's unavoidable. Why didn't YAML.load() get the same level of attention?Shouldn't Ruby's object deserialization functions have been meticulously inspected and vetted, given that these functions make rather obvious attack vectors? Ned Batchelder's suggestion for providing separate
.load() and .dangerous_load() methods seems to make a lot of sense, if we think that it's even a good idea for YAML to ever be allowed to parse completely arbitrary objects.posted by schmod at 1:31 PM on February 3, 2013
It is always loaded at runtime, project.clj is a file included in each clojure jar uploaded to clojars, that contains metadata about that project (version, dependencies, licensing, etc.). This is a direct analog to the ruby gems / yaml situation (or the closest that exists in the clojure world).
Right, but most of the relevant values in project.clj are actually literals. It's loaded at runtime but it doesn't need to be evaluated at runtime the way an arbitrary function would. Thus you wouldn't need a full Clojure interpreter to be able to read the relevant values out of project.clj, just one that can give you literal values.
posted by kenko at 3:37 PM on February 3, 2013
Right, but most of the relevant values in project.clj are actually literals. It's loaded at runtime but it doesn't need to be evaluated at runtime the way an arbitrary function would. Thus you wouldn't need a full Clojure interpreter to be able to read the relevant values out of project.clj, just one that can give you literal values.
posted by kenko at 3:37 PM on February 3, 2013
IIRC in idiomatic usage, ruby is even slower than bash (where every non-trivial action requires spawning a new OS level process to do it in).
whaaaat? Seriously?
posted by kenko at 3:39 PM on February 3, 2013
whaaaat? Seriously?
posted by kenko at 3:39 PM on February 3, 2013
Schmod -- absolutely you could make the argument that PHP, in and of itself, is a kind of framework. I certainly have. PHP was designed with websites in mind and while its been occasionally called into service to do other things, I'm not sure anyone really does anything significant with PHP beyond web stuff. One of the things I love about Zend as a framework that sits on top of PHP is that it doesn't demand any kind of full stack adherence. Generally speaking, Zend expects certain types of code to be in a certain place, but it doesn't try and over-reach with its completeness. That is to say, the general layout of code is dictated, but what all those underlying components do is still up to you. There's a bunch of helper stuff to help you if you want it, but you're not required to use any of the library functions. (Zend form? Whatever!) Of course, in the age of jQuery and client side templates, aren't doing much more than authentication and spitting out JSON these days.
That's not to say PHP doesn't have its share of problems. (Uhh... which parameter was the needle and which one was the haystack again?)
posted by ph00dz at 3:40 PM on February 3, 2013 [1 favorite]
That's not to say PHP doesn't have its share of problems. (Uhh... which parameter was the needle and which one was the haystack again?)
posted by ph00dz at 3:40 PM on February 3, 2013 [1 favorite]
That's not to say PHP doesn't have its share of problems. (Uhh... which parameter was the needle and which one was the haystack again?)In twelve years of PHP, I have never used
in_array() without having to look it up.posted by migurski at 4:02 PM on February 3, 2013 [2 favorites]
Man, that bastard son of an INI file format had it coming.
I really do not understand all the distate for YAML in this thread. It's a legible and structured format that makes writing config files where you need something more complicated than default=x actually pleasant.
That said, using it as an interchange format doesn't make sense, and I was completely unaware that you could load arbitrary objects (rather than just dicts or lists of strings and numbers), and that that's the default in multiple languages is just insane.
Is there a good alternative for lightly structured data? The only comparable thing I can think of is mail format.
posted by 23 at 11:22 PM on February 3, 2013
I really do not understand all the distate for YAML in this thread. It's a legible and structured format that makes writing config files where you need something more complicated than default=x actually pleasant.
That said, using it as an interchange format doesn't make sense, and I was completely unaware that you could load arbitrary objects (rather than just dicts or lists of strings and numbers), and that that's the default in multiple languages is just insane.
Is there a good alternative for lightly structured data? The only comparable thing I can think of is mail format.
posted by 23 at 11:22 PM on February 3, 2013
Ruby's a really, really nice language. Really. I love it.
And I just upgraded three rails 2.3.x apps, and next time I have a ruby web app to write, I'll definitely be thinking about looking into Sinatra, and HAML, and DataMapper or Sequel, rather than go through the overhead of learning Rails 3 and whatever massive amounts of backwards incompatibility and baffling magic it offers.
posted by Zed at 12:01 AM on February 4, 2013
And I just upgraded three rails 2.3.x apps, and next time I have a ruby web app to write, I'll definitely be thinking about looking into Sinatra, and HAML, and DataMapper or Sequel, rather than go through the overhead of learning Rails 3 and whatever massive amounts of backwards incompatibility and baffling magic it offers.
posted by Zed at 12:01 AM on February 4, 2013
> I really do not understand all the distaste for YAML in this thread
YAML was developed in a hissy fit of "bloo bloo bloo xml is hard and slow bloo bloo here's my markup language that does what I need and is fast bloo bloo everyone will love it bloo bloo sniff". At the time, the careful XML user was well able to handle those difficult edge-cases like character encoding and escaping. Yes, XML was slow a decade ago, but then again, so was everything.
Lightweight markup? You can either have explicit markup which is hard for humans to read/write, but is easy to parse in a computer, or implicit markup which reads and writes a dream but is a royal pain (filled with regexes and heuristics) to parse. Users will always try to extend implicit markup to beyond its useful scope, so parsers will slowly creep towards uselessness. Might as well embrace the XML.
posted by scruss at 10:49 AM on February 4, 2013 [2 favorites]
YAML was developed in a hissy fit of "bloo bloo bloo xml is hard and slow bloo bloo here's my markup language that does what I need and is fast bloo bloo everyone will love it bloo bloo sniff". At the time, the careful XML user was well able to handle those difficult edge-cases like character encoding and escaping. Yes, XML was slow a decade ago, but then again, so was everything.
Lightweight markup? You can either have explicit markup which is hard for humans to read/write, but is easy to parse in a computer, or implicit markup which reads and writes a dream but is a royal pain (filled with regexes and heuristics) to parse. Users will always try to extend implicit markup to beyond its useful scope, so parsers will slowly creep towards uselessness. Might as well embrace the XML.
posted by scruss at 10:49 AM on February 4, 2013 [2 favorites]
YAML was developed in a hissy fit of "bloo bloo bloo xml is hard and slow bloo bloo here's my markup language
Thank you for the explanation; that puts things in context for me.
Lightweight markup? You can either have explicit markup which is hard for humans to read/write, but is easy to parse in a computer, or implicit markup which reads and writes a dream but is a royal pain (filled with regexes and heuristics) to parse.
If you're serializing arbitrary binary objects, sure. If you're only allowing lists or dicts of strings and numbers I don't think it's that hard to use a format which is both readable and easy to parse - JSON or even S-Expressions (with a hack for hashes) do just fine here.
posted by 23 at 5:37 PM on February 4, 2013
Thank you for the explanation; that puts things in context for me.
Lightweight markup? You can either have explicit markup which is hard for humans to read/write, but is easy to parse in a computer, or implicit markup which reads and writes a dream but is a royal pain (filled with regexes and heuristics) to parse.
If you're serializing arbitrary binary objects, sure. If you're only allowing lists or dicts of strings and numbers I don't think it's that hard to use a format which is both readable and easy to parse - JSON or even S-Expressions (with a hack for hashes) do just fine here.
posted by 23 at 5:37 PM on February 4, 2013
« Older "The announcement was an honest look at the World... | Going Clear Blocked By UK Libel Laws Newer »
This thread has been archived and is closed to new comments
posted by iamabot at 6:06 PM on February 1, 2013