Gone in Six Characters
April 13, 2016 9:04 AM Subscribe
Gone in Six Characters: Short URLs Considered Harmful for Cloud Services [abstract] [pdf]
In this paper, we demonstrate that the space of 5- and 6-character tokens included in short URLs is so small that it can be scanned using brute-force search. Therefore, all online resources that were intended to be shared with a few trusted friends or collaborators are effectively public and can be accessed by anyone. This leads to serious security and privacy vulnerabilities.
In the case of cloud storage, we focus on Microsoft OneDrive. We show how to use short-URL enumeration to discover and read shared content stored in the OneDrive cloud, including even files for which the user did not generate a short URL. 7% of the OneDrive accounts exposed in this fashion allow anyone to write into them. Since cloud-stored files are automatically copied into users’ personal computers and devices, this is a vector for large-scale, automated malware injection.
URL shorteners are not supposed to obscure private resources.
This isn't a problem for URL shorteners exactly, but for using short identifiers (which could be tagged on the end of a long URL) as a way to gate resources that users will assume are private.
It's especially troubling when it comes to write access!
posted by grobstein at 9:15 AM on April 13, 2016 [15 favorites]
This isn't a problem for URL shorteners exactly, but for using short identifiers (which could be tagged on the end of a long URL) as a way to gate resources that users will assume are private.
It's especially troubling when it comes to write access!
posted by grobstein at 9:15 AM on April 13, 2016 [15 favorites]
Twitter, evidently.
posted by Dr. Send at 9:15 AM on April 13, 2016 [3 favorites]
posted by Dr. Send at 9:15 AM on April 13, 2016 [3 favorites]
URLs are not secrets.
posted by Nelson at 9:16 AM on April 13, 2016 [23 favorites]
posted by Nelson at 9:16 AM on April 13, 2016 [23 favorites]
Wasn't it more people who were using twitter and forced to work around its limitations?
posted by cosmic.osmo at 9:20 AM on April 13, 2016 [2 favorites]
posted by cosmic.osmo at 9:20 AM on April 13, 2016 [2 favorites]
This is why all my file names make Fiona Apple's second album title look short.
alternative link
posted by Nanukthedog at 9:25 AM on April 13, 2016 [1 favorite]
alternative link
posted by Nanukthedog at 9:25 AM on April 13, 2016 [1 favorite]
See also: pastebin.com scraping.
posted by Monday, stony Monday at 9:25 AM on April 13, 2016
posted by Monday, stony Monday at 9:25 AM on April 13, 2016
I haven't read the paper closely but I assume the problem is not specifically that the URLs are short therefore bad. It's that being able to guess the URL is sufficient to fuck with random other people's files. And that is a problem of the site host doing a bad job of managing user accounts, which has nothing to do with URLs.
URL shorteners predate Twitter by quite a few years, and randomly guessing shortened URLs has been a recreational pursuit for about as long.
posted by ardgedee at 9:26 AM on April 13, 2016 [5 favorites]
URL shorteners predate Twitter by quite a few years, and randomly guessing shortened URLs has been a recreational pursuit for about as long.
posted by ardgedee at 9:26 AM on April 13, 2016 [5 favorites]
Who the hell ever thought URL shorteners were a good idea in this day and age, anyway?
URL shorteners are a godsend for linking to github gists and other resources that end in huge strings of random numbers
posted by Going To Maine at 9:28 AM on April 13, 2016 [5 favorites]
URL shorteners are a godsend for linking to github gists and other resources that end in huge strings of random numbers
posted by Going To Maine at 9:28 AM on April 13, 2016 [5 favorites]
So...is there anything I should do to insure my OneDrive stored files are secure?
posted by Chrysostom at 9:29 AM on April 13, 2016
posted by Chrysostom at 9:29 AM on April 13, 2016
The way to address security through obscurity scenarios isn't "make the key longer, and therefore more obscure." The issue here is that some resources are, for lack of a common security standard, really irritating to share to a limited audience. If I send a link to a half-dozen people to a file I've shared on Dropbox, OneDrive, flickr, or any other site where I want the data to not appear in a public file listing, it's likely people are going to be annoyed if they have to have an account on those sites in order to download what's at the end of the link.
There's no good single-key way to do so that can't, with some effort, be scanned. You could have single-use keys, but you still run the chance of someone scanning possible keys and getting the file before you do. You can use really long keys, which is what this proposes, and rate limit the number of requests any given user can make, but you can partially foul any rate limiting by using a distributed network.
So if you want your file to be secure, stick it behind a username and password or any other multi-factor approach. Sucks, but it's what (mostly) works.
posted by mikeh at 9:29 AM on April 13, 2016 [10 favorites]
There's no good single-key way to do so that can't, with some effort, be scanned. You could have single-use keys, but you still run the chance of someone scanning possible keys and getting the file before you do. You can use really long keys, which is what this proposes, and rate limit the number of requests any given user can make, but you can partially foul any rate limiting by using a distributed network.
So if you want your file to be secure, stick it behind a username and password or any other multi-factor approach. Sucks, but it's what (mostly) works.
posted by mikeh at 9:29 AM on April 13, 2016 [10 favorites]
"In March of 2016, Microsoft removed the “shorten link” option from OneDrive, causing a number of user complaints. We asked MSRC whether this change was made in response to our previous report. MSRC informed us that our analysis played no role in their decision to remove this option and reiterated that they do not consider our report a security vulnerability."
You stay classy Microsoft security...
posted by zachlipton at 9:29 AM on April 13, 2016 [8 favorites]
You stay classy Microsoft security...
posted by zachlipton at 9:29 AM on April 13, 2016 [8 favorites]
Meanwhile, OneDrive seems to have eliminated shortening urls from their interface. If I click "share" on a file or folder, it gives me a long url, but the option to shorten the link is gone.
bit.ly still works on them, though.
posted by zarq at 9:30 AM on April 13, 2016
bit.ly still works on them, though.
posted by zarq at 9:30 AM on April 13, 2016
No website can do much about people using third party url shorteners, short of detecting the referer and reacting appropriately (and that is like swatting roaches). It's arguably better to let the users hang by their own actions if they feel determined to expose vulnerable data through a shortened.
posted by ardgedee at 9:39 AM on April 13, 2016 [1 favorite]
posted by ardgedee at 9:39 AM on April 13, 2016 [1 favorite]
RE: shorteners being a good idea, there are a lot of garbagey long URLs out there, where bits of crap are added to the end of the URL to show how you got there -- whether it was from an RSS feed or a search, and what the keywords were of the search, and shit like that. God knows what else. Amazon is a significant offender; so is Youtube.
When you want to share these with someone, and you don't want to spam them with a screen of URL garbage, you can either go through and try and figure out what parts of the URL you can safely delete (ideal) or drop them into a URL shortener (less ideal, because you preserve all that bullshit).
Having decent, God-fearing, clean URLs in the first place would be better still of course.
posted by edheil at 9:43 AM on April 13, 2016 [5 favorites]
When you want to share these with someone, and you don't want to spam them with a screen of URL garbage, you can either go through and try and figure out what parts of the URL you can safely delete (ideal) or drop them into a URL shortener (less ideal, because you preserve all that bullshit).
Having decent, God-fearing, clean URLs in the first place would be better still of course.
posted by edheil at 9:43 AM on April 13, 2016 [5 favorites]
that is a problem of the site host doing a bad job of managing user accounts, which has nothing to do with URLs
mikeh and Nelson really have it. improving security-via-obfuscation to be more secure with *more* obfuscation is a non-starter.
And, it's market-forces, really. A significant fraction of the user base probably tested as needing something as simple as 'email a url' for access control. long or short, it's still obfuscation. the only reason shorter is 'somewhat worse' is simplifying a brute force attack - which is totally old news.
posted by j_curiouser at 9:47 AM on April 13, 2016 [3 favorites]
mikeh and Nelson really have it. improving security-via-obfuscation to be more secure with *more* obfuscation is a non-starter.
And, it's market-forces, really. A significant fraction of the user base probably tested as needing something as simple as 'email a url' for access control. long or short, it's still obfuscation. the only reason shorter is 'somewhat worse' is simplifying a brute force attack - which is totally old news.
posted by j_curiouser at 9:47 AM on April 13, 2016 [3 favorites]
It's unfortunate that we're stuck with 1980s legacy services like Twitter that are limited to 140 characters and cannot recognize URLs to avoid breaking compatibility with mainframe platforms; otherwise nobody would need this awful hack of URL shorteners.
posted by indubitable at 10:09 AM on April 13, 2016 [6 favorites]
posted by indubitable at 10:09 AM on April 13, 2016 [6 favorites]
When you want to share these with someone, and you don't want to spam them with a screen of URL garbage, you can either go through and try and figure out what parts of the URL you can safely delete (ideal) or drop them into a URL shortener (less ideal, because you preserve all that bullshit).
Perhaps this doesn't meet your criteria for ease-of-use, but you can always share youtube videos the way they intend you to, with the "share" function below every video. It generates a shorted URL.
posted by hellphish at 10:21 AM on April 13, 2016 [1 favorite]
Perhaps this doesn't meet your criteria for ease-of-use, but you can always share youtube videos the way they intend you to, with the "share" function below every video. It generates a shorted URL.
posted by hellphish at 10:21 AM on April 13, 2016 [1 favorite]
A truly random URL with enough entropy (EG: a random 200 character string composed of upper and lower case alphanumerics plus symbols will have 1013 bits of entropy, ain't no one guessing that or even significantly exploring the hash space) is sufficiently obscure as to be proof against brute force or guessing attacks. Especially if you combine it with one time use and time expiry. Which is fine in a lot of cases. It won't stop someone you send a link to from sharing it with others (but you realistically can't stop that anyways) but it'll stop some random bot or script kiddie from owning your stuff.
But feeding it into a shorter (built in or 3rd party) collapses the name space and makes it trivial to access that information.
mikeh: "So if you want your file to be secure, stick it behind a username and password or any other multi-factor approach. Sucks, but it's what (mostly) works."
Both Flickr and Google Photos allow you to privately share but only to logged in users (an obvious ploy to increase user base) which is annoying as hell.
posted by Mitheral at 10:56 AM on April 13, 2016 [2 favorites]
But feeding it into a shorter (built in or 3rd party) collapses the name space and makes it trivial to access that information.
mikeh: "So if you want your file to be secure, stick it behind a username and password or any other multi-factor approach. Sucks, but it's what (mostly) works."
Both Flickr and Google Photos allow you to privately share but only to logged in users (an obvious ploy to increase user base) which is annoying as hell.
posted by Mitheral at 10:56 AM on April 13, 2016 [2 favorites]
So don't put private information onto publicly-accessible parts of the internet, eh? URLs are not a security feature; their whole purpose is to make information easier to find. If you're relying on someone not guessing the URL to protect your private data, I don't really know what to tell you. This is just the newest incarnation of a problem that has existed since the beginning of the web. Used to be, you could often guess your way into a webserver's filesystem if the admin hadn't locked it down—which in those earlier and more naive times, they often hadn't. It was one way to find things like porn and illicit mp3 files, once upon a time.
One thing that might be feasible is to make the files that are being shared via public link cloud drive thingies publicly accessible only for defined time windows. Like, if you generate a public link for a file in your dropbox so that you can share some big file with a friend, you would get to set how long you want the link to be valid for. You could even make it permanently valid if you didn't care about someone else one day stumbling upon it, but if it was something sensitive you could make it disappear after an hour or whatever. That would at least be somewhat secure, assuming that nobody is actively snooping on you.
posted by Anticipation Of A New Lover's Arrival, The at 11:17 AM on April 13, 2016 [2 favorites]
One thing that might be feasible is to make the files that are being shared via public link cloud drive thingies publicly accessible only for defined time windows. Like, if you generate a public link for a file in your dropbox so that you can share some big file with a friend, you would get to set how long you want the link to be valid for. You could even make it permanently valid if you didn't care about someone else one day stumbling upon it, but if it was something sensitive you could make it disappear after an hour or whatever. That would at least be somewhat secure, assuming that nobody is actively snooping on you.
posted by Anticipation Of A New Lover's Arrival, The at 11:17 AM on April 13, 2016 [2 favorites]
You could even make it permanently valid if you didn't care about someone else one day stumbling upon it, but if it was something sensitive you could make it disappear after an hour or whatever.
The main problem with this approach is that the number of people who will select the "make this permanently valid" option simply because they can't be arsed to actually think about how long they'll need to share the link will far outstrip the number of people who use the feature as intended.
I'd propose instead a maximum link duration on the order of two weeks, with notification before the link expires and an easy method to renew the shared link (which would actually generate a new shortcode). If it's good enough for OAuth...
posted by tobascodagama at 11:27 AM on April 13, 2016 [2 favorites]
The main problem with this approach is that the number of people who will select the "make this permanently valid" option simply because they can't be arsed to actually think about how long they'll need to share the link will far outstrip the number of people who use the feature as intended.
I'd propose instead a maximum link duration on the order of two weeks, with notification before the link expires and an easy method to renew the shared link (which would actually generate a new shortcode). If it's good enough for OAuth...
posted by tobascodagama at 11:27 AM on April 13, 2016 [2 favorites]
I experienced a real gotcha with using shortlinks last year.
The company I worked at had branded email signatures attached automatically to every outgoing email. Thankfully these were not the typical abuse of email signatures and were relatively compact, bereft of crap social 'share us' links, and helpfully included our office addresses with direct links to Google maps. We were a digital marketing agency that included a lot of publicity worked, so we took pride in our brand and used email like people chain-smoke cigarettes.
We used goo.gl short links for those map addresses because, hey why not, save a few bytes on the email sig and Google Maps helpfully provides a shortcut to do that right there. Ace.
We had these email sigs going for a few good years without issue. We started experiencing fake spam problems with our email though - as in other people's email systems rejecting our email. Then it started getting worse - email chains replying to us started getting randomly blocked by our own spam systems. We performed the usual triaging - whitelisting tons of addresses, dealing down detection settings, checking our SPF and DKIM settings, trying to get information from the various blockers, combing through headers, sending test emails out the wazoo, tweaking email content - the whole nine yards. External email providers weren't particularly helpful when we asked them for any info on why they were rejecting our email, and usually just said something along the lines of 'blocked for containing objectionable content' and closed the ticket.
It was finally in conversation number 4080 with our own email filterer that they identified the issue.
“A link you’ve got in your signature is causing rejections because it goes to an objectionable website.”
OK then. But we’ve just gone through all four goo.gl links again and they’re just going to Google Maps. Are you sure?
“Yes, that’s definitely it. The system is picking it up and I can see that it’s got objectionable content. The link is http://goo.gl/5lbvvh”.
- Our link was http://goo.gl/5QbvVh and went to Google Maps.
- The problem link was http://goo.gl/5qbvvh and went to some dodgy Russian porn site.
- goo.gl short links are case sensitive.
Some email filtering systems hadn’t been programmed to check links case-sensitively, and were instead performing a content check on the lower-case version of the URL, which indeed was dodgy ; even though it technically never existed in our email. The reason we went for ages without a problem was that the lowercase version never existed until one day when Porny McPornovich created a short link to their site, Google dished out the same letters in lower case, and our email goose got cooked.
posted by sektah at 12:00 PM on April 13, 2016 [19 favorites]
The company I worked at had branded email signatures attached automatically to every outgoing email. Thankfully these were not the typical abuse of email signatures and were relatively compact, bereft of crap social 'share us' links, and helpfully included our office addresses with direct links to Google maps. We were a digital marketing agency that included a lot of publicity worked, so we took pride in our brand and used email like people chain-smoke cigarettes.
We used goo.gl short links for those map addresses because, hey why not, save a few bytes on the email sig and Google Maps helpfully provides a shortcut to do that right there. Ace.
We had these email sigs going for a few good years without issue. We started experiencing fake spam problems with our email though - as in other people's email systems rejecting our email. Then it started getting worse - email chains replying to us started getting randomly blocked by our own spam systems. We performed the usual triaging - whitelisting tons of addresses, dealing down detection settings, checking our SPF and DKIM settings, trying to get information from the various blockers, combing through headers, sending test emails out the wazoo, tweaking email content - the whole nine yards. External email providers weren't particularly helpful when we asked them for any info on why they were rejecting our email, and usually just said something along the lines of 'blocked for containing objectionable content' and closed the ticket.
It was finally in conversation number 4080 with our own email filterer that they identified the issue.
“A link you’ve got in your signature is causing rejections because it goes to an objectionable website.”
OK then. But we’ve just gone through all four goo.gl links again and they’re just going to Google Maps. Are you sure?
“Yes, that’s definitely it. The system is picking it up and I can see that it’s got objectionable content. The link is http://goo.gl/5lbvvh”.
- Our link was http://goo.gl/5QbvVh and went to Google Maps.
- The problem link was http://goo.gl/5qbvvh and went to some dodgy Russian porn site.
- goo.gl short links are case sensitive.
Some email filtering systems hadn’t been programmed to check links case-sensitively, and were instead performing a content check on the lower-case version of the URL, which indeed was dodgy ; even though it technically never existed in our email. The reason we went for ages without a problem was that the lowercase version never existed until one day when Porny McPornovich created a short link to their site, Google dished out the same letters in lower case, and our email goose got cooked.
posted by sektah at 12:00 PM on April 13, 2016 [19 favorites]
If you provide an easy-to-use security feature and someone decides not to use it and gets owned because of that, I say fuck 'em. You can't engineer around stupidity. And anyway, I have lots of use cases where I want to have an indefinitely-available public Dropbox link for things that are totally not sensitive at all. I figure that this time-window thing is really just a good-enough level of security for stuff that's only mildly sensitive. For important things, one should really be using something that requires authentication.
posted by Anticipation Of A New Lover's Arrival, The at 1:16 PM on April 13, 2016
posted by Anticipation Of A New Lover's Arrival, The at 1:16 PM on April 13, 2016
I mean, Mitheral is right -- a long enough GUID-type string is going to be nearly impossible to data mine, assuming it's generated intelligently. Internal systems use system-generated GUIDs to create unique keys for databases, and I've used that type of code to insert umpty-million rows without conflict.
As long as you're on a secure-ish network, it's fine. If anyone can grab your URL, you're probably in trouble anyway, right?
posted by mikeh at 1:17 PM on April 13, 2016
As long as you're on a secure-ish network, it's fine. If anyone can grab your URL, you're probably in trouble anyway, right?
posted by mikeh at 1:17 PM on April 13, 2016
Perhaps this doesn't meet your criteria for ease-of-use, but you can always share youtube videos the way they intend you to, with the "share" function below every video. It generates a shorted URL.You know, I've trained myself not to even see "share" buttons anymore, so I normally woudln't think of this, but it sounds like that would be a good plan too. :)
posted by edheil at 1:21 PM on April 13, 2016 [1 favorite]
mikeh: "As long as you're on a secure-ish network, it's fine. "
HTTPS sessions encrypt the URL. The only thing exposed to sniffers is the domain.
posted by Mitheral at 1:31 PM on April 13, 2016
HTTPS sessions encrypt the URL. The only thing exposed to sniffers is the domain.
posted by Mitheral at 1:31 PM on April 13, 2016
If it's good enough for OAuth...
actual OAuth is the right answer. Sign in somewhere I trust to get my thing. The url solution - the less-tech-savvy solution - is just a design trade-off.
Both these options are available in OneDrive, so if you want to be secure-er, just do OAuth. Nobody is *making* you email urls.
By the time you add in the expiry and management features of a complex url-based security approach...it converges on OAuth.
posted by j_curiouser at 1:38 PM on April 13, 2016 [1 favorite]
actual OAuth is the right answer. Sign in somewhere I trust to get my thing. The url solution - the less-tech-savvy solution - is just a design trade-off.
Both these options are available in OneDrive, so if you want to be secure-er, just do OAuth. Nobody is *making* you email urls.
By the time you add in the expiry and management features of a complex url-based security approach...it converges on OAuth.
posted by j_curiouser at 1:38 PM on April 13, 2016 [1 favorite]
Well, I think there are two separate problems here: access control and discoverability.
OAuth is a technology that can be part of the solution to the access control problem, for sure.
Discoverability of things that shouldn't be discoverable is a different issue. Yes, from one view, making your space of possible identifiers too big to brute force is just "security through obscurity", but from another view it's a sensible first step that complements your actually-for-real security measures. Same reason login/password-recovery pages try to avoid leaking information about what usernames exist in the system: you have to know the safe is there before you can attempt to crack it.
posted by tobascodagama at 1:54 PM on April 13, 2016 [2 favorites]
OAuth is a technology that can be part of the solution to the access control problem, for sure.
Discoverability of things that shouldn't be discoverable is a different issue. Yes, from one view, making your space of possible identifiers too big to brute force is just "security through obscurity", but from another view it's a sensible first step that complements your actually-for-real security measures. Same reason login/password-recovery pages try to avoid leaking information about what usernames exist in the system: you have to know the safe is there before you can attempt to crack it.
posted by tobascodagama at 1:54 PM on April 13, 2016 [2 favorites]
Yes, from one view, making your space of possible identifiers too big to brute force is just "security through obscurity"
In the same way that correctly-implemented cryptography is. You're just obscuring the plaintext, after all :-)
posted by indubitable at 1:58 PM on April 13, 2016 [1 favorite]
In the same way that correctly-implemented cryptography is. You're just obscuring the plaintext, after all :-)
posted by indubitable at 1:58 PM on April 13, 2016 [1 favorite]
Mitheral: Both Flickr and Google Photos allow you to privately share but only to logged in users (an obvious ploy to increase user base) which is annoying as hell.
I don't see how this is an "obvious ploy". What's wrong with a site providing an easy way to share content with other users of that site?
BTW, Google Photos does allow you to share private content with people who are not logged in, by creating a special URL that can be accessed by non-logged-in users. Unfortunately it's less secure, because if someone you share the URL with goes on to share it with a third party (either maliciously or carelessly), you have no way of knowing who has access to your private photo, and no way to stop them from viewing it, short of deleting the photo. This problem could be ameliorated somewhat (e.g. by making it possible to deactivate the URL), but as others have pointed out in this thread, URLs basically suck as a mechanism for restricting access to sensitive content.
posted by shponglespore at 4:14 PM on April 13, 2016
I don't see how this is an "obvious ploy". What's wrong with a site providing an easy way to share content with other users of that site?
BTW, Google Photos does allow you to share private content with people who are not logged in, by creating a special URL that can be accessed by non-logged-in users. Unfortunately it's less secure, because if someone you share the URL with goes on to share it with a third party (either maliciously or carelessly), you have no way of knowing who has access to your private photo, and no way to stop them from viewing it, short of deleting the photo. This problem could be ameliorated somewhat (e.g. by making it possible to deactivate the URL), but as others have pointed out in this thread, URLs basically suck as a mechanism for restricting access to sensitive content.
posted by shponglespore at 4:14 PM on April 13, 2016
Well if I was king of the internet these companies would allow authentication by password. You could send the link and the password could be communicated out of band. This sort of thing used to really common but seems to have been superseded by facebook, google or other tracking logins.
posted by Mitheral at 4:38 PM on April 13, 2016 [1 favorite]
posted by Mitheral at 4:38 PM on April 13, 2016 [1 favorite]
First thought: Well, duh.
Second thought: Well, shit.
posted by OverlappingElvis at 1:18 PM on April 14, 2016 [1 favorite]
Second thought: Well, shit.
posted by OverlappingElvis at 1:18 PM on April 14, 2016 [1 favorite]
« Older I looked up and there you were | Curbing toxic game culture Newer »
This thread has been archived and is closed to new comments
posted by Faint of Butt at 9:06 AM on April 13, 2016 [3 favorites]