That Syncing Feeling

May 13, 2011 4:19 PM

Christopher Soghoian, who exposed the latest Facebook PR move, is now filing an FTC complaint (pdf) against Dropbox on the grounds that they gained unfair competitive advantage by lying about how files are encrypted and who has access to them. Dropbox explains how safe your files are.
posted by swift (44 comments total)

I love me some DropBox. Hard to explain why it doesn't suck as much as shared drives should, but it doesn't.

But if you believe any security statements juxtaposed to an illustration of a time-traveling DeLorean, you deserve what you get.
posted by RobotVoodooPower at 4:32 PM on May 13, 2011

But if you believe any security statements juxtaposed to an illustration of a time-traveling DeLorean, you deserve what you get.

Maybe you should read the links.
posted by Foci for Analysis at 4:38 PM on May 13, 2011 [4 favorites]

Dropbox has always struck me as a little too good to be true. I guess it is. That deduplication trick mentioned in the Soghoian blog post is unnerving.
posted by immlass at 4:38 PM on May 13, 2011

But if you believe any security statements juxtaposed to an illustration of a time-traveling DeLorean, you deserve what you get.

That's why I include cutesy illustrations with all my investment prospectuses, they allow you to lie to your customers.
posted by atrazine at 4:41 PM on May 13, 2011

If you're storing something sensitive -- locally or remotely -- it should be encrypted. By you, prior to storing it.

That solves this problem: instead of dropbox storing myconfessions.txt, it stores an encrypted version, maybe zlpbasrffvbaf.gkg, that only you can decrypt.
posted by orthogonality at 4:48 PM on May 13, 2011 [3 favorites]

I have to admit, I was a bit taken by surprise the first time I uploaded something which retained the same name as when I got it from where ever I got it from, and discovered that it appeared in my Dropbox account nearly instantly instead of having to wait through the tedious uploading process I usually have to endure. (Their upload throughput is horrible, I find.)

That was my first clue that they weren't actually always uploading what I wanted them to have, and instead were comparing hashs and such. They certainly never told me that in clear terms when I signed up for the service.

I've found that if I rename files and zip them, then they actually upload things rather than using someone else's copy. And, well, maybe that's what I should be doing anyway, given the kind of things I tend to drop in my Dropbox and why.

So, yeah. There's stuff going on they aren't really up-front with their customers about when you sign up. I still value their service, but it's not really the service they claim to be offering if they're hashing files and not actually taking your uploads all the time.
posted by hippybear at 4:49 PM on May 13, 2011

I'm guessing that Dropbox just wants to avoid the unpleasant scenario of a user losing their private key and being unable to recover their data from the server. The de-duplication is probably just a nice (for them!) side effect.
posted by jcreigh at 4:52 PM on May 13, 2011

So, yeah. There's stuff going on they aren't really up-front with their customers about when you sign up. I still value their service, but it's not really the service they claim to be offering if they're hashing files and not actually taking your uploads all the time.

Hmm. I would be pretty unfazed by this behavior, since I wouldn't expect a "could" storage service to encrypt the data in a secure way.

I'm curious: Why does it matter to you whether the upload actually happens? The same bits are available to you either way, barring an attack on the hash function that they're using. (Which, given how thoroughly, eg, MD5 has been broken, is a perfectly sane thing to be worried about.)

Or were you expecting that the service was actually secure from server-side breaches, and were disappointed to realize that they were storing your data in the clear?
posted by jcreigh at 4:58 PM on May 13, 2011

dont fucking spoil this Soghoian
posted by nathancaswell at 4:59 PM on May 13, 2011 [1 favorite]

Dropbox I have found to be a wonderful service . I can easily sync my iphone ipad and laptop stuff up .

That said - don't store anything in the cloud that you wouldn't care for someone else to read. That's just basic.

The only encryption that you can ever rely on is one that you do at your end. If you rely on someone else to encrypt then you might as well sign up for a Facebook account right now for all the security you are going to receive.
posted by Poet_Lariat at 5:02 PM on May 13, 2011

Why does it matter to you whether the upload actually happens?

The article explains the risk. It let's the Feds/whomever know that someone else has uploaded a specific file to Dropbox. They can then use that knowledge to get a court order or otherwise compel Dropbox to reveal who those users are. De-duplication is common with online backup services, it just sucks that Dropbox lied about their actual security and privacy practices.
posted by ChrisHartley at 5:13 PM on May 13, 2011

dont fucking spoil this Soghoian

Dropbox spoiled it. He's just pointing it out. I really dislike that they didn't say "my bad" on this especially when contrasted with the recent spin-free announcement by LastPass.
posted by BrotherCaine at 5:17 PM on May 13, 2011

Recently someone released a filesharing tool called Dropship that let you share files through dropbox by exchanging the hashes. If Dropbox thinks you have a file it'll let you access it, because of this they had to turn off de-duplication.

That said - don't store anything in the cloud that you wouldn't care for someone else to read. That's just basic.

In theory that's great, but in practice people will use Dropbox for things they shouldn't.
posted by atrazine at 5:18 PM on May 13, 2011 [2 favorites]

There is/was a kind of file sharing app that uses the duplication hack that drop box has been trying to stamp out called dropship
posted by Ad hominem at 5:19 PM on May 13, 2011 [3 favorites]

I don't find the arguments against Dropbox to be entirely conclusive. Perhaps there's something I'm just not understanding, and the update to Soghoian's post makes it seem as though the company is admitting to error. So perhaps someone who grasps this better than I can help me out.

The problem is that I can imagine a reasonably efficient way of producing the observed results without compromising security, or so I believe. The way to do it would be to hash files in blocks; let's say generate a 32 bit hash for every 64k block. This can be done on the client side, and the hashes are sent to the server, unencrypted. These are then compared to existing file hash lists for de-duplication. For file updates, you just see which hashes have changed.

The next step is that the server tells the client which blocks it needs. The client then encrypts these and sends them to the server.

So what am I missing?
posted by Edgewise at 5:27 PM on May 13, 2011

That said - don't store anything in the cloud that you wouldn't care for someone else to read. That's just basic.

Your sensitive data may very well be safer in the cloud. It's quite possible that a server in a cloud provider's data center is better maintained and protected than your typical corporate Windows desktop computer.

We don't encrypt all our important paper mail, faxes, or phone calls. We largely trust these forms of communication. I believe the same thing will happen with the cloud, once people get used to the idea.
posted by Triplanetary at 5:41 PM on May 13, 2011 [2 favorites]

Wuala has been more upfront about their deduplication system : Wuala encrypts all files using an AES128 key easily derived from the file's SHA1. You only need the encrypted file's SHA1 for downloading, but you'll need the unencrypted file's SHA1 for decryption. Wuala themselves should never know unencrypted file's SHA1. Dropbox might do exactly the same thing. Or maybe they just keep your keys like the article implies.

Yes, even encrypted de-duplication like this fails dissidents badly : Wuala cannot easily read the file themselves, but once they Swiss government possesses the file they may demand that Wuala reveal the first uploader. There is also a risk that AES128 interacts badly with their SH1 based passwords, but presumably AES256 and SHA256 mitigate this problem significantly.

That said, encrypted de-duplication is an infinitely better method for sharing the family photo album than posting it openly on facebook or flickr. Any real dissidents should know Swiss and American companies cannot be trusted too.

I'm mostly worried that the MafiAA will get court orders asking for all owners of particular encrypted SHA1s to sue them all for distribution.. nevermind they might never have even possessed the file themselves, much less shared it.

I'm nevertheless cautiously hopeful that such encrypted deduplication might make file sharing even more widespread than currently. If you modify your copy, then you & all friends who copy yours obtain a version the MafiAA will most likely never unearth. I'm also cautiously hopeful that Dropbox, Wuala, etc. all start adding social networking features that move people away from the privacy assrape known as facebook.

Of course, you should never trust closed source encryption applications like Dropbox and Wuala in the first place, but that was obvious to everyone here, right? Again, family photo album? Your golden, check. Your contract negotiations? Umm NO!!
posted by jeffburdges at 5:44 PM on May 13, 2011

The client then encrypts these and sends them to the server...
So what am I missing?

Suppose Alice uploads the block with hash 0xbeef, but she encrypts it first. So the server knows the unencrypted block's hash, but only has access to the encrypted block. Now Bob uploads the same thing: 0xbeef, encrypted with his own key. The server is de-duplicating, so it only stores one copy of the block, but needs to be able to serve it to Alice and Bob when requested. If the server store's Alice's block, it can't send it to Bob, because it's encrypted for Alice. And vice-versa if it only stores Bob's block. So either it stores both encrypted blocks (thus losing de-duplication), or it stores it in a way that it can be decrypted and encrypted with the right key prior to download --- which voids the purpose of encrypting it in the first place. You cannot have both de-duplication and encryption.


Recently someone released a filesharing tool called Dropship that let you share files through dropbox by exchanging the hashes. If Dropbox thinks you have a file it'll let you access it, because of this they had to turn off de-duplication.

There's a simple probabilistic way around this. To "upload" a file, the server asks the user to prove they have the file by computing the hashes of several randomly-chosen (by the server) ranges of bytes within the file. Now there's no unique hash that can be used to fool the server.
posted by qxntpqbbbqxl at 5:47 PM on May 13, 2011 [1 favorite]

We need a little dropbox & wuala de-de-duplicator that recognizes filetypes to safely insert garbage into the beginning of files. I vaguely though wuala handled the chunking up into blocks after the encryption phase, making file end modifications workable, but if dropbox or anyone encrypts after chunking then you must modify the file's beginning.
posted by jeffburdges at 5:49 PM on May 13, 2011

As I said qxntpqbbbqxl, you most definitely can have deduplication and encryption : You simply use the unencrypted file's SHA256 for it's AES256 encryption key. Your database records the SHA256s for both before and after encryption, but you only ever give the serve the encrypted file's SHA256 during retrieval.

Is there a cryptographic problem with doing this using only 128 bits for the SHA & AES like Wuala does? Who knows, but I'd bet any such issues become far far less severe if you use 256 bits.

Is this "military grade encryption"? Absolutely not, your eavesdropper Eve may easily discover whether you're storing any file that she's also seen.

Does it beat facebook for sharing your photos? Yes obviously! Is the risk that the MafiAA will know your mp3 & movie collection worth the "instant" downloads? I donno, maybe.
posted by jeffburdges at 6:00 PM on May 13, 2011

If your using dropbox's history features, you should maybe consider a full fledged version control system. :)
posted by jeffburdges at 6:06 PM on May 13, 2011

As I said qxntpqbbbqxl, you most definitely can have deduplication and encryption : You simply use the unencrypted file's SHA256 for it's AES256 encryption key. Your database records the SHA256s for both before and after encryption, but you only ever give the serve the encrypted file's SHA256 during retrieval.

If Alice and Bob both have file X, they'll both hash to the same SHA256, and therefore both be encrypted by the same key. Thus, Alice and Bob will have the exact same (encrypted) file uploaded.

If someone's looking to know if their file lurks somewhere on the cloud, they'll just have to hash it and encrypt it using that key. If it uploads quick, get out the warrants!

Yes, Joe Q. Public won't be able to decrypt an arbitrary file into its original. But, rainbow-tables style, given a hash of a file, you can see who owns it and thus, in a fashion, 'decrypt' a known file.
posted by soma lkzx at 6:08 PM on May 13, 2011

Frankly, if Dropbox touted this type of de-duplication as a feature: You are only charged for the non-duplicated bytes in your account and that you upload I would feel much better about the whole thing. Users would understand the de-duplication risks and would also be able to use the functionality the way that I am now: I don't have to shuffle around ISO's (or copy and paste the links) anymore worrying that I'm going to saturate my little DSL line. I can bundle all of my development tools, most of which are de-duplicated already by others and when I upgrade them, my bandwidth usage is minimal and my remote hosts get the files almost instantly.

That said, I never have anything even remotely resembling sensitive on Dropbox.
posted by Freen at 6:10 PM on May 13, 2011 [1 favorite]

There aren't any successful rainbow tables attacks on SHA256 yet, right? I'm extremely doubtful that rainbow tables help much against even Wuala's weak SHA128 version, given we're talking about huge files, not little passwords.

If we're talking about poor security today, I've noticed that abebooks.com's "retrieve password" button actually emails you your old password, rather than sending a rest email. Ergo, you should not give them any password about which you might care. You should not let them store your credit card information for speedier purchases either. And you should only use a real credit card there, not a debit card, but that's universal. I'd still choose abebooks over amazon of course, just reenter your credit card every time you order.
posted by jeffburdges at 6:19 PM on May 13, 2011

Of course, you should never trust closed source encryption applications like Dropbox and Wuala in the first place, but that was obvious to everyone here, right? Again, family photo album? Your golden, check. Your contract negotiations? Umm NO!!

This. Let's see. A person downloads and executes a closed source program that has access to userland files. Then uploads the files to server that allows web access. They are then surprised that dropbox can give access to the files under warrant. Add to this that the windows version requires admin levels to install and you are pretty much closing the door well after the horses have left.
posted by zabuni at 6:30 PM on May 13, 2011

You don't need a rainbow table, only a list of hashes for the files that you're looking for, e.g. RIAA is gonna find all your Miley Cyrus mp3's.
posted by soma lkzx at 6:31 PM on May 13, 2011

There is another serious "threat model" for ordinary people that doesn't require the MafiAA's newest Dr. Evil suing everyone who copied some film : Imagine you and Mr. X both like similar obscure music. Imagine the FBI dislikes suspiciously named people like Mr. X. Our helpful friends at Dropbox now happily build the FBI a list of people sharing obscure files with Mr. X. Enjoy your water boarding!
posted by jeffburdges at 6:45 PM on May 13, 2011 [1 favorite]

jeffburdges, I believe they are right. The point is, what copy of the data is the server carrying? If it's the one that's uniquely decrytable by Alice, then it will be meaningless if served to Bob, and visa versa.
posted by Edgewise at 7:27 PM on May 13, 2011

jeffburges, You don't need a "successful rainbow table attack" on a hash. A rainbow table is just a table that has the computed hashes for the beginning of passwords precomputed for you so that you get to skip a bunch of work when trying to brute force a password. Rainbow tables can be built for any hash function.

You're right though about tables not being useful though. They only help you for short strings.
posted by joegester at 7:56 PM on May 13, 2011

Would be interesting to just generate random "hashes" and see how many match up.
posted by Ad hominem at 8:10 PM on May 13, 2011

Would be interesting to just generate random "hashes" and see how many match up.

No, it probably wouldn't, assuming the hash function's output space isn't tiny (which, if it were, would make for a bad deduplication tool) -- the number of unique files on dropbox is many orders of magnitude smaller than the hash space, and unless the hash function is deeply flawed, the odds of picking a random hash to match an existing file is vanishingly small.
posted by axiom at 8:30 PM on May 13, 2011

assuming the hash function's output space isn't tiny (which, if it were, would make for a bad deduplication tool)

Ok, according to the dropship source it is SHA256, and the file is split into 4MB blocks. So you would need the list of hashes for all the blocks to make it worthwhile. So it would be good for filesharing, but not trolling for random files I guess.
posted by Ad hominem at 8:51 PM on May 13, 2011

All I use Dropbox for is easily moving files between my desktop and netbook and for storing personally encrypted mail archives downloaded from my GMail account (that relatively recent reset thing made me paranoid).
posted by Samizdata at 9:36 PM on May 13, 2011

Huh, this duplication must explain why the PDFs shared around my gaming group got corrupted on my end. I can only assume that my friends are uploading copies of their PDFs with content but they come out, on my end, with corrupted layers or text or missing content altogether.
posted by Severian at 5:57 AM on May 14, 2011

wait you're telling me a cloud is insecure??? clouds are synonymous with impenetrable in my lexicon!!
posted by Eideteker at 8:06 AM on May 14, 2011 [1 favorite]

The folks who get mocked for not taking companies like Dropbox at their word are the ones with the smiles on their faces now.
posted by vanar sena at 8:29 AM on May 14, 2011

jeffburdges: If your using dropbox's history features, you should maybe consider a full fledged version control system. :)

Like SparkleShare.
posted by vanar sena at 8:33 AM on May 14, 2011 [2 favorites]

assuming the hash function's output space isn't tiny

Actually, you need a really large hash space to reduce collisions to an acceptable level. A 64-bit hash space would definitely be insufficient. A 128-bit or larger hash space is probably fine. See: birthday attack.
posted by ryanrs at 9:43 AM on May 14, 2011

I found DropBox's behaviour quite shady personally. To me, it seems that they went out of their way to give the end user the impression that their files were completely secure, when in fact they could be read by any DropBox employee with access to the global encryption key ("We use AES!"), and not only that, not all the DropBox clients bothered to use a secure connection for the upload.

If they'd been upfront about it in the first place, I wouldn't have had much of a problem: from an end-user point of view it's still a great, well-engineered service. But it isn't secure, and they had no right to give end-users the impression that it was. Even relatively sophisticated users were taken in: only if you assumed bad behaviour on the companies part in the first place would you suspect that statements like

"[a]ll files stored on Dropbox servers are encrypted (AES-256) and are inaccessible without your account password."

really meant

"We encrypt everything with a single key which anyone who has access to (law enforcement, DropBox employees, anyone who manages to hack into the DropBox network, etc, etc) can use to read all your files."

posted by pharm at 1:47 PM on May 14, 2011

I've never used dropbox but you can get an S3 account and upload your files directly to amazon if you want. Why pay an intermediary? S3 is easy to use if you get a client and I've been using it since before dropbox existed. Yeah, you have to upload your own files but it's not that big of a deal. It's true they're not encrypted, but you can encrypt yourself if you want. And beyond that it makes sharing easy. You can share with an unlimited number of people, you just have to pay Amazons (very cheap) bandwidth costs.

I'm guessing that Dropbox just wants to avoid the unpleasant scenario of a user losing their private key and being unable to recover their data from the server. The de-duplication is probably just a nice (for them!) side effect.

They should make it clear that they are retaining a copy of your key and give you the option to discard it.

Recently someone released a filesharing tool called Dropship that let you share files through dropbox by exchanging the hashes. If Dropbox thinks you have a file it'll let you access it, because of this they had to turn off de-duplication.

Yeah, and dropbox actually threatened to sue the author and claimed a DMCA request had been sent on it.

We need a little dropbox & wuala de-de-duplicator that recognizes filetypes to safely insert garbage into the beginning of files. I vaguely though wuala handled the chunking up into blocks after the encryption phase, making file end modifications workable, but if dropbox or anyone encrypts after chunking then you must modify the file's beginning.

Or just encrypt the file before posting. Simple solution: keep everything important on a truecrypt volume, then upload the changes to it.
posted by delmoi at 4:23 PM on May 14, 2011

Dropbox provides 2GB of storage space to its customers for free. Consumers can purchase additional storage space, by signing up for one of two “Pro”service plans, offering 50GB for $9.99/month or $99.00/year, and 100GB for $19.99/month or $199.00/year.

By the way. Amazon charges 10¢ per gigabyte with no minimum. Storing two gigabytes on S3 costs 20 cents a month And storing 50 gigabytes would cost $5 a month. So drop box is providing a wrapper to S3 for twice the price.

Seriously, it's a crappy deal people. I encourage everyone to sign up for S3 and try out some of the clients. I'm not trying to shill for Amazon but their web services are awesome.
posted by delmoi at 4:26 PM on May 14, 2011 [2 favorites]

It is rather strange, but still quite fun to see my research and activism highlighted on the blue.

Thanks

Also, those of you seeking Dropbox alternatives - I pay for Spideroak. I have heard great things about Wuala, and if I could deal with the lack of GUI, I would use Tarsnap.
posted by genome4hire at 10:15 PM on May 14, 2011 [3 favorites]

delmoi - could you provide some more info on how you're using S3 for your data? Backups/sync/sharing? What tools and how much data?

I'm interested in using it for encrypted duplicity backups (around 200GB data all up, incrementally updated), so I'd love to hear how other people are using it.
posted by vanar sena at 6:35 AM on May 15, 2011

There are duplicity instructions for S3 here.
posted by jeffburdges at 2:53 PM on May 15, 2011