A System to Transparently Compress Hundreds of Petabytes of Image

https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/horn

by todsacerdoti • 6 years ago

69 18 6 years ago

jpdaigle• 6 years ago

Fascinating, but wasn’t... Dropbox supposed to have client-side encryption? If the backend is recompressing JPG images, that means it has full access to the file’s plaintext. That’s a bit uncomfortable.

lsiebert• 6 years ago

Presumably shared images?

twright0• 6 years ago

Dropbox does not have client-side encryption; its servers have full access to the content of synced files. You can verify this yourself by uploading a common image filetype like jpeg and seeing that the website shows you compressed file thumbnails.

(I'm a former Dropbox employee who worked on related things, though not specifically the linked paper)

pmiller2• 6 years ago

Yep, that makes sense. IIRC, they also have some sort of global deduplication system, don't they? That also implies they have access to your unencrypted files.

Edit: That's what I get for commenting before reading the paper. They literally mention deduplication in there.

londons_explore• 6 years ago

Mega does client side e2e encryption (or at least they claim to), yet their web interface still shows thumbnails...

I'd guess they implement it by having the client upload an encrypted version of both the original image and a thumbnail, and then decrypting the thumbnails on the client at display time. Javascript has reasonably performant crypto primitives now, so it's very doable to download data from a server, decrypt it with a client side key, and display it.

If mega can do this, it's a shame other services don't.

foobiekr• 6 years ago

Dropbox has "client-side" weasel encryption: they use TLS. It is obvious that they do not encrypt the files in any way that would prevent access to Dropbox itself, and probably not to any Dropbox employee.

harikb• 6 years ago

Dropbox never claimed end-to-end client side encryption. That said, neither do any of the popular cloud drive options - for example Google drive or One drive.

End to end client side encryption is a tough problem from usability. Most people don’t like being told they can’t recover their “cloud backup” if they lose their laptop or phone or forget their password.

I think even iCloud only does end-to-end for specific sensitive data.

There are special providers like SpiderOak who do this, but it is non trivial to use safely.

justinc8687• 6 years ago

I've always found Cryptomator or encfs to be great solutions for this.

Tommy_Flowers• 6 years ago

The only safe assumptions about encryption of your data in the cloud is that 1) its not encrypted, or 2) it is encrypted but at some unknown time in the future there will be a security compromising bug, and the data will be in the wild. But you won't know its in the wild, because the service provider doesn't know about the bug, or it isn't in their interest to tell you.

toast0• 6 years ago

It seems you're right that Dropbox has the plaintext server side from elsewhere in the thread.

It would also have been possible that they did the recompession client side though, and send enough metadata to calculate metrics. I had thought that's what they were doing, to save bandwidth/reduce upload times as well as save space on the servers.

WalterGR• 6 years ago

“As of February 2017, it had compressed more than 203 PiB of user JPEG files, saving more than 46 PiB.”

Not too shabby.

CorrectHorseBat• 6 years ago

I've tried lepton and packJPG (on which lepton is based) for my personal collection and found packJPG to be faster, give better compression and a lot simpler to compile and install. Only problem was that it did occasionally fail to compress an image.

zcw100• 6 years ago

It compresses them one image at a time. You'd think that if you had millions of images there would be opportunities to index and cluster similar images to obtain further compression.

willis936• 6 years ago

Then the images are no longer individual. That would be fine for read-only, but I would guess a fair amount of the data is hot.

A System to Transparently Compress Hundreds of Petabytes of Image

Related Stories