DigitalOcean leaks customer data between VMs

by sneak • 13 years ago

395 200 13 years ago

yeukhon5• 13 years ago

Hmm have you done this with private disclosure? If not, please do that next time. It is unethical to disclose security-related issue of some PaaS publicly without first going through the responsible party first. Even if a security issue you want to report is actually something only the owner can see... I don't know what this do because I don't really use DO, but just judging the way it is reported, it looks like you haven't...

nknighthb3• 13 years ago

Even if one bought the ridiculous notion that private disclosure of vulnerabilities is a good thing, why would you think it would be necessary to privately "disclose" to anyone what Digital Ocean already publicly documents in their API?

yeukhon5• 13 years ago

[edit]: okay I misunderstood the issue here.

Some day I would forget to close my home door with a bang. No home invasion yet because nobody notice that the door wasn't fully closed. Now I am telling people I will forget to close my door, and I will get an invasion.

Some people's repo have password committed. It only takes someone to google that to find out. If someone posts that on HN, yeah, it's on the web anyway, it only takes one person to make my password known instantly.

sneak12• 13 years ago

http://www.wired.com/wiredenterprise/2013/04/digitalocean/

https://www.digitalocean.com/blog_posts/resolved-lvm-data-is...

This sort of terrible behavior was by design and known. They got written about and claimed to have fixed it, but didn't.

icelancer4• 13 years ago

It is unethical to disclose security-related issue of some PaaS publicly without first going through the responsible party first.

Says who? You?

DO has a history of not responding to issues UNTIL they are publicly disclosed. And in any case, your iron-clad "argument" is a matter of opinion and nothing else. Many people prefer full disclosure.

yeukhon5• 13 years ago

Says who? You?
And in any case, your iron-clad "argument" is a matter of opinion and nothing else.

First, let me repeat: I did get the story mix up and the ethical approach I am referring to doesn't quite apply in the current story.

DO has a history of not responding to issues UNTIL they are publicly disclosed.

Does not matter what happen between DO and whitehats. If an OS command injection is discovered, even if DO has a history of not responding to security issues, the moment the vulnerability is discovered, a whitehat should alert DO privately first. If they ignore it again, then of course you can let the public know and let your zero-day exploit begin. Regarding this, public disclosure before private disclosure is unethical.

mikeash14• 13 years ago

There is someone in this thread claiming to speak for DO, and he says that it's fully documented:

https://news.ycombinator.com/item?id=6983520

There's no point in private disclosure for something that the company themselves documents.

Of course, he also whines about responsible disclosure. The mutual exclusivity of these two things does not seem to have occurred to him....

nbpoole• 13 years ago

Interesting: this sounds like a recurrence of the same issue which was described a number of months back:

https://www.digitalocean.com/blog_posts/resolved-lvm-data-is...

At the time, the blog post claimed that the issue was resolved and that data was now being wiped by default. I wonder why that would have changed.

sneak12• 13 years ago

I'm the one who reported it. I was able to recover someone else's web logs from 29 December 2013 on a VM an hour old.

kylek• 13 years ago

have you gone to DO directly with this first?

sneak12• 13 years ago

According to them, it's either a bug they already fixed in April, or leaking data today by design?

https://www.digitalocean.com/blog_posts/resolved-lvm-data-is...

https://twitter.com/jedgar/status/417515181418479616

coolj9• 13 years ago

I don't really see how this is leaking data. I can use the dropbox API to make a bunch of files with private data shared with the world. But dropbox isn't "leaking" my data, I'm using the API in such a way that makes my data accessible. Not an exact analogy, and I would agree that the option should be on by default so people who know what they're doing can opt-out and everyone else gets a safer default, but this isn't a "data leak."

jlawer9• 13 years ago

The bug in April was about the option not working.

The feature now works if you use it. By design if you don't use it you don't wipe the disk (saving you money).

0x0• 13 years ago

It would be interesting to see a survey of the various cloud VM services; if any of them will return non-zero (data) blocks for uninitialized storage.

Sounds like a major risk if SSH, SSL, passwords etc can leak this easily.

coolj9• 13 years ago

Also, even if they zero the primary block device, what about swap if it's provided as a separate block device?

jlawer9• 13 years ago

DO provide a single block device (so swap is on the same block device), that gets zeroed if you set the scrub data option.

jlawer9• 13 years ago

Talk about a link bait title. Its a bit hard to call it a leak,Its a configuration option that is well presented in the web UI. It is optional as it adds ~ 10 minutes of billing to the small 512mb VMs and as such it is optional if you do it.

If your using an overlay or API on top of a cloud or service, its the overlay's responsibility to ensure a consistency with your expectations. The API is consistent with the UI.

While other cloud providers accept the time that this takes as non-billable, DO don't. By getting higher utilization is how they are able to offer their prices and still have some modicum of service.

sneak12• 13 years ago

Let's be clear about what this is: they are charging their customers after their customers have deactivated a service (destroyed a VM) to not create the situation wherein they give that data to another customer later.

What sort of mental gymnastics are required to make that a reasonable choice?

jlawer9• 13 years ago

If the hardware is spending its cycles on your workload, then it definitely makes it a reasonable choice. Its not like they can sell those cycles to someone else until your job is done.

Besides we are not talking about a high margin business here. $5 vms when most providers are charging 4x times that. Its not unreasonable to expect that your going to have to pay for extras. Similar to a budget airline, you get what you pay for. You want a service that includes that cost in your other fees... then use AWS, rackspace or one of the 1000s of others.

Dylan168072• 13 years ago

they are charging their customers after their customers have deactivated a service (destroyed a VM)

They are charging their customers for the number of minutes it takes to safely destroy the VM. This is not a charge for something coming 'after'. It's fundamentally a charge for their actual server use, not a bonus fee.

What sort of mental gymnastics are required to make that a reasonable choice?

They aren't charging for security, they are giving you the option to buy less server time if you don't need security, or handle it yourself by wiping only the sensitive files. There are no mental gymnastics here.

wmf• 13 years ago

I think the issue of defaults is orthogonal to the issue of billing. For example, they could make scrubbing on by default (or even mandatory) and still bill customers for it. Of course they would have to disclose this.

jlawer9• 13 years ago

I completely agree for the UI. For the API I think it is completely fine to have it not as default (In fact I would argue having a boolean controlling an optional action default to false for an API is the most correct action).

AaronFriel6• 13 years ago

Please see my post debunking everything about DigitalOcean's need to spend even one minute scrubbing user data. The fact that they are using SSDs makes it extraordinarily cheap for them to scrub customer data using the TRIM command (on Linux: by sending the BLKDISCARD IO command). With that, they can logically zero hundreds of gigabytes of customer data within seconds.

jlawer9• 13 years ago

I agree that there are ways to implement this. And perhaps they should (mind you from experience not all SSDs do this properly).

I simple can't blame them for delivering what they say they are going to give me, even if they could have built their infrastructure better.

amalag3• 13 years ago

Yes I can't believe people are flipping out over a Fog issue. If you care about the privacy of your data, it is your responsibility to make sure it gets erased. If Fog wants to put that as a default, sure do it.

amalag3• 13 years ago

Actually thinking about this more, I am starting to understand the outrage. I think the commenters are right, it should be secure by default.

mikeash14• 13 years ago

Fail safe. Fail safe. Fail safe.

It's fine to make this optional. But it needs to have large flashing red warning lights all around it and it needs to be off by default.

PhantomGremlin• 13 years ago

Anyone who's tempted to use "the cloud" for anything sensitive should first be forced to write out, at least 5000 times, in longhand:

"The cloud. Somebody else's computer".

I think cloud computing is great for the right applications, as long as people understand the risks.

But there will always be problems like this. Always. This is part of the hidden cost of "simple cloud hosting".

orblivion• 13 years ago

What about complicated cloud hosting? Rackspace is still "Somebody else's computer".

xSwag• 13 years ago

TL;DR: In the DigitalOcean web panel you can check the "scrub data" checkbox when destroying a VM. When using the API this option is not ticked. This can lead to other customers being able to retrieve your data.

The author thinks that this is a security issue because this option should be enabled by default. However, (I assume) it's not in Digital Oceans interest to do full disk scrub because it reduces the lifespan of their SSD.

If a user forgets to log out of Facebook on a public computer, is it Facebook's responsibility? Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault?

sneak12• 13 years ago

I think it's less in Digital Ocean's interest to silently give customer data to a subsequent user of that block device?

jlawer9• 13 years ago

Why?

If customers don't care enough about the data why should DO?

DO is a budget provider. Their main hook was cheap SSDs, and is still cheap costs. They billable time it takes to wipe a VM adds to the cost. If you want the data zeroed... then pay for it, otherwise it must not be important enough.

Stealth-2• 13 years ago

The term "disc scrub" makes little to no sense in terms of SSDs, which makes this issue even more complicated. People who work in the disc recovery field have been dealing with this since SSDs became popular.

An SSD is limited by its number of writes. To compensate for this, the SSD has very complicated on board logic that abstracts the actual SSD away from what it tells the OS system. This allows it to do certain tricks to save writes. However, when you are "scrubbing" an SSD, internally the SSD might be writing somewhere else entirely. Scrubbing is not considered an effective way of wiping SSDs, from what I believe.

eurleif• 13 years ago

Ok, so the data isn't necessarily destroyed immediately after a scrub. But how does that play out at the level of VM users? Is there a normal usage scenario where the portion of the SSD containing my deleted data is made readable to someone else's VM, or will it be inaccessible to normal VM users until it's overwritten?

mikeash14• 13 years ago

There is a vast difference between writing out zeroes to the SSD but still having some of the original data potentially persist on the SSD but unreachable without special techniques, and not zeroing out the SSD and giving the device to a new VM and letting it trivially access everything that was previously there.

If I can provision a new VM and cat /dev/vda and see data from the VM that previously occupied that spot, then you are doing it horribly, horribly, horribly wrong.

That zeroing out the data leaves open a different and vastly more difficult attack path doesn't make that any less true.

nwh3• 13 years ago

If it was a lifespan issue they could just encrypt the partitions (minimal overhead with AES) and then destroy the key with the droplet.

Due to the way that SSDs wear level, even if every block returns 0, you don't know if that's the same block or if the real data is squirrelled away on out-of-life blocks. From what I understand the SSD has a lot more capacity than advertised and moves the blocks forward as sections of the chip wear out. Assuming you could reprogram the controller (or use a different one), you could go back and read the old blocks and recover data in the clear.

pilsetnieks• 13 years ago

Not several times more capacity, it's generally about 7% (that's why you often see 120 GB or 240 GB drives that are in reality 128 GB or 256 GB.)

spullara• 13 years ago

You're right. Encryption is the right answer. What does Amazon do with their SSDs? Anyone know?

rythie3• 13 years ago

The problem with encryption at that level is that trim no longer works as there are no longer any zeroed blocks - so writes are slower. Also there will be more write applification as every write will cause a block to be erased.

sneak12• 13 years ago

Similarly, if a user does not correctly delete data on a budget host, is it the hosts fault?

What does the verb "destroy" mean?

Screenshot taken today (30DEC13) from a fresh blank Digital Ocean server I just provisioned:

http://i.imgur.com/fJOxRN9.png

EdwardDiego• 13 years ago

What are we looking at in your screenshot? I've been Where's-Waldoing it for the word 'destroy' but can't find it.

icmp• 13 years ago

nice post - also #fmc says hi ;)

rgbrenner• 13 years ago

Wait a second. Why does this option exist at all? Are you saying there are people who DO NOT WANT their data destroyed when they destroy a VM? Who are these people?

jd007• 13 years ago

It's probably there for DO's sake. SSDs have limited rewrite cycles, and each secure wipe will definitely shorten the lifespan of the drive much more than a simple quick wipe. This translates into cost savings.

userbinator• 13 years ago

It's much faster and there are certainly people who don't care because the data is worthless to anyone but them.

iagooar• 13 years ago

I use DO for bootstrapping and trying out new ideas, programs & co. There is no information I can think about I would wish to get wiped off of the SSD.

dspillett3• 13 years ago

If you care about your data then the erase will not protect you enough anyway, if you think it does then you've probably misunderstood the underlying architecture a little.

The data for your virtual machine's virtual disk(s) will not always be in the same place, it may get moved around the storage volume or moved between storage volumes and this will be completely transparent to you. It may also get stored on backup volumes too. When you "secure wipe" the disk as you kill the VM all you are doing it wiping the data that is in the current location not any latent copies that may be sat elsewhere (as backups or "ghost" data sat in currently unallocated bits of physical media).

The only way to grantee your data is stored securely and is gone securely when you want it gone, is to use full-volume encryption from the start and make sure that the key's are never stored at the provider side (this does mean that you need a mechanism by which you can provide the key(s) to the VM whenever it reboots for some reason). That way there is no need to wipe the current store or worry about ghost data elsewhere: just destroy all copies of the keys for those volumes. Of course there is still some risk as the keys need to be in RAM somewhere so the encrypted volumes can be accessed at all, but once you get to that level of concern you can only be sure you are secure by having your own physical kit.

Of course not all current hardware sharing solutions can support full volume encryption as you don't really have a proper volume(s) that you can encrypt and put filesystems in, just a part of a larger filesystem pretending to be one (and you can't mount things so you can't use a file-based volume instead)...

tl;dr: wiping the VM's disks does NOT protect you from this sort of thing, using properly implemented full volume encryption will (as much as is possible in a shared virtualisation host, iff your hosts's solution can support FVE in the guest VMs/containers at all).

tombrossman• 13 years ago

I'm surprised I didn't see anyone raise the issue of Data Protection given DO's European presence. This comment below [1] appears to confirm that customer data is mishandled in violation of EU Data Protection law. When you ask 'is it the hosts fault?' I think the answer is a most definite YES.

Surely some lawyer out there took a look at this, so maybe I have missed something, but this looks like a big problem to me.

From my own experience using DO I can say I'm a happy customer and I plan to keep using them. I tick that box when I've used anything remotely sensitive in a VM when destroying it, and leave it empty other times (like when I've made a mess experimenting with something and want to quickly trash & re-create a droplet).

[1] https://news.ycombinator.com/item?id=6983260

AlexHamilton• 13 years ago

DO is a data processor under EU data protection law, while the customer would be the data controller. EU data protection law currently (it will change with the new regulations) only imposes legal duties on the data controller. As such, it is the customer's legal problem if it (or its data processor) has failed to handle personal data correctly.

nhaehnle• 13 years ago

They don't have to do a full disk scrub on the physical SSD to fix this security problem. All they have to do is what every sane sparse disk image implementation in the history of mankind has done: catch reads from unwritten blocks in software and return a block of all zeros.

drzaiusapelord• 13 years ago

The pro-DO bias of HN is showing again. If this was Amazaon or Linode, there would be endless hand-wringing and calls for lawsuits. When its DO doing something incredibly stupid and short-sighted, its 'blame the customer.' This is penny pinching at the cost of security, which is something we often complain about when it comes to other companies. Why does DO get a free pass?

"Oh, there's an option for that" to stop default dangerous behavior should not be excusable. This is further proof on why DO isn't enterprise ready and still a toy for barely stable dotcoms.

Thank god for EU privacy laws. If these guys have any EU presence, they'll be forced to clean up their act via regulation. Clearly, the invisible hand of the market and "there's a option for that" is the fail its always been in regards to security.

warrenm2• 13 years ago

They have a datacenter in Amsterdam

crohrc2• 13 years ago

Glad somebody pointed this out. It's also hilarious to me that we were all screaming for heads with the GD (mt) acquisition a few months back, and this is just a "pat on the back" type of thing.

Their growth and offerings are great, but they were going to run into problems sooner or later. As we've seen with many other hosts in the past, this is probably just the beginning.

ahallock• 13 years ago

I was with you until you started talking about the "invisible hand of the free market" failing and how instead we should be using State force to solve these problems. Isn't your comment a free market action, raising awareness about this problem? Won't this thread cause DO to take action? Say it doesn't. There are plenty of other hosting companies to switch to. And also there are times when I might be willing to trade security for convenience, price, practicality, etc. As a consumer, I want this choice. I think the free market is working fine. You can't bully your way to good security.

nwh3• 13 years ago

It's almost public knowledge as there's an option in the GUI called "scrub data", the absence of a tick there implies that it's not going to be erased before the VMs partition is reassigned. I had a chat to the support months back about "erase data" not being in the API at the time, and the solution I came to was just to ditch their API and go back to scraping their forms for the option.

That said, this would probably go down better for the company and the community if you tried a private disclosure rather than posting about it on Github.

https://www.digitalocean.com/security

1945• 13 years ago

I've been a DO customer for about a year and some, this is news to me. So no, I wouldn't say it's public knowledge.

That said, I haven't really destroyed many of my VMs.

nwh3• 13 years ago

http://i.imgur.com/Aliyewf.png

I suppose when I made that comment I was expecting everybody to be like me; eager to flip switches to see what they do.

threeseed• 13 years ago

If the "securely scrub" parameter has ANY kind of performance impact which I suspect it does then isn't it a good thing that DO gave their users the choice ? There are plenty of use cases where I would trade privacy/security for performance.

Anyway disingenuous title to say the least.

mikeash14• 13 years ago

It is good to give users the choice. It is bad to default to the dangerous choice. It is even worse to not describe the bad consequences of the default anywhere.

Nowhere in the DO UI for destroying a droplet does it indicate that they will leak all of your data to the next customer if you don't check the box.

sillysaurus23• 13 years ago

There is a simple solution to this: don't trust providers to do what they say they'll do with your data. You should scrub any drive that's ever contained sensitive info before you throw it away, and terminating a VM instance is precisely equivalent to handing the VM's harddrive to your provider.

It's pretty easy nowadays to scrub a drive. Writing zeroes would suffice.

Personally, I'd worry more about what data is being leaked when your VM is paged to disk on your provider's servers. Parts of each of your VMs will probably reside in the pagefile at some point, so therefore writing zeroes won't save you if the provider has bad disposal practices (like not scrubbing before disposal). So it seems impossible not to have to trust a cloud computing provider whatsoever; some basic trust seems to be a requirement.

But that minimum level of trust should be the extent to which you trust them. Not scrubbing your drive before handing it over is placing faith where faith doesn't belong.

sdfjkl• 13 years ago

You may not always have the chance to scrub it yourself, for example when your VM has a hardware issue.

cordite• 13 years ago

Just a random idea from an ignorant individual..

What if DO actually encrypted the SSD space with a key that they only have, and a new key is created for each droplet?

Then any droplets that are created later in a deleted space will just see effectively random data, no?

thirsteh2• 13 years ago

You can put the key on the disk itself too--you just need to zero out the space containing the key when you're done. That's how phones with encryption and remote wipe features usually do it.

In fact, a lot of SSDs, e.g. Samsung's, already work this way, transparently AES-encrypting data before it is written. (With AES in hardware, the overhead is negligible, even for high-performance drives.) The "Encryption" feature they advertise actually just lets you set your own key for the encryption key container--it happens either way.

serf• 13 years ago

Intel FDE, as far as I know, also works this way.

The key is protected. In the event that the drive must be re-provisioned (like in the case of a lost password), the decryption key is simply overwritten by the new key, rendering the original data unreadable.

VLM• 13 years ago

Maybe they "do stuff" with the data, like they've got an internal process to give historical access to the NSA or any random .gov upon request. No need to assume that massive pushback against obvious security improvements could only be ignorance or incompetence. Adding security could throw a wrench in the works if there is a business process or procedure that directly relies on insecurity. If they have a national security letter requiring them to do this, they won't be able to talk about the topic intelligently in public.

Or TLDR your idea would be great if they wanted a secure product, but they may legally be prevented from providing a secure product or even talking about the topic.

sneak12• 13 years ago

Oh, hey guys, they've responded. It's no big deal, they just disabled the security because because _users were complaining_.

Turns out it "add[s] a very large time to delete events" when you actually delete things when a user makes an api call to DESTROY. Who knew?

http://i.imgur.com/MFW8ng6.png

jlawer9• 13 years ago

Damn straight.

If I have something important on a VM it is purged before issuing a destroy. I will overwrite the block device myself so I know it is empty.

Why should I have to pay for DO to do the same again just because someone can't be bothered to read the manual.

josegonzalez• 13 years ago

Well that's stupid. They should also take care of the case where I delete a vm. What do I care that it takes a long time for them to delete the data? That shouldn't cost me any money, as I have freed the resources from my account.

AaronFriel6• 13 years ago

As I replied to you elsewhere, I feel your comment here may lead people to believe that it would cost DigitalOcean money to correct this behavior.

This is wrong, it is costing DigitalOcean money not to fix it - in terms of SSD lifetime, not TRIMing those blocks increases fragmentation of the internal physical layout of the pages of flash memory. The behavior of DigitalOcean's virtual machines has surprisingly managed to achieved the worst possible outcome. Their hardware is being misused and their customer's data is being mishandled.

switch0072• 13 years ago

I got the same response from them via Twitter. I don't think it's acceptable.

I worked for a hosting company that made data destruction their problem (i.e. we wiped the disks after the instance was terminated) because we didn't want new customers seeing non-zeroed disks and thinking we do not care about security.

AaronFriel6• 13 years ago

This is a huge problem and there seems to be a good deal of misinformation about this issue that has confused things. I'm going to debunk two things: first, that DigitalOcean is not violating user expectations (they are), and second, that doing this correctly is difficult (it isn't). The tl;dr is that if DigitalOcean is doing this, they are not using their hardware correctly.

First, it's not uncommon for virtual disk formats to be logically zeroed even when they are physically not. For example, when you create a sparse virtual disk and it appears to be XGB all zeroed and ready to use. Of course, it's not. And this doesn't just apply to virtual disks, such techniques are also used by operating systems when freeing pages of memory - when a page of memory is no longer being used, why zero it right away? Delaying activities until necessary is common and typically built in. Linux does this, Windows does it [http://stackoverflow.com/questions/18385556/does-windows-cle...], and even SSDs do it under the hood. For virtual hard disk technology, Hyper-V VHDs do it, VMWare VMDKs do it, sparse KVM disk image files do it. Zeroed data is the default, the expectation for most platforms. Protected, virtual memory based operating systems will never serve your process data from other processes even if they wait until the last possible moment. AWS will never serve you other customer's data, Azure won't, and none of the major hypervisors will default to it. The exception to this is when a whole disk or logical device is assigned to a VM, in which case it's usually used verbatim.

This brings me to the second issue. Because using a logical device may be what DigitalOcean is doing, it's been asked if it's hard for them to fix it. To answer that in a word: No. In a slightly longer word: BLKDISCARD. Or for Windows and Mac OS X users, TRIM. It takes seconds to execute TRIM commands on hundreds of gigabytes of data because, at a low level, the operating system is telling the SSD "everything between LBA X and LBA X+Y is garbage." Trimming even an SSD with a heavily fragmented filesystem takes only a matter of seconds because the commands to send to the firmware of the SSD are very simple, very low bandwidth. The SSD firmware then marks those pages as "free" and will typically defer zeroing them until use. Not only should DigitalOcean be doing this to protect customer data, but they should be doing it to ensure the longevity of their SSDs. Zeroing an SSD is a costly behavior that, if not detected by the firmware, will harm the longevity of the SSD by dirtying its internal pages and its page cache. Not to mention the performance impact for any other VMs that could be resident on the same hardware as the host has to send 10s of gigabytes of zeroes to the physical device.

Not only is DigitalOcean sacrificing the safety of user's data, but they're harming the longevity of their SSDs by failing to properly run TRIM commands to clean up after their users. It hurts their reputation to have blog posts like this go up, and it hurts their bottom line when they misuse their hardware.

Edit: As RWG points out, not all SSDs will read zeroes after a TRIM command, so other techniques may be necessary to ensure the safety of customer data.

sillysaurus23• 13 years ago

For example, when you create a sparse virtual disk and it appears to be XGB all zeroed and ready to use. Of course, it's not.

From within the VM, all the VM will see is zeroes. It sounds like DO is giving VM instances direct access to the underlying SSD or something like that. In fact, I'm having a hard time figuring out precisely how this is occurring. Whenever you create a new VM, how can the VM possibly be reading data from the host's harddrive? Isn't that the definition of a security problem, since VMs are expected to be isolated?

I hope someone will explain the underlying technical details more deeply, because this is very interesting.

AaronFriel6• 13 years ago

Please read to the end of my comment - it appears what DigitalOcean is doing is giving the VM access to a logical device that is preallocated. Perhaps carved out of LVM or MD or some other logical disk. KVM's default behavior when using these sorts of devices is to present to the VM whatever data already existed at the lower level.

rwg2• 13 years ago

In your second paragraph, you're conflating two different things. File-based disk images don't leak data when they're deleted because the filesystems those images live on ensure that (non-privileged) users can't get at data from deleted files. Sparse images can be smaller than the data they contain because...well, they're sparse images. They're files with holes in them, and the filesystem automagically turns those holes into zeroes on read.

Now, about Trim... Trim is only an advisory command. You tell the disk, "I'm not using these LBAs anymore, so feel free to do whatever with them." The disk has the option to completely ignore your Trim command, and even if it does mark those LBAs as unused in whatever LBA->NAND mapping table it uses internally, the disk can also continue returning the old data on reads of those LBAs if it wants to. There are disks that make the guarantee that Trim'd LBAs will always read back zeroes until written again (an ATA feature called Read Zero After Trim), but I'm guessing DigitalOcean isn't using SSDs that support RZAT since that's generally only found on more expensive SSDs, like Intel's DC S3700.

What I'm getting at is that Trim isn't guaranteed to do what you think it does. Unless the disk supports RZAT, the only way you can guarantee that the disk won't return old data in response to a read command is to write zeroes over that block.

If you're a VM provider and can't count on Trim doing what you want it to (reading back zeroes on Trim'd LBAs) because your drives don't support RZAT and you don't want zero out partitions at creation or destruction time, the right thing to do is encrypt every partition with its own randomly generated key at creation, then destroy the key when the partition is destroyed. Users will see random data soup on their shiny new block devices, which isn't as nice as seeing a zeroed out block device but is still nicer than seeing some other user's raw data. (Also note that doing this doesn't stop you from also issuing a Trim for a partition when destroying it so the SSD gains some breathing room.)

oakwhiz2• 13 years ago

Unless the disk supports RZAT, the only way you can guarantee that the disk won't return old data in response to a read command is to write zeroes over that block.

While this is true (the disk will never respond with old data once you have zeroed it out) it is important to remember that even zeroizing the disk yourself isn't a guarantee that the old data is actually gone from the disk itself - the disk may present itself as a raw block device, but internally it may use error correction, write amplification prevention, or error prevention schemes which may mean that old data will remain on the disk even though you have written zeroes over it. For example hard disks remapping bad sectors, or SSDs relocating chunks of data when the EEPROM gates in that chunk are starting to wear out. You would have to use forensic means to recover this information but it still remains. The only way to guarantee that the information cannot remain on the disk is to use encryption and make sure that the unencrypted key never touches the disk.

AaronFriel6• 13 years ago

You're absolutely right and thank you for the clarification. I didn't intend to conflate sparse files with file-based disk images, but I was trying to convey that there can be a difference between the logical data of a disk image and the physical data, and that deferred zeroing is the default and the expectation of developers and sysadmins. Images can be sparse and/or file-based, as the features are orthogonal, if cross-cutting.

More importantly, you clarify that RZAT is a necessary feature for what I'm mentioning to work properly. You're right. They should both be ensuring the blocks served to customer VMs are zeroed on use and ensure that they are appropriately running TRIM commands to ensure maximum performance from their hardware. Not all SSDs perform RZAT, and it wouldn't be a bad idea for the host to ensure the device is logically zeroed for the VM anyway.

DigitalOcean could easily switch to doing both, or at least guaranteeing the former by creating new logical disks for customers as every other vendor does. If, as they have blogged about in the past, they are directly mapping virtualized disks to the host's LVM volumes, they are unnecessarily complicating their hosting set up and making their host configuration more brittle. With thin-provisioned/sparsely-allocated or with file-based virtual disk images, they can more flexibly deploy VMs with different disk sizes with minimal changes in host configuration.

Alternatively they could trivially ensure that even forensic tools would have a very difficult time erasing volumes by enabling dm-crypt on top of LVM, and resetting the key every time a virtual machine is deleted. This could reduce performance on some SSDs (particularly SandForce based models) but would allow minimal changes to their configuration to ensure deleted data is unrecoverable.

dspillett3• 13 years ago

TRIM at the point of killing the VM doesn't really help though - all you are doing is wiping the blocks where the data is currently stored. Your logical volume could have existed in many different physical places on those disks or on others as the host rearranges for any reason or swaps old hardware for new, so there could be "ghost" copies of your (slightly older) data all over the place, people could have had it mapped into their new volumes ages ago already.

The only way to ensure your data is secure is to use encryption to start with (preferably full-volume encryption, and make sure the keys are not stored at the providers end, so you'll need some mechanism for giving the VM the keys when it reboots and will have to trust no one can somehow read them from RAM) then you don't need to wipe the data at all: just destroy all copies of the keys and the data is rendered unreadable (to anyone given a new volume that spans physical media where your data once sat, it is indescribable from random noise).

lsc• 13 years ago

note, sparse files have... performance issues. in the very best case you are going to end up with a lot of fragmentation where you aren't expecting it. I was on sparse files in 2005; I'm having a hard time finding my blog posts on the subject, but I didn't switch to pre-allocation 'cause I like paying for more disk, you know?

You are right about the zeroes, though; sparse-files solve that problem. and this is what I personally find interesting about this article. I would be very interested to find out what the Digital Ocean uses for storage. This does indicate to me that they are using something pre-allocated; I can't think of any storage technology that allows over-subscription that would not also give you zeroes in your 'free' (un-allocated) space.

For virtual hard disk technology, Hyper-V VHDs do it, VMWare VMDKs do it, sparse KVM disk image files do it. Zeroed data is the default, the expectation for most platforms. Protected, virtual memory based operating systems will never serve your process data from other processes even if they wait until the last possible moment. AWS will never serve you other customer's data, Azure won't, and none of the major hypervisors will default to it. The exception to this is when a whole disk or logical device is assigned to a VM, in which case it's usually used verbatim.

Yeah, the thing you are missing there? VMWare, well... it's a very different market. Same with Hyper-V. And sparse files, well, as I explained, suck. (I suspect that to the extent that Hyper-V and VMware use sparse files, they also suck in terms of fragmentation, when you've got a bunch of VMs per box. But most of the time if you are running VMware, you've got money, and you are running few guests on expensive, fast hardware, so it doesn't matter so much.)

Most dedicated server companies have this problem. Most of the time, you will find something other than a test pattern on your disks, unless you are the first customer on the server.

No matter who your provider is, it's always good practice to zero your data behind you when you leave. Your provider should give you some sort of 'rescue image' - something you can boot off of that isn't your disk that can mount your disk. Boot into that and scramble your disk before you leave.

I know I had this problem, too.. many years ago when I switched from sparse files to LVM-backed storage. Fortunately for me, if I remember right, Nick caught it before the rest of the world did. I solved it by zeroing any new disk I give the customer. It takes longer, especially when I ionice the dd to the point where it doesn't kill new customers, but I am deathly afraid (as a provider should be) of someone writing an article like this about me. Ideally, I'd have a background process doing this at a low priority on all free space all the time, but making sure the new customer gets zeroes, I feel, is the most certain way to know that the new customer is getting nothing but zeroes.

Zeroing an SSD is a costly behavior that, if not detected by the firmware, will harm the longevity of the SSD by dirtying its internal pages and its page cache. Not to mention the performance impact for any other VMs that could be resident on the same hardware as the host has to send 10s of gigabytes of zeroes to the physical device.

Clean failures of disks are not a problem. Unless you are using really shitty components (or buying from Dell) your warranty is gonna last way longer than you actually use something in production. Enterprise hard drives and SSDs both have 5 year warranties.

The dd kills disk performance for other guests on spinning disk if you don't limit it with ionice or the like, and that's the real cost. I would assume that cost would be much lower on a pure ssd setup.

neom6• 13 years ago

There are a couple of things I wanted to say, and I can speak with some authority on the subject as I speak on behalf of DigitalOcean.

This was mentioned to me on twitter hours ago, prior to this post. The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible and I would be amiss to point that if anyone did ever find a software vulnerability filing it and waiting 24 hours for the appropriate response is preferred. - https://www.digitalocean.com/security

As far as I can tell here, there is no unexpected behavior that isn't documented or stressed. In both our API documentation, and our control panel, we note that you must either pass a flag or check the box to security delete the data. As far as I can tell, the flag is currently functionally correctly. so..

Is the complaint that customer data is being leaked from VMs? That the flag being passed via our API/Dash isn't actually working? Or, that our policy on not doing a secure delete by default isn't something you agree with?

innoying• 13 years ago

It doesn't seem like a security issue by DO in any way. It seems that the fog api (the project this 'issue' is filled for) doesn't allow a user to access the required flag to scrub the drive. Not a DO problem in any way I can tell assuming the scrub parameter is working correctly.

nixgeek2• 13 years ago

This is a well-documented situation that almost every provider of 'consumable infrastructure' before DigitalOcean came along has faced and solved.

It is disheartening to see the same mistakes being made.

Whilst I absolutely see their USP was always solid-state storage, and that has pitfalls in terms of how you can scrub data to avoid it being leaked, the platform should take every precaution to protect customers data.

There shouldn't be an option to 'scrub data' and it shouldn't be defaulted to off so they can save some hassle, and avoid spending a few dollars. It shouldn't be an option because it should be on all the time, anything else is surprising the customer mightily.

"What do you mean, my data leaked? Oh that's fine" -Nobody

"Why'd you pick a provider who doesn't take our companies information security seriously?" -Every boss anywhere

mikeash14• 13 years ago

It is absolutely a DO problem in two ways: 1) they default to bad behavior and 2) their UI does not make it at all clear what the consequences of that bad default actually are.

lawnchair_larry• 13 years ago

Your attitude and response here is exactly why disclosing this was in fact "responsible".

neom6• 13 years ago

I asked three questions so I could address the title of the tread, that our users data is being leaked.

thatthatis• 13 years ago

As someone who isn't a DO customer, the thing that most dissuades me from this about becoming a customer is that this is a case of insecure by default. How many more of those are lurking around?

yrro2• 13 years ago

Here's another compound fuckup for you:

1. DigitalOcean users are unable to install their own kernel updates!

2. DigitalOcean have to bother making a new kernel image available via their admin interface; they haven't done this for over six months of Debian kernel updated in my experience.

3. Even if DigitalOcean did make a new kernel available, there's no notification to inform the customer that they have to log in to the admin interface and pick the new kernel from the list, then reboot their VM.

4. The list of kernels in the admin interface is sorted... bizarrely. I check it every so often and there is no sensible overall naming scheme; you are presented with a popup menu listing every single kernel for every single distribution; the latest kernel for Debian is in the middle of the list.

5. My attempt to resolve these issues with DigitalOcean support covinced me that the person I was corresponding with has no idea what a kernel even is, much less that DigitalOcean's list of available kernels is... lacking.

This situation, plus a couple the longstanding lack of progress towards IPv6 support; lack of ability to control kernel parameters; lack of a way to snapshot the filesystem for backups; makes me an unhappy DigitalOcean user who is going to jump ship for Bytemark at the earliest opportunity.

coolj9• 13 years ago

Or, that our policy on not doing a secure delete by default isn't something you agree with?

This one. Choosing insecure defaults for a virtualization API is a Bad Idea. As a rule of thumb (to put it bluntly), people are dumb. If you give them a loaded gun, they will shoot themselves with it. And they will blame you for it. At least put the safety on and make them take a conscious step before blowing their face off. Don't mean to tell you your business, but seriously, insecure defaults are a Bad Idea for a virt API.

mikeash14• 13 years ago

While you're totally right, this doesn't even come down to "people are dumb". The documentation is simply lacking here. At no point that I can find do they discuss the ramifications of not using the scrub option, and even a smart person could reasonably expect that not using that option still doesn't leak your data, just in some other way (one less safe against certain attackers, presumably).

rythie3• 13 years ago

As a DO customer I find this completely unexpected and unacceptable. Secure delete should be always done before transferring to another customer, if this takes time it should scrubbed as queued job, before the next customer gets it.

codygman2• 13 years ago

Agreed (also as a DO customer), DO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270

It's a win-win :)

integraton• 13 years ago

Any company that has ever stored data on DigitalOcean now needs to operate under the assumption that other DigitalOcean customers have accessed it.

Even if every staff member believes they checked the "Scrub Data" checkbox or used the API flag when destroying droplets, human memory is unreliable and people make mistakes.

This is a very serious security issue and it's appalling that anyone is making excuses for it, and it's even more appalling that the company responds by blaming customers.

Customers should not be able to access other customers' data under any circumstances. It shouldn't even need to be stated that providing access to other customers' data should not be the default.

icelancer4• 13 years ago

The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible

Oh bullshit. Don't deflect the issue here by complaining that you don't like full disclosure policies that many security experts agree with. (Such as, I don't know, Bruce Schneier?) If you want to get into secondary levels of annoyance, how about the fact there have been multiple instances in the past with DO that were only resolved by open/full disclosure on forums?

codygman2• 13 years ago

DO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270

It's a win-win :)

csdreamer7• 13 years ago

I have to agree with the other posters, this response is disappointing. The initial concern the other posters have is obviously the lack of a secure default.

Now i've known about this for quite some time and I mostly use DO for development and testing Chef recipes so I don't have an issue with it being off by default. I love the transparent $5 pricing. But I also don't store sensitive information on DO, yet. I also assumed when I was a newbie any sensitive info would be deleted.

The easiest option would be to include a notice next to that checkbox that if there was any sensitive information you should select this option. I understand that SSD wiping would drive up your costs. Another option would be to switch to mechanical hard drives. I don't how easy that option is with your setup.

pedrocr• 13 years ago

So you're saying, in the same response that 1) there was no "responsible disclosure" and 2) that this wasn't actually a security issue. You can't have it both ways.

mikeash14• 13 years ago

"I never borrowed your saw, and anyway it was broken when you lent it to me."

ceejayoz• 13 years ago

Oh FFS. You can't whine about "responsible disclosure" while saying "As far as I can tell here, there is no unexpected behavior that isn't documented or stressed."

If it's documented, you've already disclosed it yourselves.

mikeash14• 13 years ago

"Or, that our policy on not doing a secure delete by default isn't something you agree with?"

This one. You have chosen a default that fails deadly. It's like designing a car that explodes when you turn it off. Oh, there's a button over here you can push to disable the explosion feature. That doesn't really make it better.

You've created an option that can and will deeply screw many of your users. The mere existence of the option is not wrong by itself. But the fact that it can and will so easily screw so many people means that the option needs to have lots of flashing warning lights around it and it needs to be on by default.

I just checked out the "Destroy" tab for my droplet. There is absolutely no, none, zero indication that failing to check this box will allow the next person to occupy my spot to read all of my data. Here is the exact text:

"This is irreversible. We will destroy your droplet and all associated backups."

"Scrub Data - This will strictly write 0s to your prior partition to ensure that all data is completely erased. Estimated Destroy Time: 11 minutes 22 seconds"

I would expect "destroy your droplet" to mean that the data gets destroyed. I would expect the "scrub" option to be for paranoid people worried about the FBI seizing your equipment and using electron microscopes to extract residual data. At no point does anything in here give me any expectations that the default is "hand over all of the data currently on the VM to the next random stranger who walks in the door".

Do you really speak on behalf of DigitalOcean? If so, you need to get your head straight fast, because this is not even remotely acceptable. You cannot defend the current practice, because it is not defensible. If you don't understand why that is, you need to sit down and think about it until you do.

Right now, as a customer of yours, my thought is this: if you think this isn't important and doesn't need to be called out, what else have I missed? What other crazy data leaks do you allow by default with the defense that I could turn them off if I cared? I hope and assume the answer is "none", but now I'm rather worried.

I can kind of sort of understand how one might end up building a system like this, thinking that it was a good idea at the time. But I cannot understand at all how someone could possibly defend it once it's pointed out that it's terrible.

neom6• 13 years ago

Hey Mike,

I'm going to give you a call later this afternoon but I wanted to clarify. First, Moisey and I worked on this this morning, so worth a read: https://digitalocean.com/blog/

Second, the way this was approached was super super confusing, originally as it was in 140 characters, on the twitters. Maybe my short comings for not totally understanding the situation before I spoke, but information was fairly fragmented.

First: Yes, I do speak on behalf of DigitalOcean.

My original understanding was that this issue was with the secure delete flag not working when being passed. This promoted me to request, and continue to request, that if it was the case, security@ was notified with an outline of what is going on per http://digitalocean.com/security/ - If it's an expected behaviour, while still not good at all, it isn't my call nor was I prepared to call the company into the office at midnight on a Sunday knowing we would issue a software update in the morning. Had the flag not been respected, I would have immediately called the senior engineering team as well as Ben and Moisey, so fully understanding the situation was very important.

As a customer, I'd like you to know we do take security very very seriously, it's something we discuss going into everything, as we appreciate healthy conversations about the way our product works.

I spend 4 hours last night trying to figure out exactly what was going on, it felt very difficult to get a straight answer of "your policy fucking blows and you better change it tomorrow" - That's something I can take to the bank, but I'm sorry if I wasn't clear.

kolev2• 13 years ago

This is bad for Digital Ocean as the checkbox is nothing but a legal excuse to let beginners shot themselves in the foot and for a new company, this isn't a situation you want to be into. Look at Amazon - how do they solve the same problem? Is it harder for them? Is there a checkbox? When you compete with Amazon you have to be that and better, not worse, and treat beginners better than experts!

cones688• 13 years ago

Amazon doesn't do SSD VMs for $5 a month... If you are beginner you are using the Web GUI which clearly states you need to click the box to securely wipe all the data (which has a cost associated with it) if you are using the API then you should RTFM (Read the F Manual) if you build a service on top of the API then it is the service's fault for not including (which is what the gist is about)

owenmarshall• 13 years ago

From above: http://i.imgur.com/Aliyewf.png

What that poorly worded checkbox says to me: "tick this box if you want to prevent DigitalOcean from reading your stuff."

Nothing in that web GUI "clearly states" that not ticking the box will allow the next random user to read my files.

kolev2• 13 years ago

Well, I don't think they want to advertise themselves as "the subpar service that doesn't honor privacy and trade secrets for just $5 per month". Again, ToS and manuals are legal safeguards, not moral ones.

mikeash14• 13 years ago

The opposite of "securely wipe all the data" is not "give all of your data to the next customer".

The GUI does not indicate anywhere that data is leaked if you don't check the box.

tachion2• 13 years ago

This reminds me my own story: few weeks ago I was trying out their service and on newly created droplet I've noticed a... shell history of downloading and executing a shell script:

    1  clear
    2  ls
    3  clear
    4  wget https://kmlnsr.me/cleanimage.sh
    5  rm cleanimage.sh
    6  cd /tmp/
    7  wget https://kmlnsr.me/cleanimage.sh
    8  chmod +x cleanimage.sh
    9  ./cleanimage.sh

This looked very disturbing, so I went and check what that script is, and it is available to read for everyone, and seems to be a part of their provisioning procedure for the vm's, written by some guy who works for DigitalOcean as 'Community Organizer' (however, at that point I thought the website might be created by an attacker and misleading).

Not only it looks bad and alarming to customers, but also poses a security threat, where an attacker could target his website and/or server and replace the script with something nasty inside. How long before they'd notice such fact? No idea, but I've opened a ticket about it right on, giving them some advice on why its bad (availability, scaling, performance, security and PR reasons) but also how to better handle it, and it seems nothing has been done about it so far.

That rings a bell in my head not to use Digital Ocean service as things they do are looking pretty amateur.

coolj9• 13 years ago

To be honest, this looks like an artifact of the base image creation process.

"This file is used to clean up traces from DigitalOcean images before being published."

I don't think someone actually logged in and ran those commands on your instance. I could be wrong, but I'd bet this is just from sloppy creation of the base image leaving weird stuff in history after the image was published.

tachion2• 13 years ago

I have not said that someone does it manually, I only said, that on first look, that looks really suspicious. And, at the end, the support told me that:

"This is Kamal Nasser's script that has been set up to run on the images. The cleanimage.sh script sometimes doesn't clear the history. Thank you for bringing this to our attention. I've brought this to his attention.

There is nothing to worry about with this."

So in fact, it seems like it is being used, instead of being a leftover in shell history. In addition to that, I was later answered to the same ticket, by different support member that this script is not being used and just sits on that web page, but it all looks really bad in terms of professionalism.

quotemstr2• 13 years ago

By the way: that script is meant to be sourced, not execed: the HISTFILE manipulation at the end only works if the script is sourced --- that's why you see this script in your shell history.

iancarroll• 13 years ago

Yeah, I've seen this on my instance as well.

If the server was compromised... :\

odonnellryan• 13 years ago

I had this exact same thing.

harkness0310• 13 years ago

lol of course.

bliss_kiss• 13 years ago

these problems cannot be avoided

godyo• 13 years ago

leaks leaks leaks. no good!

dpacmittal• 13 years ago

This makes me wonder if AWS secure on this front?

tszming• 13 years ago

Customer instances have no access to raw disk devices, but instead are presented with virtualized disks. The AWS proprietary disk virtualization layer automatically resets every block of storage used by the customer, so that one customer’s data are never unintentionally exposed to another. AWS recommends customers further protect their data using appropriate means. One common solution is to run an encrypted file system on top of the virtualized disk device.

[1] http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper...

patrickg_zill• 13 years ago

Is it the same issue, or a different one, from this one back in April of 2013?

http://www.wired.com/wiredenterprise/2013/04/digitalocean/?c...

bsaul2• 13 years ago

Please DO, do something brave and make the default behavior sane ( there is absolutely no way i would except another customer to get my data with a one line shell script, ever). I LOVE your service, so dont screw it PLEASE !!

comice• 13 years ago

Since day one, Amazon EC2 used a copy on write system with their LVM volumes to protect against this problem (without them having to do expensive zeroing operations).

This has been an identified and solved problem for YEARS. No excuse for a modern VPS/IaaS provider to be leaking customer data in this way, except incompetence.

zimbatm• 13 years ago

Amazon shifts the costs of zeroing on the user of the new VM. Because the blocks are zeroes on first access writes are twice as slow on the first pass. You can try that out easily by running `dd if=/dev/zero of=/mnt/test` twice.

rpedela• 13 years ago

Yeah, but that isn't a big deal especially when you consider the alternative: leaked data.

norlowski• 13 years ago

Linkbait. Title should be "Digital Ocean API is not told to scrub (securely delete) VM on destroy"

mikeash14• 13 years ago

"Don't scrub" does not imply "give the data to the next guy".

If I delete a file on my computer without doing a secure delete, I know that the data is still on the disk and can still be recovered. However, I also know that in normal operation of the computer, that data will never show up again. There are no circumstances where I can create a new file and have it get filled out with the contents of the deleted file. There are certainly no circumstances where another user on my computer can do that and get my data.

I would have every expectation that this scrub option works the same way. That it defends against specialized recovery efforts, not random people making new VMs. DO's documentation says nothing to indicate otherwise.

bachback2• 13 years ago

for those wondering, how that compares to AWS. AWS sec policies http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper...

mark_lee• 13 years ago

I just moved one of my apps from linode to DO several days ago, some explanations here just blowed my head off.

peterwwillis• 13 years ago

FWIW, this kind of problem is minor in comparison to the potential exploits that have and will continue to crop up in shared computing environments.

https://www.cs.cornell.edu/courses/cs6460/2011sp/papers/clou...

https://www.informationweek.com/security/risk-management/new...?

http://www.insinuator.net/2013/05/analysis-of-hypervisor-bre...

http://slashdot.org/story/12/03/02/0059202/linode-exploit-ca...

http://thoughtsoncloud.com/index.php/2013/07/how-your-data-l...

I was trying to find any cases of a public cloud provider's customer data being leaked or easily visible on the internal customer network, but didn't come up with anything. Somebody's got to do a study on the major cloud providers and see if the good old methods to subvert network routes still works, or if you can easily mitm vm neighbors. (My guess is you can...)

nphase• 13 years ago

I have created a uservoice issue to scrub by default, please vote: http://digitalocean.uservoice.com/forums/136585-digitalocean...

mendelk• 13 years ago

DO released a blog post on the issue[0]

[0] https://www.digitalocean.com/blog_posts/transparency-regardi...

bachback2• 13 years ago

New response https://digitalocean.com/blog_posts/transparency-regarding-d...

I like DO as a service, but this is kind of strange. Humans act always the same. When catastrophe hits they want to sit it out, underestimating the impact

jnankin• 13 years ago

what i dont understand is, why would any one want to START with a VM that has some one elses data on it? Forget people wanting their data scrubbed on delete.

k3oni• 13 years ago

Onapp has the same issue by default, it's not set by default to zero fill the old disks are being destroyed. This need to specifically be enabled in the config file by the administrators.

In case you are running a VM on top of their platform you may want to check to make sure this is enabled.

DigitalOcean leaks customer data between VMs

Related Stories