Tell HN: Docker just ate 19GB of production data

https://news.ycombinator.com/item?id=20499397

by fhackenberger • 7 years ago

75 46 7 years ago

news.ycombinator.com

Be very careful with the live-restore feature of docker. Running 'docker volume prune' just removed all my named volumes, which were used by running containers.

See https://github.com/moby/moby/issues/38883

gervu• 7 years ago

Automation of any sort will sometimes accidentally your data, whether due to periodic hiccups, system instabilities and bugs, operator misunderstandings or errors, or random cosmic ray strikes.

The exact reason it blows up isn't even necessarily all that important, other than in its effect on what you should be doing to reduce the probability of downtime. Well-engineered systems are routinely developed from less than completely reliable parts. Stuff fails, we design for it.

It's certainly not reason not to use it, if it's resulting in a net positive gain in your ability to get things done and maintain control and transparency over your deployed systems.

But it's certainly a good reason (among a long list of good reasons) to make sure you have a good backup routine in place, including regular testing of both their integrity and your ability to restore a working prod system from them quickly.

dsfyu404ed• 7 years ago

Automation of any sort will sometimes accidentally your data,

Distributed scalable automation will accidentally your data slightly more often. The more stuff you have the more edge cases and bugs you have.

Scale big fail big as I like to say.

boobsbr• 7 years ago

Accidentally what? =)

aukaost• 7 years ago

accidentally the whole thing

hinkley• 7 years ago

your data.

TrinaryWorksToo• 7 years ago

Delete, but the word delete is deleted. Yo dawg.

goalieca• 7 years ago

You weren’t around in the slashdot days were you?

jtchang• 7 years ago

This definitely sounds like a bug.

docker volume prune says:

"Remove all unused local volumes. Unused local volumes are those which are not referenced by any containers"

If it removed a local volume that was being used by a container that is kinda bad.

RocketSyntax• 7 years ago

Not sure what kind of company you work at, but I'd export a copy of your logs so you don't get canned

stcredzero2• 7 years ago

This makes it sound like it's quite common to use docker containers operating in a heavily stateful fashion. Is that indeed common nowadays? (Though, the state in this case is only counted on to persist in the named volumes.)

sz4kerto2• 7 years ago

That's completely fine given that the important state is on volumes (named, persistent, bind mounted, whatever).

johnchristopher3• 7 years ago

Well, you are supposed to be able to delete all your volumes, containers and networks and then regenerate it by running the recettes (edit:I mean.. recipes :D) (Dockerfile, docker-compose, kubernetes, volume backup, etc.).

stcredzero2• 7 years ago

Well, you are supposed to be able to delete all your volumes, containers and networks and then regenerate

So then Docker is designed to treat all of those as disposable.

I just searched "recette" and only came up with French cooking references.

sz4kerto2• 7 years ago

I really-really hope you are not relying on Docker only when protecting 19G of data. Docker volume operations are the equivalent of playing with sudo rm -rf, shit's going to happen once in a while.

scarface743• 7 years ago

I am a Docker newbie but I thought it wasn’t considered best practice to use Docker for anything where you care about the data in the container. I’ve only used for API’s batch jobs, etc.

cheez2• 7 years ago

Docker volumes are persistent, unless they're not :-)

markbnj2• 7 years ago

You're correct that container root file systems should be considered ephemeral, and writing anything that needs to be persisted to them is a smell. However you can mount persistent volumes to a container specifically for the purpose of deploying stateful applications and referencing persistent data. How safe you can consider that data to be depends on the underlying tech that is provisioning the storage. For example a GCE persistent disk with the retain flag set is probably pretty damn durable. I would still back it up, however.

frenchman99• 7 years ago

`docker volume prune` is specifically there to remove volumes, so backing up before using it seems to be mandatory, just in case. But yeah, if this is a bug, it's a nasty one.

praseodym• 7 years ago

In the Moby issue you mention that you are using live restore (https://docs.docker.com/config/containers/live-restore/) which is most likely where the problem is. Docker daemon restarts, existing containers are kept alive, but the restarted Docker daemon doesn’t know about those existing containers yet and thus thinks their volumes are unused.

DannyB2• 7 years ago

Computers are wonderful. They can do the same work that would require a thousand people to accomplish in the same amount of time.

Flip side . . .

Computers are terrible. They can screw things up so bad it would require a thousand people to accomplish in the same amount of time.

clinta• 7 years ago

This bug specifically says it affects anonymous volumes. If you had it delete a named volume that sounds like a new issue.

orf2• 7 years ago

1. Why are you running docker volume prune in production?

2. Why are you running docker on ad-hoc machines you need to prune?

3. Why do you even need root access on production machines to fiddle around with docker commands?

While this is obviously a bad bug (and there are many with Docker), it seems more of an operational procedures failure than anything else. You could be saying:

“Beware of rm -rf /, it just deleted 20gb of production data”

Ok. Sure. But why are you tools and procedures putting yourself in a position to make that mistake?

reaperducer• 7 years ago

One of the most bothersome part of HN is when someone tells us about something that happened, and out come a ream of second-guessing replies. "Why didn't you just do this?" and Why didn't you just do that?" and any number of "It's so easy to just thing instead!"

We don't know his environment. We don't know his company's policies. We don't know his hardware, connectivity, or budget issues. These kinds of passive aggressive responses are almost never helpful.

orf2• 7 years ago

When you reduce it down the title here is “giving people access to running arbitrary, manual and presumably unrestricted maintenance commands in production leads to issues”.

That’s not a surprise, and maybe the issue at the core here is not really Docker. That’s all.

int0x80• 7 years ago

Well, no. rm -rf / is a completely different beast. It is documented and expected for starters.

They may have valid reasons to do that, even if not common.

sethammons• 7 years ago

thinking of the `rm -rf` one, here is a fun take:

  export $WORKDIR=Home/me/proj
  ...
  rm -rf /$WORKDIR

If something unsets $WORKDIR or does not set it at all, wave bu-bye to your everything. And before you say "who would do that?!" -- I believe I heard that happened to a build of RedHat that also had some kind of force push and auto-pull and build on their version control so every connected person had their version of the software nuked. If not for the non-connected individuals, the entire software would be gone apparently. Or so the legend goes.

remram• 7 years ago

Also the Steam Linux client.

wiredfool• 7 years ago

You just won the “I dropped the production db” achievement.

It’s surprisingly easy with docker, especially when dealing with .... legacy systems.

LiamPa2• 7 years ago

please assign this bug to an engineer.

The joys of open source users..

ironmagma2• 7 years ago

Seems like a reasonable ask.. I’m not sure what the problem with this is?

LiamPa2• 7 years ago

Maintain an open source project and you will understand the problem of users having the ‘just fix it’ attitude.

acid303• 7 years ago

My browser ate 16GB of ram while I've been reading this. The system crashed but the tabs were here there after a reboot. I'm not even mad anymore.

Tell HN: Docker just ate 19GB of production data

Related Stories