Minify your container

by JordanTenn • 4 years ago

175 131 4 years ago

sockmeistr• 4 years ago

Isn't this incredibly dangerous? I know everyone likes to pretend they have perfect code coverage, but just ripping stuff out that wasn't called during 'probing' feels like the perfect way to make rare code paths even more dangerous.

lapser2• 4 years ago

If you have a good pipeline to prod, should be okay. You should hopefully have plenty of automated tests to ensure it doesn't get to prod if there are errors.

axelthegerman• 4 years ago

should be okay is definitely not enough for me to ship things to production.

And while I do have automated tests, they might sometimes stub system calls as I'm mostly testing my code to keep things stable and fast.

I'd rather explicitly declare my dependencies and use the same container for development, test and production to feel much more confident that it includes actually everything that's needed.

Volundr• 4 years ago

I think a "good pipeline to prod" with sufficient automated tests to ensure nothing is broken is the exception not the rule. Even in places that think/say they have a "good pipeline to prod". It's something that takes a shocking amount of engineering effort to do well, and tons of discipline to maintain.

bigpod16• 4 years ago

thats why you should test and if there are stuff htat needs to be included but arent and you know wont work fail the test and add --include-path to your docker-slim command to ensure something is added

CubsFan1060• 4 years ago

I guess it only seems dangerous to me if you blindly follow it's recommendations. Feels like it could generate a list of "things you may want to consider", that you'd then be able to use to take a look at your container.

bigpod16• 4 years ago

it sometimes doesnt work sure but thats why we have tests and test i minify all my containers nowdays and in most cases it works in those that it doesnt i figured out the pattern when and why for my apps and use include flags to ensure things remain inside

mplewis6• 4 years ago

docker-slim is incredibly dangerous and should never be used for a production app.

nicce• 4 years ago

I guess the question is in which way dangerous? It might lead for crash for sure, but is that crash controlled? If it is, then it is just a crash. Stability vs. minimal attack surface

But I agree, this is just bandaid for lazy bois. Better use Bazel etc. for distroless builds

bigpod16• 4 years ago

i would disagree i use em in production apps, i configured it and it works if you do it blindly it happens that sometimes things break but if you configure it, it will work

jzelinskie• 4 years ago

Even prior to docker-slim there were tools like Quay.io that "did the right thing" by squashing images to just the contents of the final image layer.

The best thing you can do is use minimal images and multi-stage builds. This should help you immensely to reduce your attack vector and do standard software bill of materials, too.

fwip4• 4 years ago

The quay.io squashing optimization is a lot safer though, right, as it doesn't remove anything that should be visible to the container?

I agree that the multi-stage builds are the best option, but it can be hard to know if you've included everything that is required or if you've accidentally excluded something that is important in rare cases.

OMGWTF• 4 years ago

kkrieger (https://www.pouet.net/prod.php?which=12036) is an impressive 3D shooter in only 96 KiloBytes. As one of their optimization techniques they recorded all code paths and discarded unused parts. At least in the first version this was why you could only use CursorDown in the menus and CursorUp did not work.

saurik• 4 years ago

I can't tell if this is praise or condemnation by demonstration ;P.

jamesfinlayson• 4 years ago

kkrieger was my first thought too - I think there was some other important piece of functionality that got stripped as well (I want to say something to do with hitboxes or collision detection).

killingtime74• 4 years ago

If you have good integration tests is it still a problem?

fwip4• 4 years ago

It depends on how comprehensive they are, and how important it is that your container operates correctly.

For example, even the best integration tests (for small/mid-size companies) don't always include tests that exercise weird paths around dates/times - leap years, leap seconds, daylight savings time, etc. We often trust that our datetime library or code will handle these for us, but what if the configuration is stored in a file that isn't accessed during your integration tests?

Best case scenario is you hit the error-path soon in production and your code either crashes or does something correct-enough with a fallback path, but a worse scenario is you start losing critical information and don't realize it/fix it until it's gone on for a while.

mplewis6• 4 years ago

In a non-trivial app, you can never guarantee that your "good integration tests" cover every edge case. If you could, we wouldn't have outages in production.

tmpz22• 4 years ago

Not really. There's a few common problems doing this with Go and for example, Alpine as a base image. You'll have to manually grab something like sqlite.c or some openssl stuff (its been awhile) but after that it works just fine.

Its actually a good way to find dependencies you didn't know were there. As long as you're diligent this isn't something you should be shaking in your boots over.

encryptluks22• 4 years ago

The dependencies are part of the build process. It seems like something apko would be better suited for.

mnutt• 4 years ago

Sandstorm does something similar in its packaging process. In practice it works pretty well; besides integration tests exercising functionality, the packaging generates a manifest list people typically check into source control, making it easy to see a diff between versions. I’m sure I’ve had a bad build at some point, but on the whole it works fine. Most dependencies tend to get referenced up-front anyway and any issues that arise are usually from shelling out to binaries in rare code paths.

kodah• 4 years ago

This kind of looks like a tool that does the reverse of what scratch does. Instead of only including the binary and any dynamically linked dependencies, it tries to figure out a minimum set of dependencies based on access.

In practice, I'm curious how error prone the result is.

bigpod16• 4 years ago

it is error prone somewhat but it has flags to allow you to fine tune what gets added back in. great thing is you can work with any base image and language including those that wont work with scratch

CameronNemo5• 4 years ago

Is this an official Docker project? How is this not trademark infringement?

gtirloni2• 4 years ago

It doesn't seem to be associated with Docker Inc.

Docker's trademark guidelines say that products, services and technology that are not their own shouldn't use the Docker name so it seems it's a matter of time before they get a nice letter from some lawyer.

CameronNemo5• 4 years ago

I'm not cheering for that, but I also think impersonation is dishonest.

JordanTenn2• 4 years ago

The product is a loose partner of Docker. They are an official Docker Desktop Extension. DockerSlim is an open source tool that led to the creation of Slim.Ai (and now Slim.AI and therefore Docker Slim) are loose partners.

https://hub.docker.com/extensions/slimdotai/dd-ext

bigpod16• 4 years ago

docker-slim started at docker hackathon in 2015 and company behind Slim.AI has an extension for docker desktop in its marketplace.

alberth2• 4 years ago

Off topic: wish there was a slim variant of FreeBSD.

Seems like all past attempts have stalled and/or are dependent upon FreeBSD creating a standard for what’s in a minimal userspace .

CameronNemo5• 4 years ago

Even further off topic but perhaps relevant: Chimera Linux, which consists of the FreeBSD user land ported to Linux. I wonder if q66 has OCI images published...

https://chimera-linux.org/

cperciva2• 4 years ago

Once pkgbase lands we'll probably see more progress there.

alberth2• 4 years ago

Given that it's been in the works for what seems like a decade, do you think pkgbase will be finalized anytime soon?

Just curious (please don't take my comments as being negative)

https://wiki.freebsd.org/PkgBase

jollyllama• 4 years ago

Wow, I remember when not including debug symbols was a slim image.

saidinesh54• 4 years ago

Why would someone want to use this instead of say base images made specifically for containers? like alpine for eg.?

And for languages like golang (in their examples) - why/how would anyone get such huge container images in the first place? Doesn't go give a neat statically linked binary?

bigpod16• 4 years ago

you can use whatever base image you want lets say ubuntu:latest (i dont like alpine) and normaly base images tend to include a lot of stuff that doesnt have any place in container think why do i need a tool for ext4 managment inside contianer makes no sense ok for production throw it out thats what docker-slim does and gets rid of vulnarabilities in programs that are not used by your program by simply getting rid of them

saidinesh54• 4 years ago

Ubuntu-minimal ( https://canonical.com/blog/minimal-ubuntu-released ) doesn't have any of those binaries though.

And that's also why you have multistage docker builds. To make sure your production container doesn't have all the unneeded files from your development container. https://docs.docker.com/develop/develop-images/multistage-bu... .

ufmace• 4 years ago

Good question. Alpine is already small enough that it seems a little odd to go to elaborate measures to reduce image size further. Seems better to me to start with a minimalist image and only add what you need to make your app work than to start with a huge image with everything, install your app, and rely on something like this to find only the things you don't need to remove and not make any mistakes.

xtracto• 4 years ago

Right Go binaries would be able to go with "scratch" imagine which only contain the kernel.

piperswe• 4 years ago

They don't even contain a kernel - Docker containers use the host kernel. Container runtimes based on VMs like Firecracker's firecracker-containerd typically supply the kernel themselves.

kylequest5• 4 years ago

One word... libc :-) There's tons of info on the internet about the gotchas in Alpine. Even fly.io gave up on using/supporting Alpine based container images and they are probably one of the most competent and capable engineering teams out there.

saidinesh54• 4 years ago

Honestly I didn't know that. I have only used alpine based images for some pure python webservers and was fairly happy with the size and no bugs so far. So what is the go to distro for base image for containers these days? Ubuntu-minimal?

And yeah libc was a pain for us even for AppImages. You'd think that something as fundamental as C library would be standardized on Unixes...

tbabej• 4 years ago

For workloads where the image size was critical, I have achieved a similar result with using strace to collect the required files and then limiting the image to only those files in the build process.

It's a neat approach, but ultimately brings non-negligible amount of uncertainty as you can never be 100% sure your test set of inputs did not miss a particular edge case which will require to have a file present in the container that no other input does.

bigpod16• 4 years ago

yes that tends to be the problem with docker-slim as well that is why it includes flags like --include-path with which you can easily achive such fixes

personaly i highly recommend as it works in most cases and gets rid of those vulnerablities that come with things like bash or passwd that you dont need in prod apps

rockemsockem• 4 years ago

I've had great success with reducing image size by running docker-show-context (https://github.com/pwaller/docker-show-context) and eliminating big and unnecessary files that it reports. This seems to go just a bit further than that with what seems like more complexity. I got timeouts when following their instructions to run it on two different containers, one of which is just a very simple web server.

CameronNemo5• 4 years ago

This is interesting for optimizing build time. But I think it works a bit different from docker-slim, which is focused on the final resulting image size.

Dive is a good tool for the latter IME. https://github.com/wagoodman/dive

It doesn't do the work for you, but it does single out the big layers in your image.

password43212• 4 years ago

If you want a quick double check to verify a handful of common cleanups, check out https://github.com/viraptor/cruftspy, which reports cruft like logs, caches, and git repos.

viraptor• 4 years ago

Like others here, I wasn't very happy about / trusting automatic coverage, so I made this instead https://github.com/viraptor/cruftspy

Instead of going extreme with coverage analysis, it shows places that can be manually cleaned during the build process. Maybe someone will find it useful. Smaller space gains, but gives more confidence.

bigpod16• 4 years ago

i have docker-slim in CICD completly automated seems to not have a problem as i ahve configure it per pipeline, maybe check out examples for docker-slim

jewayne4• 4 years ago

I have a minor in math, and I don't know what "shrinking by 30X" means. To me, decreases always start from 100%. So I think we are talking about a ~97% decrease in size?

JordanTenn2• 4 years ago

Thanks for this note. I'm part of the DockerSlim and Slim.AI ecosystem. Will take this feedback and rework the way we phrase things. Thank you!

reilly3000• 4 years ago

it means... a lot!

rr888• 4 years ago

Thanks I hate this, but seems to be everywhere now. "This products is now 3 times cheaper!", WTF. They still haven't got to percentages yet, like 200% off!!.

bigpod16• 4 years ago

its more about being more like people say as smaller by 200% isnt as understandable as 30 times smaller

bigpod16• 4 years ago

this is not valid but its what people say 30 times smaller.

jewayne4• 4 years ago

I know and it drives me crazy. "Bigger" and "smaller" express differences, not fractions or multiples.

fb03• 4 years ago

I don't have a minor in math, and I instinctively thought "shrinking by 30x" means 1/30 of size.

jonas21• 4 years ago

Yeah. "growing" = numerator. "shrinking" = denominator.

It's nice because they're inverses - if you shrink by 30x, then grow by 30x, you're back where you started, whereas a 97% decrease in size followed by a 97% increase in size leaves you at ~6% of the original size.

Fnoord• 4 years ago

I have a minor in math, and I don't know what "shrinking by 30X" means.

X is input

Y is output

X / 30 = Y

All you need to know, no minor required, as its taught on elementary school (age 11/12 or so?).

Karellen4• 4 years ago

So, shrinking by 2x means dividing by 2?

But... doesn't shrinking by 1/2 also mean dividing by 2?

Therefore, 1/2 == 2x ??

I feel like my elementary school math is letting me down somewhere.

OJFord3• 4 years ago

Do you know what 'two times smaller' means?

jewayne4• 4 years ago

No I don't. That's my point. To me, the number that's two times smaller than x is -x. (x-2x)

inopinatus• 4 years ago

If you try pronouncing that aloud as “…by a factor of 30” it’ll seem less ungrammatical.

AtNightWeCode• 4 years ago

For compression or similar ratio is used.

mplewis6• 4 years ago

docker-slim is the wrong solution for container optimization. You can't just have a program rm -rf files that it didn't think were in use. What if you missed a code path?

https://twitter.com/ariadneconill/status/1506482425458798593

bigpod16• 4 years ago

use --include-path to ensure its in

hnarn• 4 years ago

Why did you link a tweet to someone saying they “feel like [they] should do a blog about docker-slim at some point”? What does this contribute?

mplewis6• 4 years ago

This is a Twitter thread that continues below at the following tweet:

https://twitter.com/ariadneconill/status/1506483943352250371

Sorry – I forgot that the Twitter UI doesn't always lend itself to proper threading.

fwip4• 4 years ago

I know your question was rhetorical, but probably depends on how mission-critical the code path is (and the consequences of hitting those missing files).

If you're running a website and the removed dependency is related to a feature that is uncommon enough that isn't covered by your automated tests, maybe .1% of your users experience a broken page.

If you're running critical infrastructure and the removed dependency has to do with leap-second handling, maybe eight months from now, everything crashes and you lose millions of dollars.

kylequest5• 4 years ago

Please explain how it's wrong without simply saying you prefer other solutions, which was the case with Ariadne :) Dead code elimination is a common construct in software engineering. There's nothing magical about apps and their dependencies. They are relatively straight forward to identify and for the web apps with static assets there are helpers to help you ensure you got everything.

aejnsn• 4 years ago

“Find SSL Certs”

So we’re promoting secrets being saved within a container image artifact? Ummmm?

frenchman99• 4 years ago

The cert is usually the public key. The private key is usually named key. So it doesn't promote secrets being saved within a container as far as I can see.

Karellen4• 4 years ago

Wait, I thought the cert was the CA's signature of the public key.

jmercan• 4 years ago

Personally I feel like shrinking images by guessing unused parts is an a good way to have an image explode in your face randomly in the future. (Probes and heuristics missing critical but rarely used parts and more) Also wouldn't it hurt reproducibility? Temporary runtime monitoring doesn't exactly sound like a deterministic metric.

A containerizable project probably has its requirements known and well-specified? I think building on top of a base with a smaller unused surface is a better idea than using analysis that might backfire. These days I am using apko + melange for my personal images and they are super neat.

davidtpate• 4 years ago

Some form of tree-shaking type of thing would probably be quite handy for images, but yeah I'm a bit wary here as well. First thought would be what happens when it hits Out-of-Memory, DNS timeout, or loses network connectivity or another edge case that totally happens in Production.

Removing those code paths would not be a good thing, but I guess if you build your apps right you could just have your container orchestration system recover by replacing the Pod.

game-of-throws• 4 years ago

I wouldn't want anything killing pods every time there's a network timeout. That sounds like a quick way to turn a tiny problem into a huge problem.

kylequest5• 4 years ago

There's always a trade off. You are willing to do more low level work "manually" assembling container images your way with apko. You are also willing to accept Alpine randomly exploding in your face :) Different people have different preferences in terms of what risks they will take. For example, I wouldn't use Alpine even if somebody paid me money to use it :) That's because Alpine is not a standard Linux distro and you need to be able to "own" it with all of it's gotchas and incompatibilities. Not everybody is capable of doing that. I know I'm not that good :-) In other cases people can't change their base images.

password43212• 4 years ago

The commercial version could include a small filesystem shim that tracked references to missing files and maybe even did some work to fill them back in over the network!

moritonal• 4 years ago

Surely we'd struggle to trust a super secure container that non-deterministically downloads code over the Internet.

hacknews20• 4 years ago

Uh oh, someone mentioned “shim” which means a hack to make something that shouldn’t work, work.

nikau• 4 years ago

gotta wonder how much you are shaving off your cloud spend doing this

HowardStark• 4 years ago

Is there an equivalent tool for a normal running Linux system?

OJFord3• 4 years ago

Not the same, but potentially similar in intent, I use aconfmgr to track what's installed, changes to configuration files, and any potential changes/whole files left behind by some quick test or since uninstalled software.

(Also, even primarily to me, but less relevantly, it's great for gitting configuration & its reasons, and syncing across machines.)

sequoia• 4 years ago

Here's my less magical, more manual post on the subject of reducing docker image sizes: https://sequoia.makes.software/reducing-docker-image-size-pa...

siddontang2• 4 years ago

We build our binary first with one image as the builder image, then use `copy` to copy the binary from the builder to the final executable image like alphine.

an example Dockerfile likes:

  FROM golang:1.18.1-alpine as builder
  # RUN apk add, wget, etc, and build the binary

  FROM alpine
  # or FROM scratch
  COPY --from=builder builder/binary /binary

  ENTRYPOINT ["/binary"]

U1F984• 4 years ago

For Go you can use FROM scratch and save a couple more megabytes.

lrvick2• 4 years ago

This works on any language. I only use scratch in prod. Even for nodejs or python... compile a static interpreter binary and truck on.

Dev tools like bash, ls, grep, etc, have no place in production and only increase attack surface.

iruoy• 4 years ago

You'll need to make sure that the binary is statically linked though.

fwip4• 4 years ago

Out of curiosity, what does alpine provide for your container that you need? (I assume otherwise you'd be using `FROM scratch`.)

maccard• 4 years ago

I use wget from it for health checks [0]

[0] https://stackoverflow.com/questions/47722898/how-to-do-a-doc...

siddontang2• 4 years ago

yes, `FROM scratch` may be better most of the time. I just use `alphine` for many years, and have not tried `scratch` before.

dlor• 4 years ago

A common gotcha is ca-certs and tzdata.

kylequest5• 4 years ago

Seems like today is the day of misconceptions for docker-slim on HackerNews :-) Happy to answer any questions. AMA

One thing I'd like to mention right away is that docker-slim doesn't do function level code elimination and that simplifies what it needs to do in a significant way.

nemetroid2• 4 years ago

Happy to answer any questions. AMA

Would you like to clarify any of the misconceptions?

coding123• 4 years ago

Everyone is super negative about this. Use what works for you.

It's kinda like tree shaking in JS. Super standard practice and hugely deployed everywhere. I don't care that zip or tar are removed. I care that my app runs, passes QA and gets deployed to prod.

I don't know what other containers people are running but if you have Lotus Domino running in a container, fine, don't minimize that, I'm super happy to minimize my node process that collects HTML files from other sites.

This is great.

nijave2• 4 years ago

I feel like if you get to the point of considering docker-slim, you should consider alternate build strategies like compiling dependencies from scratch with minimal functionality (like with buildroot)

Container optimization also seems to border on premature optimization. If you're not hitting issues with your container size, don't worry about it. People may scoff at a 10GB container but for something that doesn't scale or get built a lot, it may be fine.

There are also other options like pre-pulling containers on the machines running them (which you could build into a golden image) that can get you similar startup/pull speed gains for something like host auto scaling use cases

encryptluks22• 4 years ago

10GB containers are definitely too big. All my containers are under 100MB.

atom_arranger• 4 years ago

An alternative is a multi step build, then running with Google’s Distroless https://github.com/GoogleContainerTools/distroless

Minify your container

Related Stories