Speeding up LXC container pull by up to 3x

https://databricks.com/blog/2022/01/26/creating-a-faster-tar-extractor.html

by ctecte • 4 years ago

97 22 4 years ago

scintill762• 4 years ago

GNU tar seems to auto-detect compression type as well. I don't bother passing a flag to say which type to use, when I'm extracting.

ctecte8• 4 years ago

Author here, it's true that GNU tar auto-detects compression when specifying a local tarball to extract. However when piping a file stream to tar to extract it actually doesn't auto-detect and fails if you don't explicitly specify a compression type. Should have made that more clear in the post.

scintill762• 4 years ago

Oh, that makes sense. Nice addition.

stabbles• 4 years ago

How about squashfs files for every layer and overlay mount those. At the end of the day you access probably .1% of the data, so with this you'd defer decompression to runtime and only decompress what you need

ctecte8• 4 years ago

Had not considered that. Reading about squashfs I see that it's readonly. Do you know if that's fine when mounting the directory for an LXC container? Also I wonder how much performance impact (if any) there might be on spark jobs due to the decompression at runtime.

To be completely honest this all started as a quick hackathon project just to speed up downloading before piping to GNU tar. Only later did I consider also re-implementing the tar extraction in a parallel manner. If considering changing the overall packaging method from tar to something else there's a lot more ways to consider going about this (including squashfs).

I'd say one of the nice things about this is that tar is fairly ubiquitous, not just in container images. For example, we also have tarball build artifacts in Jenkins jobs that could benefit from this tool.

tarasglek2• 4 years ago

You can cut down the startup time to milliseconds

1) either mount tar directly

2) or use squashfs and mount that...both would be mounted with some sort of rw overlay

3) underneath do some lazy httpfs type filesystem, so filesystem can be mounted while it's downloading

4) parallelize the underlying download ala aria2c

5) provide metadata to aria2c-alike downloader with file boundaries, so extract further latency/bandwidth savings

ffk• 4 years ago

Something to consider, if you are IO constrained, compression may speed up reads because you shift some of the cost of IO to the CPU.

Ultimately, you'll need to measure this to know for sure, and those results will likely only be valid on a given hardware configuration.

OverlayFS also has a "copy_up" function, where the file is copied at the initial write. Once the copy is done, I'd expect write access to be fast. Again, you'll need to measure this.

The setup could probably look like:

container read/write -> OverlayFS([mutable fs as overlay] -> [squashfs layer as underlay] -> [squashfs layer as underlay])

nhoughto3• 4 years ago

Came here to say exactly that, squashfs with overlay mount for writes beats this easily.

Hard to change the entire container ecosystem away from tarballs tho so this is still useful. Is it available anywhere? I didn’t see a link?

ctecte8• 4 years ago

Company wanted to see if there was any interest in it before putting in effort to open source :/

Hoping this is enough to release it on Github.

tyingq• 4 years ago

That's pretty interesting. One downside might be losing the ability to do checksums on what's been downloaded, as a safeguard. I wonder if there's a way to preserve that without ruining the parallel downloads and un-tarring.

ctecte8• 4 years ago

I did look into performing stuff like md5 inline while streaming the download, however it tended to slow down the extraction significantly. Other less rigorous checks like crc32 are still possible inline though with minimal performance reduction.

It should also be possible to split off and write the raw download to disk in the background with something like an io.TeeReader object [0]. You could then checksum it after the fact like you normally would.

[0] https://pkg.go.dev/io#TeeReader

fdr• 4 years ago

A long time ago, I wrote something similar ( https://github.com/htcat/htcat) to assist with Heroku's efforts to speed up moving the tar formatted application releases around. It's pretty old, it doesn't integrate tar archival itself, it probably can stand improvement. Or given its small size, even a rewrite.

My favorite hack in there (that also made it work with pre-signed S3 urls) was not using the HEAD method as is customary to determine object sizes, but instead doing a regular "GET" that, for small files, would execute on its own...but for larger would simply be abruptly closed by htcat once it reached the bytes that had since been fetched in parallel by a range-based request sent immediately afterwards. The goal was to have htcat not offer a penalty on small files so it could be used on blended workloads without thinking.

It also found a bug in S3's range implementation. We had a problem with some object or other, I wrote in about it, and was told that upon investigation a bug had been fixed. No more problem.

Some kind soul packaged it for Debian, so it's easy to get if you need it.

ctecte8• 4 years ago

Oh nice! Yeah this is basically the same design/intuition that this project also started out with (I also just wanted some way to combine the benefits of aria2c with piping). Wish I found this earlier lol.

I only later decided to look into also parallelizing the tar extraction as well. This originally started with looking into making a change to the GNU tar source code itself. However upon cloning I realized there were a lot of design choices in tar that made the assumption of single threaded execution (global state, etc). At this point I decided it would just be easier to re-implement the tar extraction myself using golang's "archive/tar" package.

jarpineh• 4 years ago

This is interesting and seems general purpose. Not merely for container images.

There’s this option for OCI containers which I don’t pretend to understand: https://github.com/containerd/stargz-snapshotter

It is used by containerd and nerdctl. You do have to build the image with it. Images work in OCI compatible registry. By fetching most used files first container can be started before loading is finished. Or so I gather.

curt15• 4 years ago

Tangentially related - do any shops still use LXC containers instead of Docker/K8s?

Speeding up LXC container pull by up to 3x

Related Stories