XARs: An efficient system for self-contained executables

https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

by terrelln • 8 years ago

246 121 8 years ago

code.fb.com

rwmj4• 8 years ago

I'm not really a fan of containers, but I read this and thought "why not containers"?

The page mentions cryptically "They could almost be thought of as a self-executing container without the virtualization". The "self-executing" bit makes sense - you don't have to remember to type "docker". "without the virtualization" doesn't make sense unless they mean without cgroups or are talking about Kata Containers.

terrelln6• 8 years ago

XARs are just self mounting compressed readonly filesystems with an executable inside. We get hermitic dependencies by setting the PYTHONPATH, LD_LIBRARY_PATH and such in the bootstrapping script.

One big benefit is that the filesystem only decompresses the pages as needed, which greatly improves the start up time over existing solutions.

ctur12• 8 years ago

Generally XARs are lighter weight than a container. While you can (and sometimes we do) use XARs to deploy, say, a self-contained service like a website, often they are used to replace command line tooling.

Container isolation (cgroups, namespaces, etc) would make it difficult to do some of the system level tasks we use such tools for, such as configuration changes or monitoring.

Likewise, we often are replacing a PAR or C++ tool with a new XAR version, and it is nice to simply replace the executable and not have to change how it is invoked. In this regard, invoking a XAR is identical to running any normal executable or shell script.

e12e• 8 years ago

I can see why you'd sometimes want less isolation (although things like docker-compose runs fine from a docker container). But how is it "lighter" than a container? Aren't you striving for self-contained executables? What do you leave out of a XAR that you'd want to put into a container?

[ed: I now saw this question and answer:

https://news.ycombinator.com/item?id=17526178

Frankly using system/external python (or other VM) seems a bit risky... But whatever works, I guess..]

lozenge• 8 years ago

"virtualization refers to the act of creating a virtual (rather than actual) version of something, including virtual computer hardware platforms, storage devices, and computer network resources."

Containers have virtual file systems they access instead of the host one. XARs don't.

fenesiistvan• 8 years ago

I am a windows developer and the single thing that stops me porting my apps to linux is an easy to use deploy method. Is there some good way to handle this task without to spend months learning about linux administration like shell scripts, finding the best place for configs, logs on different linux distros, daemons setup, etc. Something simple and distro independent would be fine...

rwmj4• 8 years ago

I am a Linux developer and the single thing that stops me porting my apps to Windows is an easy to use deploy method. Is there some good way to handle this task without to spend months learning about Windows administration like installers, MSI, the registry, logging, services?

The non-flippant answer is to just provide the source and let the distros package it for you. It's a different model. Linux users want to get their software through an integrated package manager, and volunteers will take your software and do all the work needed to make that happen.

RcouF1uZ4gsC• 8 years ago

It is very easy on Windows to create a statically linked executable that you can just copy to wherever you want.

TimJYoung2• 8 years ago

You can typically deploy a Windows binary to targets that use much older versions of Windows, and you cannot say the same for Linux distributions. There is significant backward-compatibility with Windows binaries, and it is intentional. It provides for a much better ROI for applications that need to be around for more than a couple of years.

AnIdiotOnTheNet• 8 years ago

just provide the source and let the distros package it for you.

And then sit and scratch your head over why software vendors don't sell a version for your OS.

juliangoldsmith2• 8 years ago

Is your application open source?

If so, I wouldn't worry about packaging for every distro. Just make sure that your application isn't difficult to build, and most things (paths, etc.) are configurable. The config path can be handled with a command line switch.

Once you've gotten your application so that it can be built easily, I'd only really worry about packaging it for your distro of choice. If people are interested in your application, it'll get packaged for their distros.

If your application is proprietary, I wouldn't even worry about packaging it yet. Getting most Linux users on non-essential proprietary software will be an uphill battle.

shock2• 8 years ago

AppImage⁰ seems to be what you're looking for. XARs maybe, but I don't have experience with them to recommend them.

⓪ - https://appimage.org/

jwatte3• 8 years ago

What you want is snapcraft.io

eikenberry2• 8 years ago

IMO AppImage is better for his needs. Snap, like Flatpak, are as much distribution systems as they are packages and they are not widely enough adopted yet to make packaging for each worth it vs just providing the binary to download yourself.

sjellis• 8 years ago

For desktop applications, Flatpak does this. Otherwise, containers or the new "portable services" feature of systemd. Thanks to systemd, there are standardised ways of doing a lot of things, but yeah, distributions do vary, so you really need an abstraction, or target just a few of the popular distributions.

nvivo• 8 years ago

I must say as a long time windows developer, .net core on linux is much simpler than on windows. I'm moving everything I have from windows to Linux and the experience, simplicity, speed, stability, etc are much better on linux.

I guess of you're working with gui that's a different story. But for .net websites and background servers, simply use docker on linux and never lool back. And it's much simpler than docker for windows too.

aumerle2• 8 years ago

It's somewhat counter-intuitive that start times with XAR are lower than start times without it. Is fuse faster than a kernel filesystem? Even with compression?

terrelln6• 8 years ago

Admittedly I haven’t profiled this yet, but my guess is it is a constant overhead of setting up pkg_resources that the native code uses to load the entry point.

The test against native start speed was hot, so the pages required were already in the page cache, so the filesystem shouldn’t matter.

ctur12• 8 years ago

FUSE isn't generally lighter weight than a filesystem but it can be relatively competitive for simple use cases like a read-only filesystem. Additionally, squashfs lets you pack metadata and data very tightly, and since it is a readonly filesystem, has some optimizations normal filesystems can't (how data is placed, overhead of managing metadata operations, etc). Also squashfs lets you choose how the files are laid out and compressed so that all files of a certain type, such as all .pyc files, are close together, which increases compression ratio and reduces overhead for subsequent file accesses (i.e., can reduce random disk or flash IO).

In practice the timings of XAR vs filesystem are close enough to be "in the noise" -- it's when compared to PEX or PARs that the difference is quite large.

aumerle2• 8 years ago

Makes sense, this is somewhat analogous to the import speedup you can get by putting all the python modules into a zip file. I tend to do that when distributing python applications on windows, where the speedup is more noticeable.

terrelln6• 8 years ago

I spent some time today investigating what exactly is causing the difference between native and XAR start times. I confirmed the culprit is `pkg_resources.load_entry_point()`. Modern installations using wheels should avoid this overhead, and those native installations will be slightly faster than XARs:

black: 0.171 s (vs 0.208 for XAR) jupyter: 0.165 s (vs 0.179 s for XAR)

My test setup used the older loading method because "pip install ." won't install wheels if the wheel package isn't installed in the virtualenv.

ctur12• 8 years ago

Hi, I'm Chip, one of the authors of the blog post and XAR itself. Happy to answer any questions anyone may have about how XARs work, the way we use them, or the motivations that drove their development.

jkingsbery• 8 years ago

How do you pronounce "XAR"? "Ex-AR"? "Sar"? "Shar"?

ctur12• 8 years ago

One syllable, rhymes with "car" but with a "z" sound ("zar").

progval• 8 years ago

XAR is pronounced like "czar" (/t͡ʂar/). The 'X' in XAR is meant to be a placeholder for all other letters as at Facebook this format was originally designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives), LAR (Lua archives), and so on.

https://pypi.org/project/xar/

shock2• 8 years ago

Hi Chip, thanks for releasing this as open source. From a quick look XARs seem pretty similar to AppImages in the sense they both use an executable preamble and a squashfs so I'm wondering which would be a better choice for me to distribute my Python apps, and why. Thanks!

ctur12• 8 years ago

I am not an expert in AppImage but I think the main thing is it is pretty easy to make a XAR of a Python script and that the overhead is probably less (especially for repeated invocations).

I think XAR is simpler than the other alternatives; they have more complex specifications, being aimed at distributing full GUI applications (though I bet they work fine for these kinds of use cases, modulo perhaps quick and easy conversion of a Python program into a XAR). I suspect, but haven't measured, that XAR has lower execution overhead since it will re-use a mount point if a tool is recently invoked, rather than remounting every time.

One thing XAR doesn't really try to do is work cross-platform. It will rely on the system Python, for instance, rather than embed the interpreter and all libraries inside (it will embed the libraries your tool depends on that aren't part of Python itself). This is a pro in some cases (lower overhead), but a con if you want something you can carry across wildly different systems.

oahayder• 8 years ago

Hi Chip. How does facebook deploy XARs? Is an advantage over docker containers not having to run the docker daemon on every machine?

ctur12• 8 years ago

We generally deploy XARs like a normal executable (delivered via RPMs or other native packaging). We also have an internal packaging format we use to deploy them as well. The big thing for us is we treat them like normal executables, so any standard deployment model for an executable works for XARs.

ksec• 8 years ago

Is it related to / inspired by Ruby Packer[1]? Because I am skimming it and seems 90% similar.

[1] https://github.com/pmq20/ruby-packer

ctur12• 8 years ago

Not related, but at first glance it looks similar. However, it appears that ruby-packer uses a library to access the squashfs image rather than actually mounting it. This makes it difficult (probably impossible) to load .so files from the squashfs file without having to copy them externally or to carry other helper executables along inside the image without also having to copy them externally to invoke.

A big goal for XAR is to be utterly transparent and not require hooks into Python, Node, or other runtimes to function.

mmastrac• 8 years ago

Awesome work. Did your team evaluate creating a virtual filesystem that could process the SquashFS images without involving the kernel? Having completely independent executables that could run on _any_ system with zero additional install would be sweet.

To clarify - a stub in each XAR would act as a filesystem driver and intercept calls to open/read/etc, redirecting them to the internal data blob.

Edit: I see your comment below which answers this! https://news.ycombinator.com/item?id=17524910

rkeene2• 8 years ago

For what it's worth, this is how Tcl self-contained executables work [0], since Tcl already has a virtual filesystem mechanism.

[0] https://www.tcl.tk/starkits/

jwatte3• 8 years ago

Chip, thanks for sharing!

Could you have solved the same problem ("deploy apps with dependencies in a single file") using snaps? (snapcraft.io)

ctur12• 8 years ago

I think XARs are much simpler. snapcraft (and AppImage and other tools) seem to be much more invasive to the development flow and more opinionated about what the contents of the image look like (directory structures, yaml/xml/json files for metadata, etc).

XAR is much simpler. All it needs is a shell script (or raw executable) to run inside the XAR. Everything else is up to the bootstrapping. Note this isn't always a good thing -- opinionated software has its place. For what we use XAR for, though, having it "just work" with standard Python tooling without expecting open source libraries or modules or build tools behave differently is a strong plus.

CBLT• 8 years ago

I recently found out about Singularity[0], which seems to be very similar (squashfs application bundle). What advantages does XAR have?

[0] https://github.com/singularityware/singularity

neuromantik8086• 8 years ago

A budget? Singularity was basically born of not-invented-here syndrome in the neuroscience community and doesn't have nearly the amount of resources that Facebook is able to devote to this problem. If that were to change (i.e., Greg Kurtzer gets millions of dollars in VC or NIH funding out of nowhere) and Singularity weren't a lone developer project it might be worth considering as a viable alternative to XARs.

raghava• 8 years ago

Hi Chip!

Did you evaluate flatpak/appimage/snap before developing XARs? If so, what were the shortcomings that you noticed in them?

I have found reasonably good success with snap on Python and node apps, but am not an expert on them. Just want to know from other practitioners about any gotchas that others might have studied/stumbled upon.

Thanks!

ctur12• 8 years ago

XAR development began in 2014 and hit production use in an early form in 2015; this was before some other options were out there, or before, I believe, AppImage went the squashfuse route.

XAR doesn't aim to, say, represent a sandboxed filesystem or provide some isolation or provide a specification for GUI apps to seamlessly integrate with your desktop. Instead, the main idea is to just get a single file that, when run, mounts a filesystem and runs a command inside it. In my mind, at least, it's a simpler, smaller primitive you could build other systems on top of (like we have with deploying Python, Lua, and Node).

juliangoldsmith2• 8 years ago

This all reminds me a bit of Tiny Core Linux. IIRC, it uses SquashFS images for all its packages, mounts them in a specific spot, then uses either symlinks or UnionFS to put everything together.

mediocrejoker2• 8 years ago

I wonder how this compares to AppImages https://appimage.org/

thangngoc892• 8 years ago

- AppImages: Linux apps that run anywhere - XARs: packages Python (node.js, lua scripts app) into executable files

mediocrejoker2• 8 years ago

Thanks for providing that summary. Does this mean the two could be used together?

secure2• 8 years ago

Cool idea! Is there any particular reason to use SquashFS via FUSE instead of via the Linux kernel driver?

Slightly related: we also recently switched to SquashFS for the gokrazy.org’s root file systems.

If you’re curious about how SquashFS works under the hood, check out https://github.com/gokrazy/internal/blob/master/squashfs/wri.... I also intend to publish a poster about it at some point.

ctur12• 8 years ago

We actually started with using "real" squashfs files. This had three main disadvantages:

- We had to maintain our own setuid executable to perform the loopback setup and mount (rather than relying on the far more tested and secure open source fusermount setuid binary that all FUSE file systems rely on) - Getting loopback devices to behave inside of containers (generally cgroup and mount namespace containers) was a little tricky at times in some of our environments - We didn't want to have a huge number of extra loopback devices on every host in our fleet

In fact, after implementing the loopback-based filesystem version, we almost abandoned XAR as the downside of the security considerations and in-container behavior wasn't ideal. The open source squashfuse FUSE filesystem really is what made it possible.

Another side benefit is we could iterate far faster with squashfuse -- this let us fix some performance issues, add idle unmounting, and implement zstd-based squashfs files, and then deploy that to our fleet, faster than we could deploy a kernel to 100% of hosts.

secure2• 8 years ago

Thanks, makes sense!

fooblitzky• 8 years ago

Sounds similar to TCL's StarKits

rgovostes2• 8 years ago

Probably too late now, but xar already stands for "eXtensible ARchiver" and is a file format used on macOS in some package installers. It's notable for having an embedded XML "table of contents" that describes metadata of the archived files, so new fields can easily be added while maintaining backwards compatibility. (Compared to, say, the zip file format which does not even specify how to store Unix file modes.) https://en.wikipedia.org/wiki/Xar_(archiver)

JohnDotAwesome3• 8 years ago

Names are hard. My name is John. Programmers get really confused when I tell them that. "Did you know there's another programmer named John? You probably should have researched names before you decided to go with that one"

mindslight• 8 years ago

The larger "person's name" namespace includes the first name and last name. And people do generally treat it as a surprise when they find someone with the same name.

dsr_2• 8 years ago

Namespaces can help with that. For example, I note that you aren't just John, you're John Fawcett. And you probably have a middle name (or two, or more).

I bet you've worked somewhere with a collision on John. Did you adopt a handle, like John F, or maybe JohnDotAwesome?

skolemtotem• 8 years ago

But look, if I started a company called McDonald's, everyone would be confused and it'd probably be illegal. Complaining on forums about names is the open source community's alternative to trademarks.

kstenerud• 8 years ago

Yup, just like with people, name clashes in computers won't cause problems!

    # apt install docker
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following NEW packages will be installed:
      docker
    0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    Need to get 12.9 kB of archives.
    After this operation, 45.1 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 docker amd64 1.5-1build1 [12.9 kB]
    Fetched 12.9 kB in 1s (24.9 kB/s)
    Selecting previously unselected package docker.
    (Reading database ... 13860 files and directories currently installed.)
    Preparing to unpack .../docker_1.5-1build1_amd64.deb ...
    Unpacking docker (1.5-1build1) ...
    Setting up docker (1.5-1build1) ...
    root@testing:~# docker
    bash: docker: command not found

dsnuh3• 8 years ago

Funny story... We had a contractor start named "John Smith" and this guy just could not get on the domain, receive email, etc. All his shit was just not working until like a week after his start date.

My team doesn't handle new user creation, and it's a big company, where sometimes other teams may as well be other companies. I kept getting tickets from this guy and he's reporting that none of his stuff works. There's nothing I can do, but I try to help the guy out.

Anyway, a week later his stuff works. I guess whatever team does the new user creation never planned for there to be more than four employees with the same name, and their automation broke on "johnsmith5". Sigh.

rgovostes2• 8 years ago

Yes, if you were living in a world where the bank would only give an account to the first person named "John" to sign up, you should have researched names first. It will be fun dealing with file associations over the .xar extension.

Companies used to coordinate to make sure they didn't conflict on port numbers, Apple used to run a database for ensuring that creator codes were unique, etc. Now one of the biggest tech companies can't be bothered to check if they collide with another large tech company in a sparsely populated namespace.

jwilk2• 8 years ago

I see a recurring pattern:

https://news.ycombinator.com/item?id=17285707

fintler• 8 years ago

Same thing with Flux and Yarn.

berti• 8 years ago

Yes, I was really confused by the AppImage comparisons in this thread until I realised this wasn't the XAR I already knew. It's hard to imagine they couldn't have easily checked the name before claiming it. The (Darwin) XAR page on Wikipedia was created in 2006 [1]...

[1] https://en.wikipedia.org/wiki/Xar_(archiver)

jarvuschris2• 8 years ago

Have you looked into Habitat? It provides a similar result with a complete build workflow that works across technologies and platforms: https://www.habitat.sh/

There's a rapidly growing library of libraries and services packaged with it: https://bldr.habitat.sh

Its build artifacts can be exported to a number of formats including container images and tarballs, maybe a XAR exporter could be built: https://github.com/habitat-sh/habitat/tree/master/components...

JohnDotAwesome3• 8 years ago

Seems like Habitat (which looks awesome by the way) relies on Docker. Which, if you consider performance heuristics in the article (size, cold/hot start time), may be a non-starter for what they're trying to do.

djb_hackernews• 8 years ago

Can you expand on that? In my experience nothing about Docker implies a performance impact in terms of size or start time.

jarvuschris2• 8 years ago

It does not rely on Docker. On a Linux system you can use the build tools and there's no dependence on Docker at all, it uses chroot's for its isolated "studio" build environments. The mac and windows habitat clients use Docker to create a Linux-like environment

One of the key innovations in habitat is that it gives you reliable dependency isolation WITHOUT needing the runtime isolation of containers to achieve it. Containers become optional.

A habitat build artifact can be installed and run natively on nearly _any_ POSIX system, the state of the system doesn't matter and you get the same behavior top-to-bottom. There are a number of "exporters" available to repackage build artifacts into different runtime formats -- Docker container is just one option

saagarjha• 8 years ago

Kind of an unfortunate name, considering that xar (eXtensible ARchive) is already a thing: https://en.wikipedia.org/wiki/Xar_(archiver)

nine_k• 8 years ago

I'd hazard to say that almost any 3-letter abbreviation has been already taken, many of the easy-to-pronounce ones, multiple times.

52-6F-623• 8 years ago

Oh there are some fabulously childish three and four letter abbreviations available yet. Probably not very appropriate, though.....

https://fileinfo.com/browse/

0xbadcafebee• 8 years ago

I would use this if it didn't depend on OS-specific features. Squashfs is not portable to Windows, unless you extract it to disk.

I actually prefer the jar/Tomcat model, where the read-only image gets distributed to servers, and when you run the app the image gets unpacked to disk as needed. You could also write I/O wrappers that would obviate the need to extract them to disk, and you could even make compression optional to reduce performance hits.

It seems like all you really need is a virtual filesystem implemented as a userspace i/o wrapper. Basically FUSE but only for the one app. There's no need for the FUSE kernel shim because only the application is writing to its own virtual filesystem. So this would work on any operating system that supported applications that can overload system calls.

For example, I would start with this project http://avf.sourceforge.net/ and modify it to run apps bundled with itself. With FUSE installed, other apps could interact with its virtual filesystem, but without FUSE, it could still access its own virtual filesystem in an archive. I would then extend it by shimming in a copy-on-write filesystem to stack modifications in a secondary archive.

twsted• 8 years ago

I would use this if it didn't depend on OS-specific features.

I always make the same mistake to assume that none would deploy something on windows for a server–side environment.

ctur12• 8 years ago

I agree it is a bummer that FUSE doesn't directly work on Windows, but it should be doable -- we would love for someone to figure out the best way to do this on Windows. Happy to collaborate with anyone who'd like to make this a reality.

PeCaN• 8 years ago

While FUSE doesn't work directly, there are userspace filesystem implementations for Windows. Dokany¹ even implements most of the FUSE API.

1. https://github.com/dokan-dev/dokany

TheAceOfHearts• 8 years ago

I'm on mobile, but do you have an example of bundling a node app somewhere?

I'm curious how it compares to using something like pkg [0].

[0] https://github.com/zeit/pkg

terrelln6• 8 years ago

We currently don't have a nice open source API for building node apps, but would welcome PRs that get us in this direction!

There are two ways to build a node app using the XAR builder tools. 1. Use the `make_xar` tool which will create a XAR from a directory and takes an optional script to run on execution. 2. Use the XAR builder library to make a XAR builder that is specialized for building node apps.

dagenix• 8 years ago

Does the XAR file containing the Python executable itself, or, does running it rely on having Python installed on the host already?

terrelln6• 8 years ago

The current Python XARs rely on Python being on the system path. But it would be easy to build a custom Python XAR with the XAR builder library that includes the Python executable and makes sure to use the packaged executable.

ASinclair• 8 years ago

How similar is this to Google PAR/SAR executables for Python and Bash scripts respectively?

terrelln6• 8 years ago

Facebook's PAR is a self-extracting zip file, I assume Google's is similar. XARs are self-mounting SquashFS archives (a compressed read only filesystem). This means that XARs don't have to be extracted to a temporary directory to run, they can run in place. Zip files have to be completely extracted before running, but SquashFS decompresses pages on the fly, so startup times are much faster (especially with zstd compression).

wesleyy• 8 years ago

I don't think Google's implementation of their hermetic par files are open source.

jeffrallen• 8 years ago

nikolay• 8 years ago

Is Facebook an NIHS (Not Invented Here Syndrome) sufferer?

lttlrck• 8 years ago

Indirectly this proves to be a useful discovery mechanism for me - when tools crop up on HN I think ‘hey that’s interesting’, then oftentimes when I read the comments I find there are numerous existing solutions I had never heard of along with helpful links and insightful info :-)

It’s brilliant. It’s one of the reasons I value this site so much.

dcgudeman• 8 years ago

Requirements

Python >= 2.7.11 & >= 3.5

you need both?

jillesvangurp• 8 years ago

Not a bad idea. I wonder how this compares to ubuntu's snaps. Seems like a good idea to me but I've not really seen it used much yet.

On OS-X apps have been distributed in a .app form for ages. It's very uncommon for OS X apps to have installers or a more complicated installation (and uninstallation) than drag and drop.

So, good idea and it kind of fixes a big issue where most linux distributions seem to insist on dll hell with just about anything littering the file system with cruft and just about every interpreter out there reinventing ways to create virtual environments.

FRidh• 8 years ago

Yet another format for self-contained executables, and one that looks pretty similar to the already existing AppImage.

Note that Nix users can use `nix-bundle` to create AppImages of all the software in Nixpkgs, which is according to Repology one of the largest and freshest package sets: https://repology.org/statistics

russellbeattie• 8 years ago

Facebook spent a decade contributing virtually nothing to open source, now they're flooding the world with random projects of varying and questionable value - most developed as a result of Facebook's severe N.I.H. attitude. I'm honestly not sure which is worse.

tony-allan• 8 years ago

XAR is a simple way to package and deploy Python and similar apps. Dependencies are a real issue. Anything to improve the situation and to also deploy related files is a great idea.

I don't like a lot about what Facebook does with user data and marketing but their support of open source is better than many companies. Give credit where it is due.

XARs: An efficient system for self-contained executables

Related Stories