Nim: Deploying static binaries

https://scripter.co/nim-deploying-static-binaries/

by kaushalmodi • 8 years ago

7 3 8 years ago

Annatar2• 8 years ago

The reason why static linking has been all but banned is security and bug fixes: with static linking, the bugs and performance issues are fused into the resulting binary; fixes to libc will require a complete re-build. If you're delivering static binaries for embedded firmware images, this will quickly turn into a security maintenance nightmare, especially for end users of the final product.

Linking dynamically with libc will never be an issue on illumos- or any real UNIX-based operating system because libc includes all previous interface versions and can therefore guarantee that it is backward compatible.

Long story short: develop and build your software on an illumos-based operating system like OmniOS or SmartOS and dynamically link only with libc and you won't have to worry about static linking for convenience delivery: link on the oldest version of SmartOS and sleep well at night knowing your application will work on the latest nightly build. As an added bonus, your binary will be smaller and the runtime linker will only mmap(2) the required code pages out of libc into memory.

http://garrett.damore.org/2015/09/on-go-portability-and-syst...

kaushalmodi• 8 years ago

Hello, thanks for the detailed reply.

I am not from a software development background (I am kind of (Verilog), but am not much familiar with the best practices of software development). So you comment was quite insightful for me.

The reason why static linking has been all but banned is security and bug fixes: with static linking, the bugs and performance issues are fused into the resulting binary; fixes to libc will require a complete re-build.

Wouldn't the same apply to the whole binary itself even if it is dynamically linked? What if the app itself has a security bug? Wouldn't the overhead to fix+recompile the statically linked binary be the same as that for dynamically linked?

[Or may be the problem is that the users with the binaries build with buggy musl-gcc need to be aware that there's a security risk, and that they need to download the fixed version I later re-deployed?]

If you're delivering static binaries for embedded firmware images, this will quickly turn into a security maintenance nightmare, especially for end users of the final product.

Hmm, my goal is to deploy simple CLI apps. Here's an example of one[0] in progress. If someone wants to try this CLI app for Todoist, my goal is that they should be just able to download it and run.

Linking dynamically with libc will never be an issue on illumos- or any real UNIX-based operating system because libc includes all previous interface versions and can therefore guarantee that it is backward compatible.

Do you know if I to use illumos/OmniOS/SmartOS (this is the first time I am hearing about these) on Travis CI? And how?

Long story short: develop and build your software on an illumos-based operating system like OmniOS or SmartOS and dynamically link only with libc and you won't have to worry about static linking for convenience delivery: link on the oldest version of SmartOS and sleep well at night knowing your application will work on the latest nightly build.

Thanks! If I can do this on Travis CI (or any other auto-deployment solution via GitHub or even Gitlab), I am willing to give that a try. I am open to suggestions and pointers on how to do that.

As an added bonus, your binary will be smaller and

the runtime linker will only mmap(2) the required code pages out of libc into memory.

That part, I did not understand :)

[0] https://github.com/kaushalmodi/ntodo

Annatar2• 8 years ago

You're welcome.

"Wouldn't the same apply to the whole binary itself even if it is dynamically linked?"

No, because if the bug is in the library, under the presumption that the library is backward compatible, all that needs to happen is for a new version of the library to be installed on the system and replace the old one. Next time the application is run, the runtime linker (ld.so.1) will link the new, fixed code of the library and one's application into memory pages on the fly. Therefore, no recompilation of the binary itself will be required. That's the reason why shared object libraries and dynamic linking was invented.

The situation is quite different if the bug (or a feature) needs to be made in the binary: then one must re-compile and re-link the binary, but the shared object library can stay the same, and the code calling it from the binary doesn't have to change.

"Wouldn't the overhead to fix+recompile the statically linked binary be the same as that for dynamically linked?"

The time itself would be the same for the binary, but if the problem is in the library, things would change dramatically: one would have to fix every binary which was ever statically linked to that library, not just yours, versus just fixing the library, in which scenario the binaries would not have to change, the idea being that all binaries on the system linked to that library do not have to be modified.

"If someone wants to try this CLI app for Todoist, my goal is that they should be just able to download it and run."

Yeah, but the problem is your substrate, the OS you are building on: since GNU/Linux is just the kernel, and the userland (libc) is delivered by someone completely unrelated, there are many, many different operating systems based on that GNU/Linux kernel. They all have various versions of binaries, libraries, and even libc from different projects (some have GNU libc, some musl, ...) The correct thing to do would be to build a dynamically linked version of your binary for every GNU/Linux-based OS you want to support, but it's certainly understandable why that's impractical (infamous Linux OS fragmentation).

On the BSD's, illumos-based, and traditional UNIX operating systems, libc and the userland always come as one coherent whole, so the above concern of having a libc delivered from someone unrelated does not exist.

"Do you know if I to use illumos/OmniOS/SmartOS (this is the first time I am hearing about these) on Travis CI? And how?"

Travis CI is just a piece of software which kicks off a nightly build (fancy name: "continuous integration software"). Travis can run on pretty much any operating system, but I seriously doubt "Github" is that advanced as to run SmartOS. They probably pander to what they know and what's popular, and that's the GNU/Linux hegemony.

You could probably run Travis on SmartOS, but you'd have to do this on your own server(s): Travis people themselves punted:

"Closing this issue for now, as we have no immediate plans to add this feature.

Should we add it to the roadmap eventually, we'll make sure to update this ticket."

https://github.com/travis-ci/travis-ci/issues/1368

"That part, I did not understand :)"

http://illumos.org/man/1/ld.so.1

http://illumos.org/man/2/mmap

Nim: Deploying static binaries

Related Stories