Life without line numbers for 6% smaller Go binaries

https://commaok.xyz/post/no-line-numbers/

by caust1c • 6 years ago

95 92 6 years ago

commaok.xyz

ottoboney• 6 years ago

Your HTTP cert is invalid. Just a heads up.

donmcronald• 6 years ago

It's fine for me and was issued May 26. The URL when I visited was https://commaok.xyz

rnotaro• 6 years ago

His browser might have oppened the following link: https://www.commaok.xyz/post/no-line-numbers/ somehow.

It's serving a certificate for github.com.

stevekemp• 6 years ago

There's an error for https://www.commaok.xyz/, for what that's worth.

msbarnett• 6 years ago

If it's showing up as invalid for you, you may have locally revoked trust of the DST Root CA X3 IdenTrust root certificate.

sli2• 6 years ago

The www subdomain serves the wrong cert. https://www.commaok.xyz/ gives you an SSL error but https://commaok.xyz/ does not.

drudru11• 6 years ago

Hi : unrelated to this thread.

In an older thread you mention an acronym TFA. The thread was a discussion on sparse files and removing bytes from the front of a file.

What is TFA?

svnpenn• 6 years ago

This didnt seem to change anything for me. I am currently using Go 1.14.4 (Windows), and after trying go1.15beta1, the resultant executables are actually larger.

Another thing Ive always found strange is people always recommend (-ldflags -w), when (-ldflags -s) gives smaller result, and (-ldflags '-s -w') is identical to (-ldflags -s).

skykooler• 6 years ago

Given the article is talking about DWARF debugging formats, I believe the numbers in the article are for Linux binaries. I'm not sure if Windows binaries store line numbers in the same way.

ygra• 6 years ago

Doesn't Windows usually just use a .pdb file next to the exe/dll, containing the debugging information? So stripping debug information on Windows is just deleting (or not distributing) that file.

varbhat• 6 years ago

https://blog.filippo.io/shrink-your-go-binaries-with-this-on...

This is great article instructing how to shrink the Go Binaries. Author is lead of cryptography and security team of Go Team at Google.

distortedsignal7• 6 years ago

Completely agree.

I performed the steps in the article and reduced the output binary size from ~35 MB for raw go build to around 29 MB for go build with the ldflags all the way down to 8.3 MB for compressed with upx.

Really impressive results.

Karunamon• 6 years ago

Does UPX still trigger virus scanners, though? As much as I like the idea of tinier binaries, the chance that your user's endpoint security is going to freak out makes it not worth the hassle.

kingnothing• 6 years ago

35MB seems relatively small already. What do you get from an 8MB binary that you don't from one 4x that size?

ed25519FUUU• 6 years ago

tl;dr for those wondering: strip debugging and then compress the binary with upx:

    go build -ldflags="-s -w" main.go
    upx --brute main

Just tried that on one of my binaries and went from 12 mB down to 3.6 mB.

jaytaylor• 6 years ago

What about process startup time?

ausjke• 6 years ago

this is what I use, I do not use -s -w link flags these days, just run strip over the binary and it gets the job done. then upx to make the size further down.

paulie_a• 6 years ago

Does the smaller binary execute faster?

someguy101010• 6 years ago

No but the smaller the binary, the smaller the download size. Also it results in smaller docker containers, which is a popular way to ship go binaries.

birdyrooster• 6 years ago

I wonder how much more from javascript minifying can be reused in this space.

coldtea• 6 years ago

Probably not much? Javascript minifying is about raw source code, whereas those are binary instructions + debug info.

kardos• 6 years ago

How much better does a 6% smaller go binary perform?

jeffbee6• 6 years ago

Shrinking the line table will be irrelevant to performance. As I understand it, the tailscale binary has a hard size requirement to run as a VPN under iOS.

jeffbee6• 6 years ago

Since the Go compiler is so primitive you can strongly influence the size of the program by just restructuring it. An example: if you have a function that converts integer return codes to errors, and you have another function that calls it a lot:

  func codeToError(i int) error {
    switch i {
       case ...
       ...
    }
  }

  func stuff() error {
    if rv := callJunk(); rv < 0 {
      return codeToError(rv)
    }

    if rv := moreJunk(); rv < 0 {
      return codeToError(rv)
    }

    ...

    return nil
  }

You're going to get multiple whole copies of the inner function inlined into the outer one. This can add up, if the outer function is long and/or if the inner switch statement is huge.

You can save a lot of code by jumping through an adapter function instead, like:

  func callWithErrorsConverted(f func() int, g func(int) error) error {
    return g(f())
  }

... because now the compiler doesn't get to inline g into f. These things make a big difference in whole-program size. I wrote about it in the context of tailscale and protobufs here:

https://paper.dropbox.com/doc/Bloaty-Puffs-and-the-Go-Compil...

pantalaimon3• 6 years ago

I’m pretty happy I don’t have to worry about such details in C and can rely on the compiler to do the easy optimizations for me.

ascar• 6 years ago

Wait, isn't that trick basically about avoiding optimization (inlining the function to avoid the context switch) to reduce code size?

The C compiler would do something similar here and also increase code size for performance unless you explicitly use `-Os`

jeffbee6• 6 years ago

It's not really a language problem, it's a compiler problem, so at least in the future we might get a compiler that recognizes error-handling idioms. Or we just get llvm? The language problems, like the performance-hostile calling convention, are harder to fix.

gameswithgo3• 6 years ago

ohhh you think weird stuff doesn’t happen in C compilers? heh

rowanG0772• 6 years ago

Pretty insane that the compiler for such a popular language is so basic.

gameswithgo3• 6 years ago

its also one of the fastest these things were a choice

badsectoracula2• 6 years ago

Since the Go compiler is so primitive...

That doesn't sound like Go's compiler is primitive, a primitive compiler wouldn't attempt the inlining at all. This sounds like a performance optimization which isn't something that a primitive compiler would attempt.

lostmyoldone2• 6 years ago

The compiler is not doing any sophisticated analysis at all, it can inline since the language is designed to be easy and fast to compile, but in my opinion at the expense of a little too much.

I went through most of the compiler about two years ago to see ,amongst other, how much of its time was spent during IO since both compile time and correctness of dependency handling varied wildly in virtualized environments.

Which is where I learned a lot, as for example that the correct way to invoke the compiler is essentially impossible to do with the supplied tools, but if you do it, it was actually possible to not recompile system libraries each time you build, that refactoring code where module scope is used for a most things is incredibly tedious. To compile to/from memory, as a service I had to change approximately 10 000 lines of code without being entirely done, as everything was tied into the module namespace and a little bit too concrete vfs like file access mechanism. Thus both the lifetimes of data, and where IO went were impossible to change without changing almost every type declaration site, hence often declaration, and attribute reference cite. Anything module based needed to be threaded through in appropriate places. Yes, and because some package names are hardcoded to be forbidden by parts of the toolchain unless it knows it is compiling itself, you have to patch the compiler (iirc) and the build tool if you want it to compile itself to a different name, which you kind of have to when you are doing something incredibly experimental. Otherwise it'd just overwrite itself because of gopath. Because of how the compiler/build works, it becomes contagious, so you'll need different names for everything. Well, you could probably get away with a little less renaming if you compiled the compiler with make, but at the point I realized that, I was already too far in for it to be any point in backtracking.

Hardest issue was when it deadlocked on a mutex for a while, turns out one the main compiler go routines concurrency were not implemented correctly, so without the implicit serialization of the disk IO, it deadlocked almost every compile.

Apologies for the digression, go was kind of forced on me, and some of the most touted "opinionated" parts lost me countless of hours of grief.

pansa2• 6 years ago

You can usually save double-digit percentages by stripping debugging information: pass `-ldflags=-w` to `go build`.

Is there a way to make `go build` for a particular project always use `-ldflags=-w`? Or do I have to remember to type it in every time (or hide my `go build` command inside a Makefile or `build.sh`)?

Life without line numbers for 6% smaller Go binaries

Related Stories