Go command support for embedded static assets (files)

https://go.googlesource.com/proposal/+/master/design/draft-embed.md

by eberkund • 6 years ago

211 164 6 years ago

go.googlesource.com

nicoburns3• 6 years ago

Seems pretty complicated compared to Rust's `include_bytes!("path/to/file")`. Sounds like it might have a compile time advantage though, seeing as it produces a seperate package.

Thaxll2• 6 years ago

It seems much more powerful than the Rust version. The Rust version basicaly embed the file into a byte array that's it or I'm missing something?

nightfly• 6 years ago

If you need more check out the rust-embed crate.

euank• 6 years ago

This proposal is more powerful than that one rust macro, but rust's abilities around embedding files are much more powerful than go's approach.

This proposal allows "go build" to embed things in a very specific way, but it's not meant to be extensible.

Rust's 'include_bytes!' macro on the other hand is a macro in the stdlib that can be emulated in an external library. I'm fairly sure every feature of go's embed proposal could be implemented via a rust macro outside the stdlib.

For a specific example, I had a project where I wanted to serve the project's source code as a tarball (to comply with the AGPL license of the project). I was able to write a rust macro that made this as easy as effectively "SOURCE_TARBALL = include_repo!()" [0] to embed the output of 'git archive' in my binary.

Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns.

[0] https://github.com/euank/include-repo

masklinn2• 6 years ago

It seems much more powerful than the Rust version.

So what you're saying is simplicity is only a virtue if it supports the thing you already liked?

leetrout5• 6 years ago

Also note in this proposal:

The path separator is a forward slash, even on Windows systems.

Which is much more ergonomic for developers than rusts treatment of os specific paths.

Arnavion• 6 years ago

Forward slashes work on Windows systems for most [1] paths. You can write `include_bytes!("foo/bar.txt")` in Rust and be very confident it'll compile correctly for most of your users.

[1]: "Most" because they don't work for namespaced paths, by design. But note that the number of times you'll encounter those is limited, the Rust standard library doesn't handle them correctly in general, and a lot of other non-Rust software breaks when given them.

justasitsounds2• 6 years ago

I know it's been mentioned before, but I really feel there is a need to codify an 'internet lore law' about how many comments can appear on a Go themed HN post before someone brings up how superior Rust is

Something like: Pikes law of unfavourable comparison OR HN Golang oxidation rate

dodobirdlord• 6 years ago

Oxidation Factor: Inversely proportional to the amount of time before a comment brings up the superiority of Rust.

Go has a pretty high oxidation factor.

papaf• 6 years ago

I suspect that, programming in Rust is so onerous, that Rust programmers must take lots of breaks and troll Go threads so they feel less bad about themselves.

Or maybe they're just waiting for something to compile?

blub• 6 years ago

The Rust community does have an uptight attitude, as if Rust's success or lack thereof is a matter of life and death. This can be seen in general with languages which haven't achieved wide-spread success yet.

pjmlp9• 6 years ago

If you prefer I can provide examples on how Windows, macOS/iOS, Android resources work for their native language SDKs, or how Java and .NET embedded resources work.

change4422003• 6 years ago

The Rust promotion has unfortunately changed lately. I've noticed too that they not only tell everyone how great Rust is (which is fine) but started bashing other languages (which is sad and not doing any good for the public image of Rust).

Rust is from enthusiasts for enthusiasts. If the Rust community wants it to become more real / main stream, they need to look at how the Go team focuses on supporting devs for getting things done correctly with long term stability.

nicoburns3• 6 years ago

Well it's pretty relevant when considering a new feature for a language to consider prior art is similar languages (the key similarity in this case being the compilation model), no?

sergeykish• 6 years ago

There is certainly a low how Go community rejects proposals, criticism, comparison and a healthy chunk of its users.

Generic compile time evaluation is an interesting feature, better than magic comments in my list. Though in this case it is compiler built-in [1].

[1] https://doc.rust-lang.org/src/core/macros/mod.rs.html#1122-1...

jzoch2• 6 years ago

Your comment itself should be codified. Maybe it’s just another growing language with a large community that itself is dedicated to improving the language. Rust is newish, and changing rapidly so it’s a good candidate for comparisons between modern languages. It helps that it’s designed completely in the open so the sauce can be seen.

Not to pick on you or even the other child comments but the amount of people that complain about a language being compared to a modern equivalent is funny. What about every other thread on HN where the same thing happens? It’s alll turtles just enjoy the discussion

axaxs• 6 years ago

Can someone explain please the point of embedding bin data into a binary, vs just bundling those assets and opening as a file at runtime?

jhardy544• 6 years ago

Convenience. It's easier and more foolproof to give someone one file than it is to give them a directory.

mickaelkerjean• 6 years ago

Yes! I can't count how many people have tried to use my software (https://github.com/mickael-kerjean/filestash), flushing some entire directories with docker volume and creating support tickets and emails as things got broken.

The landscape of tools supposed to solve that problem isn't great. Things are changing way too fast with projects being archived or deprecated with mention to migrate to another solution which after another year is deprecated again. Because of those issue, I've went with go generate which isn't ideal but is at least stable. That proposal would be a game changer and hopefully it will become a reality as Brad is quite a prominent figure in the Golang world.

alain_gilbert• 6 years ago

I personally find it more elegant to distribute a single .exe file which contains all the html/js/css inside it, to customers.

Instead of having a "messy" folder on their computer.

zelly2• 6 years ago

Performance. No additional I/O at startup and guaranteed to be in memory.

pansa22• 6 years ago

Aren’t executable files loaded into memory page-by-page, on demand? There’s no guarantee that their contents are in memory.

jeffbee• 6 years ago

I don't know what platform you had in mind, but this isn't generally true for Linux. No part of a Linux executable is in memory except the page containing the main entry point, initially. You'd have to pre-fault the sections of interest and protect them with mlock to make sure they stay in memory. If you don't their cached pages may be dropped, which is essentially the same thing as swapping.

In short, these embedded assets are no more likely to be in memory than any other file-backed data. If you want to guarantee they are in memory, it is up to you to make that happen.

hombre_fatal• 6 years ago

Consider the reverse question: it's simpler to just ship files in the binary unless you need one of the specific advantages of loose-leaf files (like the user being able to modify them, other processes like Nginx reading them).

If you don't need one of the features of having "moving parts" on the filesystem, then why would you want a dependency on the local filesystem?

mcpherrinm• 6 years ago

Imagine you are shipping a library. Say, one that tells you about timezones.

You want your users to just be able to import that library and use it. The fact that you need a bunch of zone info files should be invisible to them.

Maybe those users are in a sandboxed environment that doesn't allow opening files, or you're in webassembly where files don't exist.

It's much simpler to just bake that data into the library.

DaiPlusPlus• 6 years ago

I’d prefer a library that uses the IANA tzdb directly - so I can download the latest tzdb files for my application and OS without needing to wait for the maintainer of that library to ship an update.

kevin_thibedeau• 6 years ago

You might not have a filesystem.

politician• 6 years ago

Store SQL scripts as embedded resources rather than string constants. When SQL scripts are stored as individual ".sql" files, both version control and syntax highlighting is better.

This works with most editors and doesn't require fancy multi-language syntax highlighting support. When debugging complex scripts that involve many joins or CTEs having good highlighting can prevent a lot of headaches.

dewey• 6 years ago

Sometimes I write tiny services with a web interface that I can just distribute through scp'ing the binary. In this case I can embed the favicon and other tiny files directly into the binary without making the deployment more complicated.

zabil• 6 years ago

I maintain an open source project written in golang that's test runner with an html reporting plugin.

I can use this feature to bundle the report template, css, js, images file with the plugin to make download installation and use of the plugin much easier.

jhardy544• 6 years ago

Can you run one of these files as an executable?

leetrout5• 6 years ago

You mean load the packed payload at runtime and execute it?

stblack• 6 years ago

Interesting.

zelly2• 6 years ago

It looks like this is just for static data files. To run a piece of memory as executable you'd have to explictly mark the page as executable with e.g. mprotect(2) on Linux. I guess there's nothing stopping you from doing that from Go also.

q3k• 6 years ago

You still need an ELF loader, unless you're shipping shellcode. As far as I know, there's currently no great way to load an arbitrary segment of memory as an ELF, without doing some sort of copy (eg. memfd_create, write, exec).

leetrout5• 6 years ago

That’s showing some real support to the community IMO.

There are plenty of purists that argue both for and against the concept but one cannot deny it fits very well in the sales pitch for Go. You can have a simple tool chain to ship a binary for any number of systems and that support for embedded assets would be a first class citizen.

I would also love to see them address more robust plugin options or officially adopt / endorse the pattern HashiCorp uses as they’re doing here with the bindata prior art.

api3• 6 years ago

Go is not a purist language. It's pretty relentlessly designed for maximum productivity with minimum complexity. This sort of feature is right in line with its mission.

Filligree3• 6 years ago

I keep hearing that, but Go is one of the languages I'm least productive in, and I know I'm not alone in that.

You can pick it up in a week, sure. That's not the same as productivity.

apta• 6 years ago

maximum productivity with minimum complexity

Isn't that their unsubstantiated marketing term that keeps getting repeated.

philosopher1234• 6 years ago

What pattern are you referring to do? Haven't heard of this

leetrout5• 6 years ago

https://github.com/hashicorp/go-plugin

If you squint it’s reminiscent of zeromq for IPC :)

You can contrast that with https://golang.org/pkg/plugin/

pfalafel• 6 years ago

+1 for reloadable plugins.

pjmlp9• 6 years ago

Also +1 from my side, I still look forward to have a proper plugin support in place.

lsllc2• 6 years ago

I think it's a well thought out design, particularly the implementation of some common Go interfaces (such as fs.FS) that will allow this to be transparently used with existing Go code.

The only thing I'd want is to allow environment variables in the "go:embed" statements, while my assets might be in the same repo and thus relative, they may also be in a different asset repo (if I'm using git I could use git submodule, but if I'm using something else that might not be possible).

yiyus3• 6 years ago

The best way to do this with the current proposal is to write a simple go package in your assets repository and import it. Something like:

    package assets // import "github.com/me/myproject/assets"
    //go:embed *
    var FS embed.Files

    package mypackage
    import "github.com/me/myproject/assets"
    //use assets.FS

Would that cover your use case?

lsllc2• 6 years ago

Yeah! (as long as the UI/UX folks don't mind a bit of Go code in their repo)

I was kind of hoping for:

  package foo

  //go:embed $ASSETS/*
  var assets embed.Files

But one issue with this is that it might make a dogs breakfast of the filenames within the `assets` object -- it sort of needs something like this:

  //go:embed * from $ASSETS

(and it would not include the $ASSETS path in the object).

atombender• 6 years ago

Video presentation: https://golang.org/s/draft-embed-video

brown9-2• 6 years ago

I hope this catches on as a trend, it’s really nice to see the process being so open to the community to give feedback.

ceocoder• 6 years ago

My former coworker Miki Tebeka wrote nrcs[0], this was specifically for static files - css, JS, png etc - for shipping a self contained web server binary. I still use it now and then because it is well designed and does one thing and that one thing really well.

[0] https://github.com/tebeka/nrsc

joice• 6 years ago

I am using stuffbin to embed files. github.com/knadh/stuffbin

banana_giraffe• 6 years ago

I'm happy to see this, I've always liked using embedded assets in executables to make single file distributions for tools.

That said, I really don't like the further overloading of comments. Comments are for the human, not the compiler.

kstenerud• 6 years ago

It's a consequence of go's simplicity by design. You can't actually destroy natural complexity; only move it around. And if you're unwilling to design a proper container around that slowly increasing complexity, it just squeezes out between the cracks in odd ways.

In go's case, it's manifest primarily through magic (magic files, directories, comments, names, env vars, switches, etc that you just have to know how to invoke). Another warning sign is the need for an external build tool like makefiles in order to build-in-one-command because you need extra steps beyond "go build" (such as "go generate").

Cthulhu_5• 6 years ago

I kinda agree; I get that they don't want to change the language definition itself, and by putting things in comments they don't need to do any change in the compiler, but it makes things feel unsafe, or "stringly typed" if you will.

IIRC Java eventually "solved" it by adding annotations as a language feature. I'm not saying Go should add annotations, but just sharing some language history.

And C / C++ has had compiler directives (e.g. #define) since forever, although there too (I believe) # is "just" a comment.

bww• 6 years ago

I’m aware there is demand for this sort of thing but I’d suggest that almost nobody actually needs it. This strikes me as a feature in search of a problem – especially considering how many of the examples in this document relate to static web assets.

If you’re considering bundling static assets into your binary for a service, you would almost certainly be better served by containerizing your service and copying those assets into the image.

justaguy88• 6 years ago

Sometimes the point is to not use a container

anonfunction2• 6 years ago

I disagree, one amazing aspect of Go is the static binary that can easily be distributed. I embed templates and other text files in my binaries and it makes the installation so simple and without extra dependencies or steps. Even though I'm a huge proponent of containers I don't think it is needed for things like CLI tools.

pjmlp9• 6 years ago

I was quite amazed doing static compilation with Turbo Basic, Turbo Pascal, Turbo C, Turbo C++,....

mjibson• 6 years ago

I wrote one of the listed tools (github.com/mjibson/esc) and am thrilled about this proposal. I think it's great and solves all the problems in a great way.

stevekemp• 6 years ago

I wrote an unlisted tool too, and I am also a fan of this proposal.

The fact that there are so many of these "embed file in binary" tools suggests that it really is a problem that could be usefully solved once, in a consistent and reliable fashion.

BillinghamJ• 6 years ago

It does seem like it'd benefit from a "Must" method though - so you can use it easily to populate variables at init-time

marcrosoft2• 6 years ago

No mention of go-bindata which was one of the originals :(

I think this problem is best solved by a library.

Leave the damn language alone!

cespare• 6 years ago

go-bindata is mentioned first, before the bullet list.

zshev• 6 years ago

It's right there

One of the earliest and most popular was github.com/jteeuwen/go-bindata and its forks

coder543• 6 years ago

No mention of go-bindata which was one of the originals :(

There’s literally an entire section in the appendix dedicated to `go-bindata` and explaining what it generates. It was also first in the list of libraries mentioned.

I also disagree with you entirely. The document does a great job explaining the problems with having this “solved by a library”. I’m excited about this draft design.

uluyol• 6 years ago

This isn't a language change, it's a library + tooling one.

mseepgood• 6 years ago

Leave the damn language alone!

This proposal does not change the language. "//go:" comments are comments because they are not part of the language.

tmaly• 6 years ago

This would be huge. I have had to deal with 3rd party libraries to try to embed assets for a web server.

marcrosoft2• 6 years ago

Which is the way it should be. Not everything needs to be in std lib.

donatj• 6 years ago

While I agree that "not everything needs to be in std lib", having worked with other languages that support embedding out of the box, let me tell you it's handy.

It also seems like an area where it'd be nice for there to be a single uniform way to do it.

pjmlp9• 6 years ago

Sure, but then it isn't guaranteed to work everywhere there is a compiler for the language.

yencabulator3• 6 years ago

The third party solutions have to resort to writing out source files, proper support in tooling can be much more efficient.

ocdtrekkie• 6 years ago

This is funny to me to see on HN tonight because I spent a bunch of time packaging a Go app on Sandstorm this evening, and one of the listed packages apparently was pitching a fit about some static assets not being pulled into the Sandstorm package correctly...

I resolved it, checked HN, and saw this was a peeve they wanted to solve. Hah.

AnonC• 6 years ago

I love this idea of making executables contain assets, and it fits well with the Go ethos of a single (statically built) executable. This makes distribution simpler for applications that do need to deal with assets of different kinds.

A tangential comment: is anyone else concerned a bit that discussions and debates happen on platforms like GitHub and reddit? These two platforms are large enough to be around for quite sometime, but are we making it easier to lose historical context because platform creators/designers are choosing third party platforms that they don’t host or control (or in some instances don’t or can’t pay for)?

aspenmayer6• 6 years ago

It’s concerning that the debate that formed and forms open source is not itself available on open source platforms and/or archived and licensed and available for access in the same manner and license as the code itself. Good spot, quite apropos!

Cthulhu_5• 6 years ago

Yes, but at the same time, it'll reach a much bigger audience than mailing lists (which IIRC is still the official communications platform for Go), which are still painful to work with, even with using Google's own tooling. That said, mailing lists are much more open and easily archived for posterity, moreso than Reddit.

malandrew2• 6 years ago

I would love if gob encoded files were supported natively with some special sugar syntax.

I have a project I work on where I need to package a bunch of data that won’t be available locally. Given that I’m loading data, it seems most optimal to just package it as encoded go data from the get-go.

Right now I’m using go-bindata to embed gob files.

breakingcups2• 6 years ago

Again, more magic comments.

The proposed feature is great, but the unwillingness of the Go team to use a separate, clearly defined project file or at the very least a separate syntax in your code file leads them to stuff every additional feature into comments, a space shared by human notetaking.

Let's have a look: * Build constraints (// +build linux) * Code generation (//go:generate <command> <arguments>) * Cgo, you can even stuff entire C programs in the comments (// #include <stdio.h>) * Cgo flags (// #cgo CFLAGS: -DPNG_DEBUG=1) * and now this, file embedding (//go:embed html/index.html)

Most novices would assume the commented out code does nothing, and rightly so in my opinion. Half of these features aren't even code-file specific but project-wide, making deciding which file to put them in hard and looking them up even harder.

gwd2• 6 years ago

You forgot documentation (which actually annoys me more, because I can't put comments to developers of the function right before the function, because it will be interpreted as documentation to users of the function).

I mean yeah, I think the comment thing is a bit wonky, and it's not the way I'd do it. But once you've got "//go:generate", adding "//go:embed" isn't really any worse.

EDIT: Just noticed this line in the proposal:

Another explicit goal is to avoid a language change. To us, embedding static assets seems like a tooling issue, not a language issue. Avoiding a language change also means we avoid the need to update the many tools that process Go code, among them goimports, gopls, and staticcheck.

So this partly explains the idea of putting things in comments: things classified as "tooling issues" are put there so that "language tools" aren't affected. I do agree it probably would have been better to invent a specific language construct for, "This is a tooling issue, please ignore"; a bit like the # prefix in C.

masklinn2• 6 years ago

But once you've got "//go:generate", adding "//go:embed" isn't really any worse.

There's way more than that: https://golang.org/src/cmd/compile/internal/gc/lex.go#L53

And I would argue that it's bull. Once they realised that they were introducing pragmas / attributes into the language, they should have bitten the bullet and actually introduced a clean way to annotate language items.

The first and second ones are excusable as "we'll only need one" and "well it's not worth the hassle" but at the third one it's not a special case it's a pattern. Especially as the excuse that "other implementations can ignore those" gets less and less true: a compiler which ignores go:embed can not be considered working.

mytailorisrich• 6 years ago

I'm not sure about using this syntax for embedded file, but in general I see their point:

These annotations are not go code but specific to this go source file.

The latter means that embedding them in the go source file makes them easy to find and read =, and avoid multiplying configuration files.

From there the former (adding annotations in comments) is basically the simpler, and perhaps only, way to embed the annotations while make sure the file is still valid go code.

boris• 6 years ago

This is essentially embedding bits and pieces of a makefile into source code as ad hoc comments. Which is what happens when you don't have a proper, general-purpose build system.

throwaway8943457• 6 years ago

I’m sure this comment won’t be popular, but this is making a mountain out of a molehill. I particularly like Go’s build system—specifically that it isn’t some disaster that requires learning a one-off DSL (a la CMake, Gradle, Make, etc) just to enumerate dependencies. It’s clearly not perfect, but it’s always seemed pretty reasonable to use a general purpose build tool (like Bazel or Nix or Make) to call into the Go tool for more complicated builds.

Asooka• 6 years ago

These should all just have been pragmas with their own syntax. I would even accept something as heretical as "@" for the pragma command, e.g.

  @build linux
  @generate <cmd> <args>
  @C #include <stdio.h>
  @cgo CFLAGS: -etc
  @embed path/to/file

Maybe these could even be come kind of code that you can write yourself and define your own pragmas and code transformations by passing some lib folder to the compiler with prebuilt compiler extensions... Ah well.

yiyus3• 6 years ago

Besides the precious four bytes we would save for each pragma, what's the advantage of using "@" instead of "//go:"?

gwd2• 6 years ago

Imagine writing cgo like this:

    #pragma C {
    #include <stdio.h>
    
    int foo(var * bar) {
     ...
    }

Or having documentation look like this:

    // FIXME: This is kind of a hack, and we should
    // get rid of it before v1.0
    #pragma docs {
    The FooBar function will Foo all the Bars passed to it.
    }
    func FooBar(...) {

You could even imagine modifying your editor to do proper syntax highlighting on the C code.

Ah, well -- water under the bridge at this point. I like Go as a language overall. Most of their experiments have worked out pretty well, I think; it's inevitable that some of your experiments don't work out the way you expect, and then you're stuck with them.

skocznymroczny• 6 years ago

D has the import statement for that: https://dlang.org/spec/expression.html#import_expressions

jedisct1• 6 years ago

Something I'd love to see in Zig.

lenkite• 6 years ago

Go needs proper typed annotations like Java

pjmlp9• 6 years ago

Go started being a Java 1.0, instead of learning what a modern language should be like, so every missing feature gets eventually added with a couple of hacks.

Go 15 is going to be a pleasure to use.....

jjice• 6 years ago

Is 15 the version that generics are going to be implemented in? Or are generics still in the proposal stage?

AlexMax• 6 years ago

Regardless of the bike-shedding over the exact syntax, this would be really cool to see. I have no clue how C/C++ made it into 2020 without this language feature - and yes, I know about incbin and xxd, and Go having it ahead of those languages would be ironic given how new and conservative of a language it is.

Go command support for embedded static assets (files)

Related Stories