Evaluating Bazel for building Firefox

https://blog.mozilla.org/nfroyd/2019/10/28/evaluating-bazel-for-building-firefox-part-1/

by dochtman • 7 years ago

194 166 7 years ago

blog.mozilla.org

m0zg2• 7 years ago

Bazel is excellent on Linux and macOS, but it's pretty bad on Windows. I'm not sure what (if anything) they're going to do about that. Maybe they'll fix it on Windows though. That'd be pretty cool.

Onanymous• 7 years ago

Never tried it on Windows. For me it was pretty bad on Linux and macOS.

NickGerleman• 7 years ago

Microsoft internally uses something pretty similar to Bazel. I'm not familiar enough with the two to fully understand motivations for the divergence. It supposedly initially ran poorly on Mac, just due to differences in what is cheap on different platforms. I wonder what kind of inherent performance differences you would find in something "Windows first" vs "Linux/MacOS first" https://github.com/microsoft/BuildXL

m0zg2• 7 years ago

I think there's nothing insurmountable in getting Bazel to work well on Windows. It's just a chicken and egg problem: few people use it there, so it's not up to par, so few people use it there because it's not up to par.

EricBurnett3• 7 years ago

The differences between bazel and BuildXL are less about platform and more about philosophy. Bazel is about opinionated builds - declare everything, and you get fast correct builds. Hard to adopt, but super powerful. BuildXL is more about meeting codebases where they are - take an existing msbuild|cmake|dscript build with its under-specified dependencies, and figure out how to parallelize it safely. This delivers quick wins, but comes with real costs as well (notably: no caching; labour-intensive magic to guess an initial seed of deps, with a performance cliff when something changes).

(Disclaimer: Googler, work closely with the bazel team. I like bazel; I like BuildXL for expanding the space of build ideas, but have long-term concerns it settles in a local maxima that'll be hard to get out of).

txtsd• 7 years ago

Is meson not good?

adgasf11• 7 years ago

Meson does not have the same feature set as Bazel. Most notably sandboxing and remote execution.

qznc3• 7 years ago

Other build systems (CMake, Meson, etc) should be able to gain the sandboxing feature. The remote execution feels more fundamental to me and not so easy to replicate.

adgasf11• 7 years ago

Before it comes up: Bazel ships with its own copy of the JVM so you do not need to install Java to use it.

account42• 7 years ago

Is this supposed to be a good thing?

danpalmer6• 7 years ago

Is it a bad thing?

That might make the installed size large, but if it's isolated to Bazel, managed by Bazel, updated by Bazel, etc, then why does it matter whether it's a JVM, a Python binary, a JS VM, or anything else?

PedroBatista• 7 years ago

Probably better than Nodejs Gulp Yo man ugabuga npm here's 3.4 million files for you to download or Python is it 2.7 or 3.1? Pip too and Ansible let's make that terminal a disco night of text! or some Go "Works great for my static file blog" build tool.

yk• 7 years ago

At some point someone will just bundle all the software into a single docker container, and then we are finally done with the entire installation quagmire.

qhwudbebd3• 7 years ago

...if you're happy running random third party binaries instead of building from source.

adgasf11• 7 years ago

Do you review all of the source-code that you run? If you trust the source-code due to its origin, why not trust a binary?

rjsw2• 7 years ago

Firefox will currently build on more platforms than Bazel does. Getting fed up of the endless list of new build systems.

_pmf_• 7 years ago

If making Bazel work on Windows is a side effect of Firefox using it, what's not to like about it.

adgasf11• 7 years ago

Bazel already works on Windows.

qhwudbebd3• 7 years ago

I agree, and inflicting Bazel's horrible Java dependency on Firefox would be a tragedy.

mda• 7 years ago

How is it different than depending on e.g. Python?

lima8• 7 years ago

What's wrong with the Java dependency? There's even a single-binary Bazel executable[1] that brings its own embedded OpenJDK. My company is using Bazel and the Java dependency has never been an issue.

Even if you write custom rules, you use the Python-like Skylark language and never interact with Java code.

[1] https://github.com/bazelbuild/bazel/releases (the single-binary distribution is bazel-1.1.0-linux-x86_64, it's only 40MB).

jupp0r• 7 years ago

What platforms are you talking about specifically? Bazel builds for Windows, Linux, MacOS, iOS, Android.

qznc3• 7 years ago

Just guessing: BSDs, ARM (eg Raspberry Pi), PPC, Haiku, Minix

hartror• 7 years ago

TIL Thunderbird is still a thing.

pja• 7 years ago

It’s still one of the best IMAP clients IMO.

CrLf• 7 years ago

That says a lot about the state of IMAP clients.

gspr• 7 years ago

mbsync/isync changed my email life.

http://isync.sourceforge.net/mbsync.html

boris5• 7 years ago

I wonder if they considered build2[1]? It seems to tick most of their boxes. In particular, it has first-class Windows support and, in the upcoming release, even native Clang-targeting-MSVC support (not the clang-cl wrapper)[2]. While some of the features they are looking for (like faster build times which I assume means distributed compilation and caching) are not there yet, we are working on them (and would welcome help)[3].

[1] https://build2.org

[2] https://stage.build2.org/?builds=&cf=windows_10-clang*

[3] https://github.com/build2/README/blob/master/roadmap.md

dochtman• 7 years ago

One of the important parts in the post seems to be that it is not just a C++ build system, there are many other tasks executed through the build system. In that sense, it seems like build2 isn't a great fit.

boris5• 7 years ago

On the contrary, build2 is a language-agnostic, general-purpose build system. There is the bash module[1], for example. And now that we have support for dynamically buildable/loadable build system modules, we can develop support for more languages, code generators, etc., without bundling it with the build system core. I myself am itching to take a stab at Rust. Hopefully will find time after the release.

[1] https://build2.org/build2/doc/build2-build-system-manual.xht...

jyounker• 7 years ago

I think part of the reason is battle-testing. Blaze (the Google internal version of Bazel) has been used by 20k people per day for well over five years to continually build a single mammoth shared C++, Java, Python, Shell (plus others) code base complete with a globally distributed cache system and thousands if not tens of thousands of build machines.

NieDzejkob• 7 years ago

I'm surprised to see no mention of Tup [1] in the discussion. Verifying the dependency graph for maximizing parallelism is its major feature.

[1] http://gittup.org/tup/examples.html

theamk5• 7 years ago

It seems dead? Latest commit was over a year ago, and there are over 100 open issues seemingly with no activity.

swsieber• 7 years ago

Does Tup do more than C? Firefox needs to build some Rust things too.

andrewflnr• 7 years ago

It's a very general tool. IIRC from playing with it, it doesn't give a fig about whether it's running a compiler at all. I think I was using it for a toy compiler+nasm, actually...

staktrace• 7 years ago

Firefox currently can be built with tup, although it's still experimental and not used in production.

IshKebab2• 7 years ago

What is different about Tup? It's home page contains a lot of text but zero information.

beagle3• 7 years ago

Tup runs what is effectively a bash build script, but does so under ptrace (or equivalent) so it knows about every file that has been checked (and IIRC even those that were missing). So you basically write a "from scratch" build script and that's it.

To rebuild, it will rerun the script, except that it knows which file it reads (and which mustn't exist), so if nothing is changed, it can skip that stage. There is no inherent difference between the compiler binary (say, /usr/bin/gcc), the system include (/usr/include/stdio.h), and the project source.

It makes for a super simple build script, which, after one successful build, gives tup enough information about dependencies and parallelization opportunities, while at the same time also tracks toolchains better than you can manually (does /usr/bin/gcc shell to /usr/bin/x86-gcc or /usr/bin/amd64-gcc ? did one of them change but not the other? was it the one we need? tup knows)

Unfortunately, it is nontrivial; it can uses ptrace or fuse to trace file access on Linux, IIRC has some other hack on Windows - and little to no support for other systems. Also, the overhead is in the single digit % last time I checked.

adgasf11• 7 years ago

Windows support seems to be very lacking.

dblohm7• 7 years ago

That would be pretty easy to fix, TBH. Tup's author works for Mozilla, and Mozilla has the expertise to add that functionality. As always, it's a matter of will and resourcing.

steveklabnik• 7 years ago

The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X…but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit to make itself. Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work. After several rounds of these kind of fixes, you wind up with Makefiles that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel. And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be…challenging.

We had a particularly "fun" instance of this sort of Makefile weirdness back when we used Make for Rust... https://github.com/rust-lang/rust/blob/efec34a95a76212b2324d...

(Discussed on HN: https://news.ycombinator.com/item?id=13655081)

ndesaulniers• 7 years ago

Of all the build systems I've used, Bazel/Blaze has sucked the least.

munificent• 7 years ago

Hermetic declarative build systems like Bazel feel exactly like sound statically typed languages to me. It's really hard to get all of your dependencies precisely specified declaratively. The urge to hack in a little imperative code or jam in a build step that you just "know" needs to happen in a certain order is always there and it can be very difficult to satisfy the taskmaster that is the build system. When you have weird cases like a script that produces a variable set of output files based on the content of some input file, it's tough to get that working in the build system.

But... if you climb that cliff, there are many rewards at the top. Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state), you can automatically parallelize (because the build system can tell declaratively which tasks do no depend on each other) and you can ask it interesting queries about the dependency graph. That sounds an awful lot like the runtime safety, performance, and static analysis static types give you.

The challenge is that for small programs, dynamic types are just easier to get going with. And every build script starts pretty small. So you have a plethora of simple, hackable build systems like make that work fine at first but then get more and more painful as the program grows. With software, there is at least some customer-visible pain that can enable you to justify a rewrite in a statically-typed language. But build infrastructure is always a hidden cost so it's really hard to get the resources to migrate to a different build language.

Maybe the lesson here is to bite the bullet and start with a hermetic build system at first.

adgasf11• 7 years ago

Except the Bazel syntax is really good. Compiling a few C++ files into a binary is just:

    cc_binary(
      name = "app",
      srcs = glob([ "*.hpp", "*.cpp" ]),
    )

That's more concise than most build systems AND it's cross-platform.

joshuamorton11• 7 years ago

Although I'd strongly suggest not doing that bazel works better if you have 1 library per header (or source file for non c langs). It helps to have tools to manage deps for your though.

SolarNet14• 7 years ago

Yes but there is a hundred megabytes of system behind that. Tens of thousands of lines of starlark code, and tens (maybe hundreds) of thousands of lines of java code besides.

A lot of work goes into those 4 lines. This is evidenced by the simple question of, how do you set up a custom compiler for that code properly? The answer being a couple hundred lines of starlark code, for what in other build systems is effectively `CC=clang` (to be fair bazel allows this but does not support or encourage it, especially for anything besides toy builds).

I like Bazel (a lot!) but you are oversimplifying it here.

flohofwoe• 7 years ago

Simple things like this are simple in all C/C++ build systems, the question is how the complex things are handled (multi-platform support, detecting and selecting different versions of the same compiler, cross-compiling, IDE support, multi-stage builds where source files are generated in the build which need to be inputs to later build stages, custom build steps, or bootstrapping tools which need to be used as compilers or code-generators for later build stages, etc etc etc...)

mzs4• 7 years ago

I guess but this is also portable and more people know how to extend it:

    $ cat Makefile 
    SRCS = foo.cpp bar.cpp baz.cpp
    OBJS = $(SRCS:.cpp=.o)
    PROG = app
    EXT  =
    
    .PHONEY = all clean
    
    all : $(PROG)
    
    LDLIBS = $(OBJS)
    
    $(PROG) : $(PROG:$(EXT)=.cc) $(OBJS)
    
    clean :
            rm -f $(PROG) $(OBJS)
    $ make SRCS="`echo *.cpp`" clean all
    rm -f app a.o b.o c.o
    c++ -O2 -pipe -c a.cpp -o a.o
    c++ -O2 -pipe -c b.cpp -o b.o
    c++ -O2 -pipe -c c.cpp -o c.o
    c++ -O2 -pipe  app.cc a.o b.o c.o -o app
    $

bsder3• 7 years ago

The syntax is almost never the issue.

The issue is that declarative build systems ALWAYS need an escape hatch for the exceptions--and the exceptions grow over time.

We've been here before. It was called Make. And then it was called Ant. And then Maven. And Google still built their own system. And they will build another again.

Nobody ever learns the lesson that a build system is a program--period. Sorry to burst your bubble: there will be circular dependencies--you will have to deal with it. There will be two different C compilers on two different architectures. You will have to deal with it. Libraries will be in the wrong place--you will have to deal with it. These libraries come from the system but these libraries come from an artifact--you will have to deal with it.

The only way to deal with it is to have a genuine programming language underneath.

lima8• 7 years ago

Like the article notes, you can reuse intermediate build artifacts between developers (because it's not possible for those artifacts to accidentally depend on developer-specific state)

Note that Bazel builds still depend on the ambient system environment, like the compiler toolchain and system libraries, and it's possible to poison a shared build cache.

Shared caches are typically used with remote execution and CI servers, where the build environment is fully controlled.

adgasf11• 7 years ago

Doesn't Bazel hash the compiler and env variables to prevent this?

The only way to poison the cache then is to break things at an OS level, which is highly unlikely.

jcranmer2• 7 years ago

The problem with fully declarative systems is that, if you need something that is beyond the scope of the declarative semantics, you are screwed.

Build systems have the annoying property that they are ultimately highly declarative, but they need some pretty extreme scope to handle edge cases. Any C/C++ project is going to boil down to "compile these N source files, and link them into a library/executable", but the necessary build flags is going to come from several locations, there are multiple variations of build flags and compilers that need to be used at the same time, and there is going to be project-custom code to generate source files that involves potentially building the tools to generate the files as part of the build system. And mixed-language projects add more layers of pain.

theamk5• 7 years ago

bazel in particular is pretty nice with that. There are multiple extension points -- there is embedded Python-like language, you can wrap arbitrary commands, and if this is not enough, you can dynamically general build files as well.

bennofs2• 7 years ago

Hermetic build systems make a lot of sense for companies, where you often want to control your whole stack and might even use a monorepo with all external dependencies vendored.

However, there are at least two cases where you do not want real hermetic builds:

- in the case of open source software that needs to be packaged for distributions, vendoring all the dependencies is bad. As the name already says, distributions remix software components and thus might want to compile your software against a different version of a library than the particular version you pick. - for libraries, consumers use your library just as a component and might want to combine it with a specific version of other libraries, so vendoring also sucks.

Unfortunately, many declarative build systems make it hard to separate the "building a component" part from the "composing a product" part. I could find no way in Bazel to depend on system software (using for example pkgconfig).

In my opinion, build systems for open source software (as I mentioned, for companies Bazel makes sense since you often want to vendor all your deps anyway) need to have three aspects:

- build rules for just for this component - dependencies on other components/libraries - a method to lock down components/libraries to a specific version, creating an offical "product" with an official supported combination of all dependencies.

This still makes it possible for distributions to create their own "product", without having to rewrite most parts of the build system. At the same time, developers can work with the official library versions (for compat, maybe you should even have multiple "sets" of dependencies), so getting started is easy and you don't need to install lots of dependencies first before you can hack on the project.

Often, build systems do things that are nice for onboarding (like "build systems" that pull git repos of dependencies during build, in order to have a single command to build the product from scratch after checkout). But those things directly conflict with the needs of distributions. We need to separate the "instructions to build, supplying all deps externally explictly" (in the context of building a package for a distribution) from the "just build it from the command line on a dev machine" (here, dependencies should be implictly fetched automatically, to give a nice user experience).

codesuki• 7 years ago

You could probably write a rule similar to go_repository[1] that would invoke pkgconfig to find the location of local libs instead of downloading a lib.

[1] https://github.com/bazelbuild/bazel-gazelle/blob/master/inte...

jayd16• 7 years ago

I'm trying to wrap my head around whether Bazel is worth looking into for the kind of work loads I typically work in. There doesn't seem like that much benefit if you have single language code base and especially if you already have a single build.

I suppose I see the benefit of this setup if you keep many libraries and want to pin to the latest instead of, an explicit lib version. If you're in a multi-repo system you're probably not doing this but I can see how Bazel could actually make this possible.

Can someone shed some light on the CI story? Can Bazel do code deployment as well? Is that a smart thing to do with Bazel?

lima8• 7 years ago

Bazel is worth looking into as soon as you have dependency graphs that your language's native build system can't deal with efficiently, or you have multiple languages that have dependencies on each other's artifacts.

A pure Go codebase? Not worth it.

Personally, I use Bazel even for small projects as soon as they involve things like generated code or gRPC.

Yes, there are multiple rulesets for deployments, like rules_k8s[1] and rules_docker[2]. Of course, you can easily build your own custom deployment pipeline.

[1] https://github.com/bazelbuild/rules_k8s

[2] https://github.com/bazelbuild/rules_docker

xvilka• 7 years ago

The problem of Bazel is the JVM requirement. Why not to use something native and more lightweight?

pjmlp11• 7 years ago

You can compile Java to native code.

And in the case you don't want to, it is hardly any different from shipping a bunch of .so/.dll with the application.

Evaluating Bazel for building Firefox

Related Stories