NetBSD fully reproducible builds

https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds

by conductor • 9 years ago

281 61 9 years ago

blog.netbsd.org

trengrj2• 9 years ago

Woah. This is something that would cause me to switch over from Linux.

Reproducible builds are so important, they prevent a build server or developer laptop from being a single point of failure as a tainted build can now be detected by others.

crest• 9 years ago

In that case have a look at https://wiki.netbsd.org/guide/veriexec/.

swhipple• 9 years ago

I don't quite understand the real world use-case Veriexec is designed to solve.

1) Prevent tampering by making part of the system immutable? The fingerprint isn't necessary; unconditionally prevent modification to the relevant files instead.

2) Prevent tampering by using trusted files? Normally this should be done by having a set of trusted keys, not hardcoded hashes. That way you can still securely upgrade the system.

3) Accessing files from a remote untrusted filesystem? This doesn't seem to work either; see the caveats section in veriexec(9).

Am I missing something here?

fnord1233• 9 years ago

As mentioned in the post, this is built from some tooling developed for Debian's reproducible builds project.

Valerie Young had a good progress report at Linux.conf.au this year:

Reproducible Builds for a Better Future: https://www.youtube.com/watch?v=-nMVfmZL8zk

zoulasc3• 9 years ago

No, the build system is completely unique to NetBSD. To my knowledge no other OS has the ability to cross-build any platform from any POSIX OS. NetBSD can build the whole OS from source tree to distribution media with a single command.

maheart2• 9 years ago

Debian developers have been working on Reproducible Builds for a number of years[0]. Currently the Debian amd64 distribution is ~93.7% reproducible[1]. It's possible that Debian amd64 systems will be fully reproducible in Stretch+1 (~2-3 years).

[0] https://wiki.debian.org/ReproducibleBuilds

[1] https://tests.reproducible-builds.org/debian/reproducible.ht...

ymse4• 9 years ago

GNU Guix is also working towards full reproducibility, and has an up-to-date (but incomplete) list at https://www.gnu.org/software/guix/packages/reproducibility.h...

gwu782• 9 years ago

The Debian wiki page appears to be about reproducible builds for packages.

The NetBSD wiki entry is about progress on reproducible builds for tools, kernel and userland.

As for packages, reproducible builds under pkgsrc is still a WIP.

My use of third party binary packages and pkgsrc is minimal but I am continually building custom kernels and crunched binaries with build.sh and I do make use of the binary userlands from releng.

For me at least, having reproducible builds for these alone is quite useful.

qznc• 9 years ago

93% of the distribution could be enough for almost everybody or for nearly nobody if some core packages are not reproducible yet. Is there any data on that? Maybe using the Debian popcon data?

listic• 9 years ago

How come it's 93.7%? Doesn't look like it on the graph[0].

nickpsecurity10• 9 years ago

They really dont. The compiler can still be malicious since developers might be malicious or just screw up. You're just increasing probability the binary has same vulnerabilities as the source with nothing added during compilation.

You can just as easily just download the source over one or more links to your OS then compile it locally. That was what TCSEC required for trusted distribution. TCSEC was partly designed by guy (Karger) who invented the compiling-compiler attack if you're wondering credibility. If your existing compiler is subverted, then your system is probably subverted with rootkit given that's easier than sniping one program you may never use the intended way with exotic attack. If you cant trust the repos for source or local compiler, then how did you trust your kernel or anything else to begin with?

The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.

ymse4• 9 years ago

The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.

Even if you have all that, how can you be sure no one tampered your system at some point? Reproducibility lets you compare your entire system against known-good states.

trengrj2• 9 years ago

Agree that secure source is required but the compiler trust issues become different with reproducible builds: you only need to trust the initial operating system binary and intermediate source code. The next compiler is built reproducibly from the old compiler.

The maliciousness in the compiler would have to be introduced in source code (which is harder) or somehow already in the initial operating system binary (disregarding bad hardware that is bad for all systems testing builds).

ysleepy• 9 years ago

That is wonderful.

I always dread dealing with build systems, mostly in the C land.

Deterministic behaviour, especially in this rigorous fashion, is probably very helpful for much more cases than just trust.

This looming assumption that make executes pure functions to produce output could actually become true. Now it really suffices if make triggers a target if one of the inputs changed.

ymse4• 9 years ago

Now it really suffices if make triggers a target if one of the inputs changed.

This is exactly how Nix and Guix works. If any inputs change, it will force a rebuild of all dependent packages.

lamby• 9 years ago

helpful for much more cases than just trust.

Indeed, there are a load of QA benefits for reproducible builds. Let alone the CO2 savings that result from cache hits instead of pointlessly rebuilding dependencies.

theamk• 9 years ago

FTR, Debian is like 80% reproducible https://wiki.debian.org/ReproducibleBuilds

"Reproducible builds of Debian as a whole are still not a reality, though individual reproducible builds of packages are possible. So while we are making very good progress, it is a stretch to say that Debian is reproducible."

In my experience, if you have a single home made package in C it is pretty easy to make it reproducible

tachion• 9 years ago

It's funny that to prove your argument you're quoting a paragraph that says exactly otherwise. Not to mention, NetBSD builds are about the OS, that's kernel, tooling, base and so on, and not about packages, as in Debian's case.

bcook• 9 years ago

As another comment notes, Debian packages the kernel, tooling, base, etc.

eriknstr• 9 years ago

If author is here, there's a typo in the text. It says "iso6990" but should say "ISO 9660".

zoulasc3• 9 years ago

thanks!

sly010• 9 years ago

Anyone has a list of what are the typical sources of non-determinism (other than compiler and automake versions)?

emaste4• 9 years ago

There's a fairly comprehensive list at https://reproducible-builds.org/docs/. Debian has a detailed list of reproducibility issues and affected packages at https://tests.reproducible-builds.org/debian/index_issues.ht...

ymse4• 9 years ago

Here's a smørbrødliste:

* Non-isolated build environment. This is just asking for all sorts of trouble (users, hostname, network access, etc).

* File system time stamps.

* Recording times in the build process (although even gcc supports SOURCE_DATE_EPOCH since version 7).

* Usage of CPU-specific instructions, e.g. -march=native to GCC.

pserwylo• 9 years ago

The post itself has a pretty good list of 10 sources of non-determinism:

  1. timestamps
  2. dates/times/authors etc. embedded in source files
  3. timezone sensitive code
  4. directory order/build order
  5. non-sanitized data stored into files
  6. symbolic links/paths
  7. general tool inconsistencies
  8. toolchain
  9. build information / tunables / environment
  10. making sure that the source tree has no local changes

With a bit of info on each.

vacri• 9 years ago

That's exactly what the article is about.

jrcii• 9 years ago

Of all the great OSes, NetBSD seems to get the least fanfare but it's always been my favorite. It's fast, secure, and due to a well reasoned architecture it works on everything. pkgsrc is amazing. They also have a very helpful, friendly community. It especially warms my heart to run `ps -ax` on a new install and see all of about 10 processes. The OS feels lean, neat, and organized, and I feel like I know exactly what is going on, where to find a given file, etc.

These special strengths -- vast hardware compatibility, rump kernels, now full reproducible builds, are all enabled by a greater underlying (and seemingly underrated) technical excellence.

geff82• 9 years ago

Also run "top" on the fresh install to see it uses about 10MB :D . I learned how to use Unix when I installed and used NetBSD for some years, so for me it is "Mother-UNIX". It's a pity that general hardware support is not that great anymore nowadays. But it runs quite well on the Lenovo X240/T440 range of laptops which can still be had new in some places.

hendry• 9 years ago

If you care about getting on the Web safely, Webconverger has reproducible builds at https://build.webconverger.com/

dolzenko• 9 years ago

I know, it's unrelated, but its 2017 and

Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system

seems just weird.

gumby• 9 years ago

At Cygnus we had a customer from the telecom industry. They had SLAs with their own customers that included terms like "no more than 10 minutes of downtime per decade". They paid a LOT of money to have one, consistent release (no upgrades, only bug fixes); when they reported a big and got a fix they would diff the new binary and required that every change could be traced solely to the patch issued a nothing else.

Satisfying this made GCC a lot better.

ksec• 9 years ago

Forgive my Ignorance,

OpenBSD - Absolute Security FreeBSD - General purpose

What is NetBSD aiming at?

smhenderson• 9 years ago

Maximum portability... "Of course it runs NetBSD"

From their website: NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license.

NetBSD fully reproducible builds

Related Stories