OpenBSD is so fast, I had to modify the program slightly to measure itself

https://flak.tedunangst.com/post/is-OpenBSD-10x-faster-than-Linux

by Bogdanp • 11 months ago

223 165 11 months ago

flak.tedunangst.com

rurban• 11 months ago

No, generally Linux is at least 3x faster than OpenBSD, because they don't care much for optimizations.

shmerl• 11 months ago

It's talking about a specific case: https://infosec.exchange/@jann/115022521451508325

kleiba• 11 months ago

Pretty broad statement(s).

farhaven• 11 months ago

OpenBSD is a lot faster in some specialized areas though. Random number generation from `/dev/urandom`, for example. When I was at university (in 2010 or so), it was faster to read `/dev/urandom` on my OpenBSD laptop and pipe it over ethernet to a friend's Linux laptop than running `cat /dev/urandom > /dev/sda` directly on his.

Not by just a bit, but it was a difference between 10MB/s and 100MB/s.

cogman10• 11 months ago

/dev/urandom isn't a great test, IMO, simply because there are reasonable tradeoffs in security v speed.

For all I know BSD could be doing 31*last or something similar.

The algorithm is also free to change.

sillystuff• 11 months ago

I think you meant to say /dev/random, not /dev/urandom.

/dev/random, on linux used to stall waiting for entropy from sources of randomness like network jitter, mouse movement, keyboard typing. /dev/urandom has always been fast on Linux.

Today, linux /dev/random mainly uses an RNG after initial seeding. The BSDs always did this. On my laptop, I get over 500MB/s (kernel 6.12) .

IIRC, on modern linux kernels, /dev/urandom is now just an alias to /dev/random for backward compatibility.

somat3• 11 months ago

At one point probably 10 years ago I had linux vm guests refuse to generate gpg keys, gpg insisted it needed the stupid blocking random device, and because the vm guest was not getting any "entropy" the process went nowhere. As an openbsd user naturally I was disgusted, there are many sane solutions to this problem, but I used none of them. Instead I found rngd a service to accept "entropy" from a network source and blasted it with the /dev/random from a fresh obsd guest on the same vm host. Mainly out of spite. "look here you little shit, this is how you generate random numbers"

sgarland2• 11 months ago

OpenBSD switched their PRNG to arc4random in 2012 (and then ChaCha20 in 2014); depending on how accurate your time estimate is, that could well have been the cause. Linux switched to ChaCha20 in 2016.

Related, I stumbled down a rabbit hole of PRNGs last year when I discovered [0] that my Mac was way faster at generating UUIDs than my Linux server, even taking architecture and clock speed into account. Turns out glibc didn’t get arc4random until 2.36, and the version of Debian I had at the time didn’t have 2.36. In contrast, since MacOS is BSD-based, it’s had it for quite some time.

[0] https://gist.github.com/stephanGarland/f6b7a13585c0caf9eb64b...

nicce• 11 months ago

Rather they care about the security. The same mitigations in Linux would likely make it even slower.

jedberg2• 11 months ago

"It depends"

Faster is all relative. What are you doing? Is it networking? Then BSD is probably faster than Linux. Is it something Linux is optimized for? Then probably Linux.

A general benchmark? Who knows, but does it really matter?

At the end of the day, you should benchmark your own workload, but also it's important to realize that in this day and age, it's almost never the OS that is the bottleneck. It's almost always a remote network call.

loeg15• 11 months ago

The article is marginally more interesting than the headline, and it's not very long. Go ahead and read it.

jedberg2• 11 months ago

I did read it. It talked about one specific use case, which isn't very interesting in and of itself.

the_plus_one• 11 months ago

Is it just me, or is there some kind of asteroid game shooting bullets at my cursor while I try to read this [1]? I hate to sound mean, but it's a bit distracting. I guess it's my fault for having JavaScript enabled.

[1] https://flak.tedunangst.com/script.js

nomel2• 11 months ago

And, if it hits, your cursor disappears! I wish there was some explosion.

bigstrat2003• 11 months ago

No, it's the website's fault for doing stupid cutesy stuff that makes the page harder to read. Don't victim-blame yourself here.

q3k• 11 months ago

god forbid people have fun on the internet

stavros• 11 months ago

I really don't understand this "everything must be 100% serious all the time". Why is it stupid?

nomel2• 11 months ago

You should ask for a refund!

lilyball• 11 months ago

It's extremely distracting. I'm not normally one to have issues that require reduced motion, but the asteroids are almost distracting enough on their own, and the fact that it causes my cursor to vanish is a real accessibility issue. I didn't actually realize just how much I use my mouse cursor when reading stuff until now, partly as a fidget, partly as a controllable visual anchor as my eyes scan the page.

joemi• 11 months ago

I actually can't read things on that site at all. I move my mouse around while reading, not necessarily near the words I'm currently reading, so when my mouse disappears it's haltingly distracting. In addition to that, the way the "game" visually interferes with the text that I'm trying to read makes it incredibly hard to focus on reading. These two things combine to make this site literally unreadable for me.

I don't get why people keep posting and upvoting articles from this user-hostile site.

tptacek4• 11 months ago

Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.

https://news.ycombinator.com/newsguidelines.html

Because way more people have opinions about e.g. asteroid game scripts on web pages than have opinions on RCUs, these subthreads spread like kudzu.

ummonk• 11 months ago

The described behavior sounds like significantly worse than tangential annoyance, and isn’t really a common occurrence even on modern user-hostile websites.

binarycrusader• 11 months ago

I found it exceedingly difficult to read, so I ended up applying these ublock filter rules so I could read it:

  flak.tedunangst.com##.bl.shooter
  flak.tedunangst.com##.br.shooter
  flak.tedunangst.com##div.bullet

jackyard862• 11 months ago

It still hides your cursor though.

Jtsummers• 11 months ago

He used to have a loading screen that did nothing if you have JS enabled in your browser, but no loading screen (which, again, did nothing) if you had JS disabled. I'm pretty sure it's meant to deliberately annoy, though this one is less annoying than the loading screen was.

ccapitalK• 11 months ago

It's sites like this that make me extremely grateful for firefox's reader mode.

snvzz• 11 months ago

clean mirror:

https://archive.is/wMECM

haunter2• 11 months ago

In my mind faster = the same game with the same graphics settings have more FPS

(I don’t even know you could actually start mainstream games on BSD or not)

nine_k• 11 months ago

Isn't it mostly limited by GPU hardware, and by binary blobs that are largely independent from the host platform?

haunter2• 11 months ago

Games run better under Linux (even if they are not-native but with Proton/Wine) than on Windows 11 so the platform does matter

https://news.ycombinator.com/item?id=44381144

dang2• 11 months ago

The article title is too baity to fit HN's guidelines (https://news.ycombinator.com/newsguidelines.html) so I replaced it with a phrase from the article that's hopefully just baity enough.

JdeBP3• 11 months ago

I was just about to point Ian Betteridge at the original title. (-:

dang2• 11 months ago

Just don't joke that he has retired! https://news.ycombinator.com/item?id=10393754

cout• 11 months ago

Interesting. I tried to follow the discussion in the linked thread, and the only takeaway I got was "something to do with RCU". What id the simplified explanation?

viraptor• 11 months ago

When 2 threads are allocating sockets sequentially, they fight for the locks. If you preallocate a bigger table by creating fd 666 first, the lock contention goes away.

JdeBP3• 11 months ago

It's something that has always been interesting about Windows NT, which has a multi-level object handle table, and does not have the rule about re-using the lowest numbered available table index. There's scope for reducing contention amongst threads in such an architecture.

Although:

1. back in application-mode code the language runtime libraries make things look like a POSIX API and maintain their own table mapping object handles to POSIX-like file descriptors, where there is the old contention over the lowest free entries; and

1. in practice the object handle table seems to mostly append, so multiple object-opening threads all contend over the end of the table.

saagarjha4• 11 months ago

RCU is very explicitly a lockless synchronization strategy.

loeg15• 11 months ago

This isn't why it's slow. 2x256 just isn't a lot of locks in CPU time.

bobby_big_balls• 11 months ago

In Linux, the file descriptor table (fdtable) of a process starts with a minimum of 256 slots. Two threads creating 256 sockets each, which uses 512 fds on top of the three already present (for stdin, stdout and stderr), requires that the fdtable be expanded about halfway through when the capacity is doubled from 256 to 512, and again near the end when resizing from 512 to 1024.

This is done by expand_fdtable() in the kernel. It contains the following code:

        if (atomic_read(&files->count) > 1)
          synchronize_rcu();

The field files->count is a reference counter. As there are two threads, which share a set of open files between them, the value of this is 2, meaning that synchronize_rcu() is called here during fdtable expansion. This waits until a full RCU grace period has elapsed, causing a delay in acquiring a new fd for the socket currently being created.

If the fdtable is expanded prior to creating a new thread, as the test program optionally will do by calling dup(0, 666) if supplied a command line argument, this avoids the synchronize_rcu() call because at this point files->count == 1. Therefore, if this is done, there will be no delay later on when creating all the sockets as the fdtable will have sufficient capacity.

By contrast, the OpenBSD kernel doesn't have anything like RCU and just uses a rwlock when the file descriptor table of the process is being modified, avoiding the long delay during expansion that may be observed in Linux.

tptacek4• 11 months ago

RCUs are super interesting; here's (I think I've got the right link) a good talk on how they work and why they work that way:

https://www.youtube.com/watch?v=9rNVyyPjoC4

dekhn• 11 months ago

Thanks for the explanation. I confirmed the performance timing different by enabling the dup call.

I guess my question is why would synchronize_rcu take many milliseconds (20+) to run. I would expect that to be in the very low milliseconds or less.

loeg15• 11 months ago

This is done by expand_fdtable() in the kernel. It contains the following code:

Context: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

neerajsi• 11 months ago

Reading this code for the first time, this seems to be a consequence of the separation between allocating and fd and "installing" a pointer to a file there. Allocating the fd already needs to acquire a lock. So if the install happens together with allocation, there wouldn't be a need to use synchronize_rcu to kick out other threads. The lock would do that.

agambrahma• 11 months ago

So ... essentially testing file descriptor allocation overhead

loeg15• 11 months ago

Sort of. Fd table size, which is slightly different than fds (once you reach the ulimit, there's no need to resize it larger); and only in multithreaded programs.

asveikau• 11 months ago

My guess is it has something to do with the file descriptor table having a lot of empty entries (the dup2(0, 666) line.)

Now time to read the actual linked discussion.

wahern3• 11 months ago

I think dup2 is the hint, but in the example case the dup2 path isn't invoked--it's conditioned on passing an argument, but the test runs are just `./a.out`. IIUC, the issue is growing the file descriptor table. The dup2 is a workaround that preallocates a larger table (666 > 256 * 2)[1], to avoid the pathological case when a multi-threaded process grows the table. From the linked infosec.exchange discussion it seems the RCU-based approach Linux is using can result in some significant latency, resulting in much worse performance in simple cases like this compared to a simple mutex[2].

[1] Off-by-one. To be more precise, the state established by the dup2 is (667 > 256 * 2), or rather (667 > 3 + 256 * 2).

[2] Presumably what OpenBSD is using. I'd be surprised if they've already imported and adopted FreeBSD's approach mentioned in the linked discussion, notwithstanding that OpenBSD has been on an MP scalability tear the past few years.

themafia2• 11 months ago

Yea, well, I had to modify your website to make it readable. Why do people do this?

opan• 11 months ago

It looks good to me on mobile. High contrast, lines aren't too long. What issue did you have?

themafia2• 11 months ago

There are two widgets in the lower left and lower right corners of the page which constantly shoot little bullets all over the screen chasing your mouse pointer. In the steady state on my screen there are about 20 sprites constantly moving across the page. There was no obvious way to disable them other than inspecting the page and deleting the widgets from the page.

If you want me to read your site, and you want to put detailed technical information on it, please don't add things like this without an off switch.

sweetjuly• 11 months ago

A better title: a pathological test program meant for Linux does not trigger pathological behavior on OpenBSD

apgwoz• 11 months ago

Surely you must be new to tedu posts…

ameliaquining2• 11 months ago

Still worth avoiding having the HN thread be about whether OpenBSD is in general faster than Linux. This is a thing I've seen a bunch of times recently, where someone gives an attention-grabbing headline to a post that's actually about a narrower and more interesting technical topic, but then in the comments everyone ignores the content and argues about the headline.

Const-me2• 11 months ago

Not sure if that’s relevant, but when I do micro-benchmarks like that measuring time intervals way smaller than 1 second, I use __rdtsc() compiler intrinsic instead of standard library functions.

On all modern processors, that instruction measures wallclock time with a counter which increments at the base frequency of the CPU unaffected by dynamic frequency scaling.

Apart from the great resolution, that time measuring method has an upside of being very cheap, couple orders of magnitude faster than an OS kernel call.

sa46• 11 months ago

Isn't gettimeofday implemented with vDSO to avoid kernel context switching (and therefore, most of the overhead)?

My understanding is that using tsc directly is tricky. The rate might not be constant, and the rate differs across cores. [1]

[1] https://www.pingcap.com/blog/how-we-trace-a-kv-database-with...

Dylan16807• 11 months ago

If you have something newer than a pentium 4 the rate will be constant.

I'm not sure of the details for when cores end up with different numbers.

quotemstr2• 11 months ago

Wizardly workarounds for broken APIs persist long after those APIs are fixed. People still avoid things like flock(2) because at one time NFS didn't handle file locking well. CLOCK_MONOTONIC_RAW is fine these days with the vDSO.

toast0• 11 months ago

I think most current systems have invariant tsc, I skimmed your article and was surprised to see an offset (but not totally shocked), but the rate looked the same.

You could cpu pin the thread that's reading the tsc, except you can't pin threads in OpenBSD :p

triknomeister• 11 months ago

TSC is about cycles consumed by a core. Not about actual time. And so for microbenchmarking, it actually makes sense, because you are often much more interested in CPU benchmarks than network benchmarks in microbenchmarking.

ainiriand• 11 months ago

You have to benchmark tsc against a fixed CPU speed, say 1000Mhz, then you have a reliable comparison.

tonyarkles• 11 months ago

It was a while ago (2009-10ish) but I ran into an exceptionally interesting performance issue that was partly identified with RDTSC. For a course project in grad school I was measuring the effects of the Python GIL when running multi-threaded Python code on multi-core processors. I expected the overhead/lock contention to get worse as I added threads/cores but the performance fell off a cliff in a way that I hadn't expected. Great outcome for a course project, it made the presentation way more interesting.

The issue ended up being that my multi-threaded code when running on a single core pinned that core at 100% CPU usage, as expected, but when running it across 4 cores it was running 4 cores at 25% usage each. This resulted in the clock governor turning down the frequency on the cores from ~2GHz to 900MHz and causing the execution speed to drop even worse than just the expected lock contention. It was a fun mystery to dig into for a while.

mananaysiempre2• 11 months ago

This does not account for frequency scaling on laptops, context switches, core migrations, time spent in syscalls (if you don’t want to count it), etc. On Linux, you can get the kernel to expose the real (non-“reference”) cycle counter for you to access with __rdpmc() (no syscall needed) and put the corrective offset in an memory-mapped page. See the example code under cap_user_rdpmc on the manpage for perf_event_open() [1] and NOTE WELL the -1 in rdpmc(idx-1) there (I definitely did not waste an hour on that).

If you want that on Windows, well, it’s possible, but you’re going to have to do it asynchronously from a different thread and also compute the offsets your own damn self[2].

Alternatively, on AMD processors only, starting with Zen 2, you can get the real cycle count with __aperf() or __rdpru(__RDPRU_APERF) or manual inline assembly depending on your compiler. (The official AMD docs will admonish you not to assign meaning to anything but the fraction APERF / MPERF in one place, but the conjunction of what they tell you in other places implies that MPERF must be the reference cycle count and APERF must be the real cycle count.) This is definitely less of a hassle, but in my experience the cap_user_rdpmc method on Linux is much less noisy.

[1] https://man7.org/linux/man-pages/man2/perf_event_open.2.html

[2] https://www.computerenhance.com/p/halloween-spooktacular-day...

Const-me2• 11 months ago

does not account for frequency scaling on laptops

Are you sure about that?

time spent in syscalls (if you don’t want to count it)

The time spent in syscalls was the main objective the OP was measuring.

cycle counter

While technically interesting, most of the time I do my micro-benchmark I only care about wallclock time. Contradictory to what you see in search engines and ChatGPT, RDTSC instruction is not a cycle counter, it’s a high resolution wallclock timer. That instruction was counting CPU cycles like 20 years ago, doesn’t do that anymore.

MortyWaves• 11 months ago

Fascinating how each “standard” or intrinsic that gets added actually totally fails to give you the real numbers promised.

junon3• 11 months ago

rdtsc isn't available on all platforms, for what it's worth. It's often disabled as there's a CPU flag to allow its use in user space, and it's well know to not be so accurate.

loeg15• 11 months ago

What platforms disable rdtsc for userspace? What accuracy issues do you think it has?

mrlongroots• 11 months ago

They could've just used `clock_gettime(CLOCK_MONOTONIC)`

oguz-ismail• 11 months ago

I use __rdtsc() compiler intrinsic

What do you do on ARM?

Narishma• 11 months ago

Use a Raspberry Pi or something.

loeg15• 11 months ago

https://github.com/facebook/folly/blob/main/folly/chrono/Har...

oasisaimlessly• 11 months ago

Read `cntvct_el0` with the `mrs` instruction. [1]

[1] https://developer.arm.com/documentation/102379/0104/The-proc...

JdeBP3• 11 months ago

See manual page and changelog.

* https://jdebp.uk/Softwares/djbwares/guide/commands/clockspee...

* https://github.com/jdebp/djbwares/commit/8d2c20930c8700b1786...

Yes, 27 years later it now compiles on a non-Intel architecture. (-:

madog• 11 months ago

Just use gettimeofday/clock_gettime via vDSO.

  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);

On arm64 it directly uses the cntvct_el0 register under the hood but with a standard/easy to use API instead of messing about with inline assembly. Also avoids a context switch because it's vDSO.

ethan_smith• 11 months ago

While __rdtsc() is fast, be cautious with multi-core benchmarks as TSC synchronization between cores isn't guaranteed on all hardware, especially older systems. Modern Intel/AMD CPUs have "invariant TSC" which helps, but it's worth checking CPU flags first.

zorked• 11 months ago

Invariant TSC has been around for over 15 years, probably more.

loeg15• 11 months ago

Rdtsc is fast-ish, but it's still like 10 ns. Something to be aware of if you're trying to measure really small durations (measure 10s or 100s and amortize).

signa11• 11 months ago

I don't think it is (guaranteed to be) synchronized across cores. I might be wrong about that though.

redleader55• 11 months ago

Succesive rdtsc calls, even on the same CPU, are not guaranteed to be executed in the expected order by the CPU - [1].

1 - https://lore.kernel.org/all/da9e8bee-71c2-4a59-a865-3dd6c5c9...

znpy2• 11 months ago

the first step in benchmarking software is to use the same hardware.

the author failed the first step.

everything that follows is then garbage.

saagarjha4• 11 months ago

You do understand that people who know how to benchmark things don’t actually need to conform to the rules of thumb that are given to non-experts so they don’t shoot themselves in the foot, right? Do you also write off rally drivers because they have their feet on both pedals?

znpy2• 11 months ago

i don't know about you but i did my fair share of benchmarking and performance engineering.

the first thing that you learn in that area is that unless you're comparing specific configuration of apples to apples, along with a specific configuration of the workloads, you're basically doing children play.

i mean, it's fine to play and observe, but it's just play. it's not anything to take seriously.

GTP• 11 months ago

By leaving my finger on the screen, I accidentally triggered an easter egg of two "cannons" shooting squares. Did anyone else notice it?

evanjrowley• 11 months ago

I also saw it, and it happened on a non-touch computer screen.

pan69• 11 months ago

Happened for me on my normal desktop browser, cute but distracting. It also made my mouse cursor disappear. I had to move my mouse outside the browser window to make it visible again.

IFC_LLC• 11 months ago

I'll be honest, this was the first time when I was unhappy with an easter egg and was unable to finish reading the article because of it.

Triggered on a safari on mac.

M_r_R_o_b_o_t_• 11 months ago

uwagar• 11 months ago

the bsd people seem to enjoy measuring and logging a lot.

sugarpimpdorsey2• 11 months ago

OpenBSD is many things, but 'fast' is not a word that comes to mind.

Lightweight? Yes.

Minimalist? Definitely.

Compact? Sure.

But fast? No.

Would I host a database or fileserver on OpenBSD? Hell no.

Boot times seem to take as long as they did 20 years ago. They are also advocates for every schizo security mitigation they can dream up that sacrifices speed and that's ok too.

But let's not pretend it's something it's not.

ThinkBeat• 11 months ago

You must run a different branch of OpenBSD than I.

chasil6• 11 months ago

In some defense of the parent post, a new kernel is relinked at every boot. This load is noticeable.

This is aslr on steroids, and it does vastly increase kernel attack complexity, but it is a computational and I/O load that no version of Linux imposes that I know.

Relinking the C library is relatively quick in comparison.

somat3• 11 months ago

Gotta disagree with you on your minimalist verdict, out of the box openbsd tends to have everything and the kitchen sink, I am not complaining, it is good solid well written software. but I have never found a base linux distro that was as well stocked as a base openbsd install.

    a c compiler
    a web server
    3 routing daemons(bgpd, ospfd, ripd)
    a mail server(smtpd, spamd)
    a sound server(sndiod)
    a reverse proxy(relayd)
    2 desktop environments(fvwm, cwm)
    plus many many more

Openbsd is not some minimalist highly focused operating system, I mean what on earth is it actually for? based on the included features, A desktop development system that is also the router and office web and mail server?

Personally I love it, after a fresh install I always feel like I could rebuild the internet from scratch using only what is found in front of me if I needed to.

Paianni• 11 months ago

fvwm and cwm are window managers, definitely not DEs.

anthk2• 11 months ago

OpenBSD runs much faster than 20 years ago thanks to sending locks to /dev/null.

ksec• 11 months ago

I think OpenBSD has gotten quite a bit faster in the last 4 - 5 releases. Unfortunately we will need to wait for another round of benchmarks by phoronix as it seems to have problems every time benchmarks are ran.

Somewhere along the line may be it will become fast enough and certain applications may use it for different sets of reasons.

eyberg• 11 months ago

This is kind of a stupid "benchmark" but if we're going to walk down this road:

linux: elapsed: 0.019895s

nanos (running on said linux): elapsed: 0.000886s

1vuio0pswjnm7• 11 months ago

"Usually it's the weirdo benchmark that shows OpenBSD being 10x slower, so this one is definitely going in the collection."

Could it be

https://web.archive.org/web/20031020054211if_/http://bulk.fe...

or is there another one

u64x086a2• 11 months ago

What about the FreeBSD?

u64x086a2• 11 months ago

What about this on FreeBSD? It's possible to reach the same results?

loeg15• 11 months ago

FreeBSD doesn't use RCU in the kernel, so it is unlikely to have the exact same ~bug as Linux here. Any naive implementation will have the same decent performance OpenBSD does for this program.

FreeBSD's equivalent implementation lives here (it does all of this under an exclusive lock): http://fxr.watson.org/fxr/source/kern/kern_descrip.c#L1882

It doesn't use RCU, but does do a kind of bespoke refcounted retention of old versions of the fdtable on a free list.

owl_vision• 11 months ago

What were the resource setups? How do they compare to linux resource setups?

Defaults in current are in etc.amd64/login.conf. https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc...

(p.s.: the bubble are cool. highly distracting to me, hence I could not read the article in full.)

loeg15• 11 months ago

The Linux one is essentially measuring a (surprising) sleep; not resource limited.

1970-01-01• 11 months ago

A better way to do that (sorry) is to increase the loading instead of modifying the test. I'm happy to hear reasons why their way could be a better benchmark.

loeg15• 11 months ago

The phenomenon being measured on Linux is a ~one-off event in a process' lifetime.

OpenBSD is so fast, I had to modify the program slightly to measure itself

Related Stories