Rust vs Zig Benchmarks

https://programming-language-benchmarks.vercel.app/rust-vs-zig

by alonp99 • 3 years ago

126 98 3 years ago

programming-language-benchmarks.vercel.app

EscapeFromNY3• 3 years ago

Didn't these benchmarks used to have box plots for every comparison? Or am I thinking of another site?

cristoperb• 3 years ago

You're probably thinking of this one: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

silisili• 3 years ago

Happy for this new one. The one you linked was/is maintained by someone with rather strong opinions that exclude things and may skew results.

EscapeFromNY3• 3 years ago

Thanks yeah that's the one. It's a completely different site, but the design looks similar to my eyes. That's probably why I got them confused

josephg3• 3 years ago

Tldr; they’re about even. Which is what I’d expect given they’re both performance oriented languages which compile via llvm.

As I see it, performance wise Rust has one advantage and one disadvantage compared to zig. Rust’s advantage is that it can add the equivalent of C’s noalias all over the place because of the rules imposed by the borrow checker. This can help the optimizer. And the drawback of rust is that all array accesses are bounds checked. (Well, at least in safe rust). But thanks to prediction intrinsics, the slow down from this is much less than I always expect. Bounds checks do bloat the binary size though.

So rust and zig trading blows benchmark to benchmark is about what I would expect to happen. And that’s exactly what I’m seeing here.

dleslie2• 3 years ago

Which is amazing, because Rust comes with more safety guarantees.

bsder• 3 years ago

Why is this amazing? One would expect the language with more restrictions to compile to faster code, no?

cute_boi• 3 years ago

i see bound checks as advantage. I don't want random stuff from program tbh. And, if someone really wants to disable bound checks for certain stuff, they can always do that.

bluGill• 3 years ago

I'm divided. The majority of the time i know how big the array is with information the compiler doesn't have. However I have to admit 10% of the time i'm off by one.

TazeTSchnitzel• 3 years ago

Doesn't Zig also check bounds by default?

dleslie2• 3 years ago

For slices and arrays, yes.

veber-alex• 3 years ago

Yes, in some compilation modes, but those benchmarks were compiled in "ReleaseFast" mode which disables all safety checks.

drcode• 3 years ago

I did view some of the source files though, and the rust ones look to be a lot longer :-)

galangalalgol2• 3 years ago

Nbody looked about the same, zig had really long lines for stuff that rustfmt split. But that aside, I agree the zif metaprogramming really shome here.

api• 3 years ago

Bounds checks can sometimes be optimized out, such as when using iterators or a for loop over a fixed range. LLVM is pretty decent at optimizing out redundant checks too.

throwawaymaths2• 3 years ago

You can noalias in zig.

josephg3• 3 years ago

You can noalias in C as well, but it is used so infrequently in production C code that llvm didn’t even compile it correctly until a couple years ago. Figuring out when parameters in a C program can safely be declared noalias is very tricky, and I’ve almost never seen anyone bother with it. I assume the same is true in zig.

IceSentry• 3 years ago

The point is that rust does it automatically

conradev• 3 years ago

Zig is writing custom backends in addition to their LLVM backend

josephg3• 3 years ago

So is rust. But I assume these benchmark results all come from llvm.

loeg• 3 years ago

In some sense, so is Rust -- there's work on a cranelift backend, as well as gccrust.

stouset• 3 years ago

People keep being surprised that bounds-checking doesn’t really seem to incur that much cost but frankly it seems pretty straightforward to me.

In the years of Rust code I’ve written, I don’t think I’ve ever actually indexed into an array manually. If I have it’s been an incredibly small number of cases. I’m almost always iterating, which makes bounds checks essentially unnecessary.

cmrdporcupine2• 3 years ago

I suspect it's more that bounds checking actually helps performance in many circumstances in that it can improve branch prediction. Not always, but sometimes.

pjmlp• 3 years ago

As usual, I recall the remarks of C.A.R Hoare on his 1980's Turing Award speech, regarding bounds checking on Algol compilers and customers point of view on how it should be unlawful to do otherwise.

Never, ever, since 1986, have bounds checking been the major source of performance issues on applications I have written.

Rather ill chosen algorithms or data structures.

EscapeFromNY3• 3 years ago

it can add the equivalent of C’s noalias all over the place

There's an open proposal to do this in Zig as well, with the ability to opt out at the individual parameter level (and with safety checks in debug builds).

https://github.com/ziglang/zig/issues/1108

Either way we can definitely thank Rust for blazing the trail. noalias in LLVM had never been stress-tested to that degree, and they were finding and fixing noalias-related optimizer bugs for years

cyber_kinetist• 3 years ago

Sadly, not yet. Even when (Safe) Rust's type system guarantees the noalias annotations will be inserted correctly, the Rust compiler team has been very cautious of turning this on because of numerous LLVM miscompilation issues. It was turned on two years ago in https://github.com/rust-lang/rust/pull/82834 but was subsequently rolled back in https://github.com/rust-lang/rust/pull/86036 because of a miscompliation bug.

the8472• 3 years ago

The rollback was only for the beta, it didn't land on master. So the next release (1.54) did in fact enable mutable-noalias.

You can see it in the function arguments in llvmir: https://rust.godbolt.org/z/a3c366nG6

pverghese4• 3 years ago

Most of the ones zig is slightly slower than rust are due to a less than performative bignum implementation. All the rest zig handily beats rust

Laremere• 3 years ago

Just posting numbers without commentary doesn't seem that useful. Per the GitHub "Note that implementations might be using different optimizations, e.g. with or w/o multithreading, please do read the source code to check if it's a fair comparision or not."

I think this shows that Zig isn't a slow language despite its relative youth, but it'd be much more useful if someone did the work to look through the code and provide commentary on comparable the entries are.

galangalalgol2• 3 years ago

The optimizations both looked correct. Both told the compiler to target broadwell. The fastest nbody was rust, but it was non-portably using x86 intrinsics. Zig has explicit simd vectors in the stdlib and so did better than the portable explicit simd of the third place rust entry. However, zig is using optimized float mode equivalent to gcc ffastmath so it is almost certainly getting the wrong answers simce it didn't use the iterative sqrt trick. https://github.com/hanabi1224/Programming-Language-Benchmark...

ynik• 3 years ago

Is zig's optimized float mode also extremely error-prone like gcc ffastmath?

Reminder: With gcc/clang, -ffastmath makes it undefined behavior to run a calculation that results in an infinity or NaN. Due to the way UB works, the compiler can end up miscompiling not just the floating-point calculation, but also other code nearby (e.g. delete array bounds checks).

This is why Rust does not have any fastmath-equivalent: it would allow violating memory safety in safe code.

vanderZwan• 3 years ago

so it is almost certainly getting the wrong answers

Not checking the correctness of the output sounds like a pretty bad oversight for a benchmark

littlestymaar• 3 years ago

I think this shows that Zig isn't a slow language despite its relative youth

Which shouldn't surprise anyone since the heavy lifting is done by LLVM and zig doesn't attempt to add too much semantics on top of it. (For the same reason I don't think benchmarking unsafe Rust makes much sense either)

estebank• 3 years ago

It makes sense to benchmark unsafe Rust against safe Rust, both for the same algorithm without changes as well as alternative architectures to get around the constraints of the borrow checker and type system. I would go as far as saying that if you want to use unsafe this should be a precondition. It happens too often that someone reaches for unsafe first "for performance wins" that then aren't manifested in practice.

nine_k• 3 years ago

The compiler might make unfortunate decisions which no optimization can fix. E.g. the amount of pointer chasing may affect things a lot. Mutability and aliasing influence drastically which optimizations LLVM would be allowed to make, also stack vs heap allocation for short-lived objects, etc.

igouy7• 3 years ago

Please do.

devit• 3 years ago

Seems kind of pointless, since the implementations for different languages are independently written and thus the results depend mostly on the skill of the programmers who wrote them (at least for languages that are efficient), and on the quality of the specific standard library features being used if any.

gabeio• 3 years ago

Not entirely pointless since this is open source and programmers of some caliber may be able to contribute better or more efficient code. All benchmarks should be taken with a huge grain of salt since this is rarely what these languages are used for by the majority. As well you shouldn’t be choosing a language solely based on speed and not on the individual application of the language. Memory usage though is something useful and impressive in these examples.

nmz• 3 years ago

How do you know/measure what's efficient then?

pverghese4• 3 years ago

Kind of shows how many rewrites of the rust code are needed to equalize or surpass the zig implementations which are much simpler to write

JZerf3• 3 years ago

It's not a good idea to use this site (or the Computer Language Benchmarks Games https://benchmarksgame-team.pages.debian.net/benchmarksgame/ that it is partially based off) as an indicator of how may rewrites are necessary in order to generate good programs. The skill levels of the contributors and the size/popularity of the various programming language communities can vary a lot. The benchmark rules have been changing over time and contributors have been figuring out better algorithms over time so these both result in the contributed programs getting updated over time as well. Older programming languages have been around longer than newer programming languages and will have had more contributed programs consequently. Some programming languages are under more active development so require more revamps of existing programs. Etc...

igouy7• 3 years ago

That isn't something the website was intended to show.

That isn't something the website does show.

igouy7• 3 years ago

The source code is there on the website.

Language A programs can be implemented in language B without being "independently written".

sergiotapia• 3 years ago

I think a more interesting comparison is Zig vs Nim. Where Nim beats or at the very least matches performance against Zig in multiple tests, while having an infinitely nicer, friendlier syntax.

https://programming-language-benchmarks.vercel.app/zig-vs-ni...

I love Nim so much, I wish it became more popular somehow. Unfortunately it has stagnated in popularity.

Buttons840• 3 years ago

I think Nim looks good, but I'm more likely to learn Rust or Zig personally. I've tried to put my finger on why that is, and I think it's because Nim looks like another good language that is just good all around, it makes good trade-offs and finds a local optimum being good at everything. A lot of language have tried this though, I've seen it all before. Rust did something new, they accepted that "yeah, our language might be really painful to write your seat-of-the-pants business logic in" and they did the borrow checker stuff. Zig is similar, keeping many of the pains of C, but making some important improvements.

hugs2• 3 years ago

I feel the same way. I wish Nim was more popular, but I'm not letting that stop me from using it. It's already extremely useful enough. I spent the last few weeks writing Nim code (specifically an OpenCV binding that wraps just the C++ libraries I need). Nim let's me write fast, compiled, "Pythonic" code. And the INim interactive shell let's me program "1 line at a time" the same way I would with the Python REPL or JavaScript console.

danielEM• 3 years ago

That pythonic syntax is something that prevents Nim to gain more popularity. I think it would be a great move from Nim side to introduce dual syntax support: python like and c/d/java/JavaScript/typescript/go/rust/zig ... like

lucidrains• 3 years ago

same, I am too old to really care if others cannot see this as nearly the perfect language

jadbox• 3 years ago

Afaik, nim was influenced by and originally developed in D-lang. Both are wonderful languages and a real joy to develop on. My ultimate dream is a language that's low level like zig but has the flexibility of nim.

stephc_int13• 3 years ago

I think that this kind of benchmarks should always be taken with a grain of salt, as anyone with some experience with benchmarks knows that there are many ways to distort the results.

Having said that, performance is a key feature for both languages and from what I can see the methodology seems legit.

I'd prefer to also have a C or C++ to use as a performance baseline and to get a clearer overall picture of where the current optimizers stands relative to each others.

Also, competition is good, I believe that we'll all benefit from it in the end, even if the heated debates and quasi-religious stances can be annoying, in the long run this is mostly noise.

gkfasdfasdf• 3 years ago

I imagine leetcode has a lot of data on benchmarks of highly optimized solutions implemented in different languages. I wish they would do a blog post or something.

intelVISA• 3 years ago

Huge missed opportunity imo, can't imagine it would be too hard for them to glue together either.

skemper911• 3 years ago

B->C->C++->D->Rust->Zig->ZZtop?

Kamq• 3 years ago

You forgot BCPL

tharne• 3 years ago

Zig is cool and all, but someone should really rewrite it in Rust.

soulbadguy• 3 years ago

Slightly on a tangent here. But i think someone with more knowledge of either the zig implementation/language or programming language and runtime in general should take a second pass at Zig's wikipedia page. I see a lot of assertion which doesn't seem substantiated by the provided references, some of the reference almost look unrelated and some stuff also look wrong ( i am not a zig experts)

The goals of Zig are in contrast to the many similar languages introduced in the 2020s time-frame, like Go, Rust, Carbon, Nim and many others. Generally, these languages are more complex with additional features like operator overloading, functions that masquerade as values (properties), generic types and many other features intended to aid the construction of large programs. These sorts of features have more in common with C++'s approach, and these languages are more along the lines of that language. reference (https://www.infoworld.com/article/3113083/new-challenger-joi...)

The provided reference doesn't really mention this statement, and C++ did not originate a lot of those features.

On a more abstract level, no having abstraction features doesn't always make a language simpler.

A common solution to these problems is a garbage collector (GC), which examines the program for pointers to previously malloced memory, and removing any blocks that no longer have anything pointing to them. Although this greatly reduces, or even eliminates, memory errors, GC systems are relatively slow compared to manual memory management, and have unpredictable performance that makes them unsuited to systems programming.
reference (https://docs.elementscompiler.com/Concepts/ARCvsGC/)

I am not sure how authoritative the given reference is, but even assuming it is, there is no mentions on manual vs GC speed, or anything about how GC systems are inadequate for system programing (what ever that means in this context). I know the the memory management styles trade-off are still hot debates, and a lot of progress have been made with regard to modern implementation.

Another solution is automatic reference counting (ARC), which implements the same basic concept of looking for pointers to removed memory, but does so at malloc time by recording the number of pointers to that block, meaning there does not need to perform an exhaustive search, but instead adds time to every malloc and release operation.

This doesn't sound like the best way to describe ARC..., again the provided references doesn't substantiate this definition.

Zig aims to provide performance similar or better than C, so GC and ARC are not suitable solutions. Instead, it uses a modern, as of 2022, concept known as optional types, or smart pointers.

I don't think smart pointer and optional types are the same. Quickly glancing at zig official website , this looks like a mistake

Instead of a pointer being allowed to point to nothing, or nil, a separate type is used to indicate data that is optionally empty. This is similar to using a structure with a pointer and a boolean that indicates whether the pointer is valid, but the state of the boolean is invisibly managed by the language and does not need to be explicitly managed by the programmer. So, for instance, when the pointer is declared it is set to "unallocated", and when that pointer receives a value from a malloc, it is set to "allocated" if the malloc succeeded.

Mixing optional types and they way zig uses it to represent references.

tw0610235• 3 years ago

These are microbenchmarks, thoroughly optimized to the point some of them are written almost completely in SSE intrinsics.

I would expect Rust to be 2x-3x slower in real production code, solely due to approach to memory allocation.

aw1621107• 3 years ago

solely due to approach to memory allocation

Would you mind elaborating on this? I don't think I've read anything (yet?) that touches on performance differences due to that particular aspect of the languages.

kibwen4• 3 years ago

The only thing I can think of is that Zig makes it easier to use different allocators on a per-collection basis, whereas for Rust the only way to alter the allocator used by the standard containers is to change the global allocator. But Rust programs aren't allocating that much in the first place, certainly not so much that swapping out an allocator on a per-collection basis is going to triple the performance of arbitrary programs; at best, fine-grained allocator control will give modest gains on certain extremely allocation-heavy workloads. And collections in Rust can certainly use custom allocators if they want to, e.g. https://crates.io/crates/bumpalo , so that's not something inherent to the language.

tw0610235• 3 years ago

Long story short, heap allocation is painfully slow. Any sort of malloc will always be slower than a custom pool or a bump allocator, because it has a lot more context to deal with.

Rust makes it especially hard to use custom allocators, see bumpalo for example [0]. To be fair, progress is being made in this area [1].

Theoretically one can use a "handle table" as a replacement for pools, you can find relevant discussion at [2].

[0] https://github.com/fitzgen/bumpalo

[1] https://github.com/rust-lang/rust/issues/32838

[2] https://news.ycombinator.com/item?id=36353145

Edit: formatting.

hgs32• 3 years ago

All references in Rust must have an explicit owner. It's common to create additional structures whose entire purpose is to manage the lifetime of your actual data structures. This means more memory consumption and potentially extra indirection, e.g. slot map handles vs pointers.

kaba0• 3 years ago

That’s naive to think that someone will actually make proper use of custom allocators in Zig. Low-lever, performance-oriented programming is hard, especially when you are doing a more exploratory job without clear goals up front.

And when you do know what you want, explicitly optimizing those parts is equally possible in the two languages.

sedatk3• 3 years ago

We should have developer benchmarks for programming languages such as "time to first successful compile", "time to first running copy", "time to find the root cause of a certain bug". Similar to that Sun vs NeXT competition back in the day: https://www.youtube.com/watch?v=UGhfB-NICzg

Maybe platforms like Leetcode and HackerRank can publish such statistics.

I'd find such benchmarks way more interesting than comparisons of milliseconds for solutions to some arbitrary algorithms that we'll never use in real life. I'm not saying they aren't useful, but they aren't interesting.

hurril• 3 years ago

And also: retries needed until last bug is fixed.

xigoi2• 3 years ago

How would you objectively perform such benchmarks?

sedatk3• 3 years ago

Live competition. Benchmarks would just be metrics recorded during the live broadcast, not goals. You can later aggregate those metrics to come up with accurate benchmarks about many scenarios about software development. Think of NBA but for programming.

fulafel• 3 years ago

This compares Zig unsafe mode to Rust. Is there a comparison available for the "safe" mode?

(I'm assuming the Rust code isn't using "unsafe", if it is ten the "safe" zig numbers are be uninteresting)

Rust vs Zig Benchmarks

Related Stories