Zig, Rust, and Other Languages

https://notes.eatonphil.com/2024-03-15-zig-rust-and-other-languages.html

by tim_sw • 2 years ago

139 113 2 years ago

notes.eatonphil.com

alchemio4• 2 years ago

Just a correction: most std C functions don’t allocate. strdup does but it was only recently adopted into the standard, it was previously an extension.

Similarly zig’s stdlib shouldn’t allocate behind your back, except for thread spawn where it does: https://github.com/ziglang/zig/blob/5cd7fef17faa2a40c8da23f0...

Generally speaking, it’s as mentioned just a convention. A zig library might not allow its users to pass allocators for example.

In C++, stl containers can take an allocator as a template parameter. Recent C++ versions also provide several polymorphic allocators in the stdlib. You can also override the global allocator or a specific class’ allocator (override placement new).

AndyKelley3• 2 years ago

For spawning a thread you're literally asking for the stack of the thread to be memory mapped. I fail to see how this is "behind your back".

You also linked specifically to the POSIX threads implementation of thread spawning, which is by definition supposed to play nicely with the libc posix threads API, which expects you to use the libc allocator in combination with POSIX threads API, so that's what it does.

You might as well accuse the mmap() function in the zig standard library of allocating behind your back.

MindSpunk• 2 years ago

The behind your back part is probably referring to the Args payload bouncing through a heap allocation. It isn't explicit on the signature it's making an allocation. The function has no choice though unless you leave it up to the user to keep the payload allocation live until the thread terminates.

alchemio4• 2 years ago

It’s something Zig touts when compared to other languages(1). The idea is that in the end it’s a convention that an allocator needs to be passed to indicate that the function allocates, which not even the stdlib adheres religiously to. I’m fine with it since I do believe a library writer should know best what works with their library.

1. https://ziglang.org/learn/why_zig_rust_d_cpp/#no-hidden-allo...

fsckboy• 2 years ago

he was pointing out that it does, and he was not applying to it the label "behind your back, that's why he said "except for". the wording makes perfect sense.

jcelerier• 2 years ago

strdup does but it was only recently adopted into the standard, it was previously an extension.

Does it really matter when people were already writing code with strdup when the zig and rust creators were in middle school ? It was already there in BSD 4.3 (1986) apparently.

alchemio4• 2 years ago

It really doesn’t matter and that’s my point.

uecker• 2 years ago

GNU obstacks also already exist for as long as I can remember.

Everything old is new again...

khaledh• 2 years ago

Nim is also a strong player as a systems programming language. In terms of memory management, it's configurable, and by default you get ARC (no GC). I've written a hobby kernel (if you can call it that) in Nim[1] as well as Zig[2], and I found Nim to be much more ergonomic and approachable. The fact that Zig requires weaving an allocator through most calls that may allocate gets in the way of what I'm trying to do. I'd rather focus on core logic and have ref counting take care of dropping memory when appropriate.

One thing I wish Nim had though is true sum types with payloads. I think there's an RFC for that, but it's a shame it's not in the language yet.

[1] https://github.com/khaledh/axiom

[2] https://github.com/khaledh/axiom-zig

loctal• 2 years ago

Gleam is another strong candidate too

lokelow• 2 years ago

Gleam runs on the Erlang VM, and I believe it also compiles to WASM. That's pretty different than the other languages here. Cool language though, I am a big fan of the BEAM

throwawaymaths• 2 years ago

The fact that Zig requires weaving an allocator through most calls

It does no such thing. Why didn't you just make a global allocator? Alternatively, you can have your allocator be attached to your objects.

slily• 2 years ago

People have been making jokes about node_modules for a decade now, but this problem is just as bad in Rust codebases I've seen

I agree... but I think this is more indicative of a cultural problem than a language design/standard library scoping problem. A compact standard library is more resistant to scope creep. I don't want every language to end up like C++, and I think keeping the scope of the standard library small helps avoid that. On the other hand, popular third-party libraries that provide common functions have too many third-(fourth-?)party dependencies. Keeping dependency trees small should be a priority for such tools, but convenience trumps all right now so it isn't valued as it should be.

no_wizard• 2 years ago

one way to approach this with compromise is to keep the shipped standard library small, but have officially maintained packages from the language foundation for exceedingly common functionality.

That gives an avenue for exploration, and tight controls and quality control on packages, without bloating the standard library.

MBCook3• 2 years ago

What’s so bad about “bloating the standard library”? Java has an incredible amount of stuff and it’s incredibly useful.

The compiler should be able to eliminate the unused parts during compilation, it’s not like adding JSON to the standard library is going to make a “hello world” program bigger.

theLiminator2• 2 years ago

I wish languages took this approach, not sure why they largely don't.

coldtea4• 2 years ago

A compact standard library is more resistant to scope creep

I don't see "scope creep" as a problem.

I see "Lack of standard library support for X brought 5 competing libraries for the same X functionality, no de facto standard, and projects using incompatible implementations and versions, maintained in different states of disarray" as a problem.

Never had a problem with Python standard library having "scope creep" for example. Where I do have problem is where there are several competing libraries and no single well supported one you can always depend on for some - which happens for things not in the standard library.

tialaramex7• 2 years ago

which happens for things not in the standard library.

So to be clear you don't use Requests right?

Because there is HTTP client handling right in the Python standard library, and yet it sure seems like most Python programmers use Requests, because it's better.

duped6• 2 years ago

If cargo did literally any garbage collection then it wouldn't be a problem. Most of the nonsense in a target/ directory are stale incremental build artifacts.

menaerus• 2 years ago

The problem as I understand it is not about the stale dependency artifacts in xyz directory but with the (transitive) (hell) dependency management itself. cargo makes it incredibly easy to pull in the shitload of dependencies of whom you don't have a slightest idea.

pornel3• 2 years ago

The problem is that a typical built-in standard library can't change or remove any APIs, even the ones that turned out to be mistakes or became obsolete, without breaking users. Eventually all the fancy standard parts of it get easier, faster, cleaner 3rd party replacements, and everyone is saddled with a dilemma of using the meh stdlib or a 3rd party dependency anyway.

OTOH functionality shipped as external libraries can be versioned, upgraded, and replaced by users individually, on their own schedule. The obsolete cruft can be pulled in by old projects that still need it, without burdening new projects with the same old APIs (but they still can interoperate, because multiple versions of the same library can coexist!)

For very long-lived languages (which Rust has ambition to be) a one-size-fits-all built-in big standard library is a big risk. The bigger it is, the less obvious and less timeless the APIs are. A basic Vec is easy to get right on the first try, but networking is a moving target, fashions for file formats come and go, threads/green threads/callbacks/promises/await come and go, parsers can be designed in many differently-bad ways, and OSes change — we don't have non-binary files any more, but ossified languages do! WASM/WASI has upturned even Rust's barebones standard library, and Fuchsia would even more.

This risk of maintaining mistakes forever eventually makes maintainers of the standard library overly cautious, and design very defensive opaque non-committal APIs.

IMHO languages should not have any single built-in unversioned library at all. Pull in what you use, in a version that makes sense this year, without the baggage that every API has to remain unchanged forever, and still be sensible and efficient half a century later, even on an OS that doesn't exist yet on hardware that hasn't been invented yet.

Thaxll• 2 years ago

Zig does not have a good package manager, does it?

bsder4• 2 years ago

I don't believe Zig wants to have "package manager" at all.

It has the ability to pull in a package/module. However, I believe that having a "package manager" is explicitly an anti-goal.

I'm in agreement. It's become pretty clear that having any form of "package repository" without also allocating the necessary manpower to curate that repository causes disasters.

wavemode2• 2 years ago

I don't believe Zig wants to have "package manager" at all.

? it's literally one of the project's main milestones: https://github.com/ziglang/zig/projects?type=classic

baby• 2 years ago

That sounds like a horrible decision

eyegor• 2 years ago

It has one now, but I haven't played around enough to call it "good" or "bad". It feels more like a hack than something like cargo/pip/nuget/npm/etc, but it's a lot better than the normal c/cpp experience.

build.zig.zon discussion: https://zig.news/edyu/zig-package-manager-wtf-is-zon-2-0110-...

Example howto: https://ziggit.dev/t/how-to-package-a-zig-source-module-and-...

Actual docs: https://ziglang.org/download/0.11.0/release-notes.html#Packa...

papichulo20232• 2 years ago

Are you experienced with vcpkg or just repeating old facts? Considering you are merging c and cpp seems the earlier. I would take vcpkg over npm or pip any day.

AndyKelley3• 2 years ago

I think the upcoming 0.12 release will be a good time to try it out. It was a major focus during this release cycle.

otabdeveloper4• 2 years ago

You shouldn't need one. Use Nix instead, they did 99% of the work for you already.

rofrol2• 2 years ago

Zig does not have a good package manager, does it?

This describes upcoming changes in zig 0.12 https://ziggit.dev/t/build-system-tricks/3531

coldtea4• 2 years ago

Zig is not a mature language. But it has made enough useful choices for a number of companies to invest in it and run it in production. The useful choices make Zig worth talking about.Go and Rust are mature languages. But they have both made questionable choices that seem worth talking about.

Well, Zig's native string type support, or lack thereof, is also a questionable choice. It's not like Zig did all the right choices and only Go and Rust made questionable ones.

pyrolistical• 2 years ago

What advantages would there be with a native string type over []const u8. (For those not fluent in zig, a string is typed as a slice constant bytes, where a slice is a pointer + length)

Ygg23• 2 years ago

An UTF8 String would have invariant of always having valid UTF8, omitting need for further UTF8 validation.

If you're working on bytes string in Zig the status of them is unknown, so you have to always validate even if it's just Utf8. In Rust you can omit Utf8 validation for bytes from a String.

wavemode2• 2 years ago

Well, Zig does have Unicode support in its standard library. A "String" type wrapping that functionality would be slightly safer but, I don't see it as some huge wart of the language. You could always just define your own.

floobertoober• 2 years ago

The more I've used the language, I'm starting to think that this decision makes sense in the context of Zig. Common string operations involve allocation, non-constant time access, etc. I can definitely understand how strange that sounds without the rest of the language around it as context, though.

tialaramex7• 2 years ago

Rust's String requires allocation, but str (typically seen as the fat pointer type &str) does not. "The cat sat on the mat".split('a') is an iterator over sub-strings, no more allocation needed here than for 105u32.leading_zeros()

A lot of the Rust string API lives in str, not String.

deagle50• 2 years ago

the inclusion of `bufPrint` in the standard library is really nice, you can just give it a pointer to an array. If you want to make a string on the stack in Rust you have to pull in something with a `Writer` trait like `ArrayVec` and use `write!`.

emidoots• 2 years ago

In fact, Zig has pretty much the same support for strings as both Rust and Go. The primary difference which people seem to complain about, that "[]const u8" is not written as "string", can be solved by writing `const string = []const u8;` at the top of your program if you like.

Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.' - also `String` and `OsString`.

edit: apparently Rust enforces that &str cannot be created with invalid UTF8, at the additional performance cost of runtime checks, so it has stronger guarantees than other languages like Go/Java/Zig/etc (as usual)

Go: `string`, 'in effect a read-only slice of bytes', 'a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes.'

Zig: `[]const u8`, 'some bytes, and a length', 'Zig has no concept of strings', 'by convention parameters that are "strings" are expected to be UTF-8 encoded slices of u8.', 'Generally, you can use UTF-8 and not worry about whether something is a string.' - and the stdlib has functions for working with unicode and strings.

Ygg23• 2 years ago

Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior

If you're constructing invalid `str` you already have severely fucked up. And dog is eating your HDD.

josephg4• 2 years ago

'Constructing a non-UTF-8 string slice is not immediate undefined behavior

Rust’s string constructors (for both str and String) check that the data contains strictly valid UTF8.

You can opt out of this check in performance sensitive code - but doing so is considered unsafe. In safe rust it’s impossible to construct a str / String which contains invalid utf8.

Eg: https://doc.rust-lang.org/std/str/fn.from_utf8.html

pgwhalen• 2 years ago

I am definitely not a rust expert, but as far as I can tell this is just not true? The quote you copied is from the documentation is in the context of creating a string in an unsafe manner. If you're going to include unsafe behavior, you may as well say that nothing in rust means anything at all.

duped6• 2 years ago

also `String` and `OsString`

Because these are different types with different semantics.

Why should there be one string type? Languages have to interface with the real world, where string data could be anything, but programs still need to be written for the common cases... hence, &str and String.

bobbylarrybobby2• 2 years ago

String is utf8, OsString is not (necessarily). Having two separate types mean you can guarantee utf8 when you want to, but don't have to when you can't, such as when dealing with “strings” that an OS produces, which are basically just sequences of bytes (perhaps with zero bytes disallowed). When you have one string type that has to do both jobs, you lose out on the guarantees of the more restrictive type.

thayne2• 2 years ago

and the stdlib has functions for working with unicode and strings.

Last time I looked I don't think it did. It looks like there is a unicode namespace now, but it is very primative. It doesn't have a simple way to iterate over codepoints, much less graphemes. Nor does it have anything to inspect the category a codepoint belongs to.

bsder4• 2 years ago

Everybody says they want a built-in String type.

And then everybody proceeds to write their own String library anyway.

"The only winning move is not to play."

pornel3• 2 years ago

Rust has a nice solution of having a &str slice (equivalent of C++ std::string_view) as the lowest common denominator and a deref operator that can easily coerce various string types to the slice.

This allows Rust to have all kinds of string flavors (fixed-len or NUL-terminated or growable, on stack or heap or in ROM, with SSO, with CoW, interned or refcounted, atomically or not, and so on), but they all coerce to the basic &str, so they have the same basic methods, and are compatible with most functions that just want a string without caring how it's allocated.

You can use your own weird string type if you want, but usually you're not forced to convert or copy it to use it with standard library functions or 3rd party dependencies.

DinaCoder985• 2 years ago

And then everybody proceeds to write their own String library anyway.

Is this true? It was (is!) certainly true for C, but C has an especially emaciated expectation for string processing primitives. Any runtime developed after like 1995 that I can think of has fixed this by providing a sane string implementation people generally agree upon.

MBCook3• 2 years ago

I’ve used Java my entire career. I have seen a custom string type nor a post advocating it in Java.

Arnavion3• 2 years ago

Zig is practically alone in that if you write the next() method and and don't pass an allocator to any method in the next() body, nothing in that next() method will allocate.

It's not quite as simple as that. This can allocate just fine:

    fn next(self: *Self) {
        self.array_list.append(5);
    }

... where `array_list` is of type `std.ArrayList(u32)`. `std.ArrayList(T)` takes an allocator at construction time (`array_list = std.ArrayList(u32).init(allocator);`) and stores it as a field, so its methods don't need to take an allocator parameter.

(And yes, there is a version of `std.ArrayList(T)` called `std.ArrayListUnmanaged(T)` that does *not* take an allocator in its constructor and does take an allocator in all its methods that need to allocate.)

AndyKelley3• 2 years ago

That would not compile because the append operation can fail, and errors cannot be ignored.

tialaramex7• 2 years ago

The exact code written won't compile but it gets the idea across - the original claim from the blog post was just false as far as I can tell. We can't in fact be sure that no allocation took place just because we didn't provide an allocator. It's a convention at most, probably for code that's written by Zig programmers who prefer this style, it's true but you could say the same about any convention for any language.

rwbt• 2 years ago

There's also Odin[0] too. I experimented them all and Odin is pretty nice. Nim is also good too but a lot more features.

But - I concluded that language matters a lot less compared to APIs. Yes, the language should have enough good features to let the programmers express themselves, but overall well designed APIs matter a lot more than language. For example -tossing most of the C stdlib and following a consistent coding style (similar to one described here -[1]), with using Arenas for memory allocation, I can be just as productive in C.

[0] https://odin-lang.org [1] - https://nullprogram.com/blog/2023/10/08/

duped6• 2 years ago

People have been making jokes about node_modules for a decade now, but this problem is just as bad in Rust codebases I've seen.

Just w.r.t the issue of size (not of scale) - cargo caches sources in ~/.cargo so they're not shared by every project on the system. Additionally, rustc uses dead code elimination by design so there's not nearly as bad an issue as in JS where it's not possible to tree shake out every unused class or method.

Most of the bloat of target directories are stale incremental build artifacts, because cargo does not do any automatic cleanup.

ViewTrick1002• 2 years ago

Automatic GC of the cargo cache is available on nightly. Really looking forward to it landing on stable. For large projects I've seen the cache grow to hundreds of GB, and not noticing it until the compilation fails because the disk is full...

https://blog.rust-lang.org/2023/12/11/cargo-cache-cleaning.h...

galangalalgol2• 2 years ago

I've had the same issue with conan, and realizing only the. That is why our pipeline took so long, copying tens of GB to the runner. Cargo is mild by comparison.

thayne2• 2 years ago

WRT the standard library. There are tradeoffs. Yes, a big standard library that has everything you need is great, as long as it is well designed, well maintained, and has some mechanism to prevent the whole thing bloating every executable.

But executing well on that is difficult for a number of reasons:

- the release cycle of the standard library is (usually) tied to the compuler, which often means it can't evolve quickly.

- backwards compatibility is a much bigger deal for the standard library. Which means if you have a big library you will eventually have a big pile of deprecated APIs you will probably never be able to actually remove. It also feeds into the next point - new development in stdlib can be hindered by the need to get it right the first time. - the creators of the language probably aren't experts in every area. Which means in order to include say, a good compression library you either need to recruit someone who is an expert to write and maintain the package in your stdlib, or link to a library in another language, and the latter is still really an external depency. - a large stdlib is a large maintenance burden

Python has a large standard library, but it has parts that are deprecated and/or largely abandoned, including packages for technologies that are no longer widely used. And its http packages are rarely used directly because third party packages like requests or urllib3 are better.

Go's standard library that is praised here isn't perfect either. The log and flag packages are often insufficient, and the implementation of net.IP is suboptimal [1].

I think probably the best balance is to start with a fairly small standard library, and when community libraries for key functionality become popular and stable, then pull them in to the standard library or otherwise make them official. And maybe have official libraries that are external to the stdlib and maybe have less rigid backwards compatibility garantees that can move faster than the language itself.

[1] https://tailscale.com/blog/netaddr-new-ip-type-for-go

klabb3• 2 years ago

The log […]

There’s log/slog now. Is that sufficient?

and flag packages are often insufficient […]

Yes. Flag is really primitive and awkward to use. On the other hand, in a typical “imperative shell” style CLI, sharing flags between packages is not exactly crucial, which is the main reason to put it in std (interop). I’ve never really suffered from importing a 3p flag parser in my own code.

and the implementation of net.IP is suboptimal [1].

Fixed for a long time. See net/netip (although I am annoyed it’s not universally available in all net functions). It’s based on the implementation from your link.

Arnavion3• 2 years ago

Yes. The article also says:

Having a large standard library doesn't mean that the programmer shouldn't be able to swap out implementations easily as needed. But all that is required is for the standard library to define an interface along with the standard library implementation.

But "Just have the standard library provide an interface" also fails when the interface itself is not guaranteed to be correct. In Rust for example, the original definition of the Future trait in the futures 0.1 crate was incompatibly different from what it eventually became in the futures 0.3 crate, and the futures 0.3 version is what was imported into libstd as `std::future::Future`. If the 0.1 version had been defined in libstd it would not have worked with async-await, so either it would've stifled async-await development, or it would have to be deprecated but still kept around for backward-compatibility alongside the newer trait.

eviks• 2 years ago

3 years? 68 lines of code. Is it not safer at this point to vendor that code?

Why, though? There is no bitrot or build flakiness with unnecessary changes if no changes happen, so what's the actual argument for this? Why is it missing from the article?

lamontcg• 2 years ago

Similarly, if you go looking for compression support in Rust, there's none in the standard library.

I have no idea why that is considered a criticism.

Compression support is the kind of thing which is going to evolve over time and compression formats will change and performance and APIs of different libraries will get better.

You avoid the problem where the compression library in the std library is the one that experienced programmers tell all the noobs that they should never use.

aldanor• 2 years ago

I'd replace "similarly" with "fortunately"

electroly• 2 years ago

You avoid the problem where the compression library in the std library is the one that experienced programmers tell all the noobs that they should never use.

This is not inevitable. In the .NET world we have great compressors in the standard library. Zip and gzip aren't going anywhere. I totally disagree with the idea that we would have been better off without those compressors in the standard library.

Zig, Rust, and Other Languages

Related Stories