kobzol

The new Zig I/O idea seems like a pretty ingenious idea, if you write mostly applications and don't need stackless coroutines. I suspect that writing libraries using this style will be quite error-prone, because library authors will not know whether the provided I/O is single or multi-threaded, whether it uses evented I/O or not... Writing concurrent/async/parallel/whatever code is difficult enough on its own even if you have perfect knowledge of the I/O stack that you're using. Here the library author will be at the mercy of the IO implementation provided from the outside. And since it looks like the IO interface will be a proper kitchen sink, essentially an implementation of a "small OS", it might be very hard to test all the potential interactions and combinations of behavior. I'm not sure if a few async primitives offered by the interface will be enough in practice to deal with all the funny edge cases that you can encounter in practice. To support a wide range of IO implementations, I think that the code would have to be quite defensive and essentially assume the most parallel/concurrent version of IO to be used.

It will IMO also be quite difficult to combine stackless coroutines with this approach, especially if you'd want to avoid needless spawning of the coroutines, because the offered primitives don't seem to allow expressing explicit polling of the coroutines (and even if they did, most people probably wouldn't bother to write code like that, as it would essentially boil down to the code looking like "normal" async/await code, not like Go with implicit yield points). Combined with the dynamic dispatch, it seems like Zig is going a bit higher-level with its language design. Might be a good fit in the end.

It's quite courageous calling this approach "without any compromise" when it has not been tried in the wild yet - you can claim this maybe after 1-2 years of usage in a wider ecosystem. Time will tell :)

I also don't know all the details, but the Rust parser tokens contain horrible crimes, primarily because of macros. All I wanted to say was that applying DoD to the parser in Rust would (IMO) be much more difficult than in Zig, because language differences and different approaches to error reporting. Not saying it's impossible ofc. That being said, I don't really think so much effort would be worth here, the gain would be minimal in the grand scheme of things; we have bigger perf. problems than parsing.

I don't think we hit a bedrock. As I wrote, we have a lot of ideas for massive improvements. But we need more people to work on them.

I'm not sure how that works. You either let the compiler compile your whole program with AVX (which duplicates the binary) or you manually use AVX with runtime detection on selected places (which requires writing manual vectorization).

You have a fair point, I agree that while compiler performance is a priority, is is one of many priorities, and not currently super high on the list for many Rust Project developers. I wish it was different, but the only thing we can do is just do the work to make it faster :) Or support the people that work on it.

For many use-cases yes, but there are crates bottlenecked on different things than the codegen backend.

But I don't think that's the point. We could get rid of LLVM and use other backends, same as we could do other improvements. The point is that there are also other priorities and we don't have enough manpower to make progress faster.

I think a reasonable comparison would have to be DoD Rust parser vs current Rust parser. Comparing across languages isn't very useful, because Zig has very different syntax rules, and doesn't provide diagnostics near the same level as Rust does. The Rust compiler (and also its parser) spends an incredible amount of effort on diagnostics, to the point of actually trying to parse syntax from other languages (e.g. Python), just to warn people not to use Python syntax in Rust. Not to mention that it needs to deal with decl and proc macros, intertwine that with name resolution, etc. etc. This all of course hurts parsing performance quite a lot, and IMO would make it both much harder to write the whole thing in DoD, and also the DoD performance benefits would be not so big, because of all the heterogeneous functionality the Rust frontend does. Those are of course deliberate decisions of Rust that favor other things than compilation performance.

Great post!

I have some random guesses as to why the 40% vs 60-70% memory issues percentage:

- 180k is not that much code. The 60-70% number comes from Google and Microsoft, and they are dealing with way larger codebases. Of course, the size of the codebase in theory shouldn't affect the percentage, but I suspect in practice it does, as the larger the codebase is, the harder it is to enforce invariants and watch for all possible edge cases.

- A related aspect to that is that curl is primarily maintained by one person (you), or at most a handful of contributors. Of course many more people contribute to it, but there is a single maintainer who knows the whole codebase perfectly and can see behind all (or most) corners. For larger codebases with hundreds of people working on them, that is probably not the case.

- Curl is used by clients a lot (probably it's used more by clients than servers, for whatever definition of these words) over which you have no control and monitoring. That means that some UB or vulnerabilities that were triggered "in the wild", on the client side, might not ever be found. For Google/Microsoft, if we're talking about Chrome, Windows, web services etc., which are much more controled and monitored by their companies, I suspect that they are able to detect a larger fraction of vulnerabilities and issues than we are able to detect in curl.

- You write great code, love what you're doing and take pride in a job done well (again, if we scale this to a large codebase with hundreds of developers, it's quite hard to achieve the same level of quality and dedication there).

(sent this as a comment directly on the post, but it seems like it wasn't approved)

That didn't even occur to me, tbh :) But it doesn't have to be SQL linting, I just wanted to appreciate the mindset of not being lazy/afraid to write an unorthodox test.

If I only insert data into the DB once, I could miss important states. Like, I could add non-NULL data to a NOT NULL column, then make it NULL, and then make it NOT NULL again. If I don't insert NULL into the column in-between the last two migrations, I won't trigger the issue.

If you're writing applications or tests, it's mostly the simpler kind of code. If you're writing reusable (or perf. critical code), you will start seeing generics and lifetimes much more often.

(author of the post)

Just to clarify a bit, the test ofc isn't a fully general solution to solving issues with database migrations (I hinted what that might be in the blog post), although it's still useful to provide a nice error message even if a more general solution was implemented.

That was not at all the goal of the post. I just wanted to appreciate how easy it was to achieve this specific task in Rust. In any other systems programming language that I used (even most other languages, except maybe for Python), I would never even imagine something like this being feasible, and so easy to do. That's it :)

I don't really see an attack surface for a dev dependency.

Yeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet.

Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.

Could be done, yeah, but 20s isn't that much, and I'd like to avoid adding more test-only magic environment variables zo configure this (our end-to-end tests are in Python and they use HQ as a binary).

It is sadly not propagated to grandchildren.

I tries the subreaper approach, but it doesn't help. The children are reparented to the worker, but when the worker dies, they are then just reparented to init, like normally.

I do use setsid when spawning the children (I omitted it from the post, but I set it in the dsme pre_exec call where I configure DEATHSIG) but they don't receive any signal, IIRC. Or if they do, it does not seem to be propagated to their children.

The stdlib already mostly does all of that :)

Check out https://kobzol.github.io/rust/2024/01/28/process-spawning-pe....

That's a very good point! But yeah, we use the single threaded runtime, so this shouldn't be a concern.

The Rust bhild config defaults are pretty fine for the general case. It's just that not everyone has the general case :) In Rust the normal distribution will be quite flattened.

Yeah, I actually generated these small charts out of a flamegraph, because it contains too much information and isn't easily split into three distinct parts. And once you condense the information into just 3 blocks, then using a flamegraph doesn't really add any further value, IMO.

There have been some recent improvements to this, but yeah, it can be still quite large. There is a WIP development of a garbage collector in Cargo that could help with this.

It didn't really require establishing a working group :) That is just an effort to improve Rust binary sizes in general.

In the end, changes in Rust can take a very long time (years) sometimes. It's maintained and developed by volunteers, and it has been growing exponentially these past few years, so not everything can move forward as fast as we would like.

There is some work underway to enable removing the backtrace generation/parsing from Rust binaries. It's hardcoded for now though.

You can't really use dead code elimination on debug symbols, because you don't know which symbols will you need. You would need to know where and how will your program crash or if the user will want to use a debugger on it.

I have implemented the fix and got it approved in the span of about 3 weeks (and that was over the Christmas!). Doesn't seem that bad to me, given that it's a change that will affect pretty much any Rust user by default.

Mostly the formatting machinery from stdlib. You can go sub 100 KiB or even less without it, but unless you target embedded, there's really a reason to do that, IMO.

Rust supports linker plugin LTO (https://doc.rust-lang.org/rustc/linker-plugin-lto.html).

I have used Python for 10+ years, and I'm able to forget what are the input arguments/return type of a function after 30 minutes of working on some other part of the code :)

I don't think that it's a good idea in general to inherit the construction functions. But if you wanted to do that, classmethod is the way to go, of course.