rnijveld

I find this a very weird way of formulating this. Even if AI keeps its promises and is the future of computing (hopefully without destroying the environment), should it not be applauded that there are still organizations funding work in non-AI related technology? You can say many things about AI, but a lack of funding is not an issue at all I would say. Meanwhile all that other stuff still needs maintenance and development, or would you consider anything not focussed on AI wasted?

You realize this article talks about Rust on embedded hardware specifically, where you don’t have threads or big runtimes? There is no hate going on here either, just attempts to make things better. Might I suggest you click through to the homepage and I think you’ll figure out the rest.

As someone working on an NTP implementation (specifically ntpd-rs) I have to add some context to this: I do believe that donating to the Network Time Foundation is fine, but it is not required to keep the Network Time Protocol up in any way.

Firstly, the most important reason the ntp.org domain name is so well known is because of the NTP pool, which is an entirely separate project (the Network Time Foundation calls it an associated project), which was allowed to use the `pool.ntp.org` domain name, but does not directly receive significant funding from the Network Time Foundation as far as I understand (I do not know the details of the domain name arrangement). That pool project was developed independently of the Network Time Foundation and is run by a different group of volunteers, mostly being developed and maintained by Ask Bjørn Hansen and hosting servers entirely consisting of (sometimes professional) volunteer operators. This is what many NTP implementations, specifically many Linux distributions, use as their standard source of time. But it does not appear to depend much on the Network Time Foundation for continued existence.

Secondly, despite all the claims made on the Network Time Foundation site, the IETF took over development and maintenance of the NTP protocol for something like two decades now already under the NTP working group. This was all done with the Network Time Foundation fully agreeing this was the way forward. But for some reason they still consider themselves exempted from any process that the IETF uses and consider themselves as the true developers of the protocol. They constantly frustrate the processes that the IETF uses, claiming that they should receive special treatment as being the 'reference implementation'. Meanwhile, the IETF NTP WG does not have a concept of the reference implementation at all, instead considering all NTP implementations equal.

Aside from this frustrating stance, the Network Time Foundation also didn't do much work on trying to forward the standard at all, instead relying on the status quo from the late 90s and early 2000s. Meanwhile the IETF NTP WG worked on standardizing a way to secure NTP traffic (with regular NTP traffic being relatively easy to man in the middle, with older implementations even being so predictable that faking responses didn't even need reading the requests). That much more secure standard, NTS, was fully standardized in September of 2020, but the Network Time Foundation continues to not implement this standard. All of this has resulted in almost every Linux distribution that I know of replacing their ntpd implementation with NTPsec (with ntpd not even being available as an alternative anymore for installation).

Meanwhile people also started working on NTPv5, in order to remove some of the unsafe and badly defined parts of the standard, and in general bring the spec back up to date. As part of this process, it was decided some time ago that in contrast to the previous NTP standards, the algorithms specifying what a client should do in order to synchronize the time should be removed from the standard (the algorithms specified in the previous standards were not being used by any implementation, not even the ntpd implementation by the Network Time Foundation itself). NTPv5 instead focuses on the wire format of NTP packets and the simple interactions between parties. Yet despite there having been a consensus call on this, and despite no current implementation following the exact algorithm as specified in NTPv4, the Network Time Foundation continues to frustrate the process by claiming that these algorithms are an essential part of the standard.

All of this frustration was also a large part of why the PTP protocol was eventually developed at the IEEE. That is to say: even though the operating mode of PTP is often quite different to that of NTP these days, the information that needs to be transferred is essentially the same, and the packets could have trivially been defined to be the same as long as NTP had built in a little bit of additional flexibility a little bit earlier. This would have also helped NTP in the end (with for example hardware timestamping only being implemented for PTP right now, even though it could have been just as useful in NTP), and with PTP now also aiming to introduce a simpler client-server model via CSPTP that looks a whole lot like what NTP was trying to achieve all this time with its most used operating mode.

It is my belief that the Network Time Foundation continues to push themselves in a corner of more and more irrelevance even though that did not need to be. The historical significance of David Mills' ntpd implementation is definitely there, and we should applaud the initial efforts and their focus on keeping the protocol open and widely available. And I do believe that the current people at the Network Time Foundation could still provide more than enough valuable input in the standardization process, but they cannot claim anymore to be the sole developers of the NTP protocol. Times have changed, there are now multiple implementations with an equally valid claim. Especially with GNSS (specifically GPS) being under attack more and more these days, we need alternative ways of synchronizing computer clocks to a standard time in a secure way. NTP and NTS are perfectly positioned to take on that task and we need to make sure that we keep the standard up to date for our evolving world.

Edit: if you want something else to donate to, I would consider donating to the IETF, NTPsec, or maybe donating some time to the NTP pool. I would also link to donations for Chrony (one of the other major NTP server implementations) but they do not appear to offer anything. Linking to my own project's donation page does not seem fair considering the contents of this post.

The ntp pool is actually independently run and funded and has nothing to do with the NTPd implementation nor the NTP Foundation, other than them allowing the pool to use that DNS name.

The features we specifically don’t support are those related to direct LDAP support within sudo, so things like loading a sudoers file directly from LDAP. Sudo-rs will use any user retrieved via NSS, such as when configured using SSSD to load LDAP users. And from the authentication side you can use whatever PAM supports, so anything like Kerberos etc, which again can be coupled with the same LDAP database.

I would argue compile time changes don't matter much, as the amount of data going through zlib all across the world is so large, that any performance gain should more than compensate any additional compilation time (and zlib-rs compiles in a couple of seconds anyway on my laptop).

As for dependencies: zlib, zlib-ng and zlib-rs all obviously need some access to OS APIs for filesystem access if compiled with that functionality. At least for zlib-rs: if you provide an allocator and don't need any of the file IO you can compile it without any dependencies (not even standard library or libc, just a couple of core types are needed). zlib-rs does have some testing dependencies though, but I think that is fair. All in: all of them use almost exactly the same external dependencies (i.e.: nothing aside from libc-like functionality).

zlib-rs is a bit bigger by default (around 400KB), with some of the Rust machinery. But if you change some of that (i.e. panic=abort), use a nightly compiler (unfortunately still needed for the right flags) and add the right flags both libraries are virtually the same size, with zlib at about 119KB and zlib-rs at about 118KB.

Ah right! I always forget about that since we don’t implement the management protocol in ntpd-rs. I think it’s insane that stuff should go over the same socket as the normal time messages. Something I don’t ever see us implementing.

Probably, but we still need to parse that string on the client side as well. If you’re willing to do the work I’m sure we would accept a pull request for it! There’s just so many things to do in so little time unfortunately. I think reducing our dependencies is a good thing, but our dependencies for JSON parsing/writing are used so commonly in Rust and the way we use it hopefully prevents any major security issues that I don’t think this should be a high priority for us right now compared to the many things we could be doing.

I also agree that there is too much churn in the Rust ecosystem and that we should try and slow things down in the coming years. ntpd-rs also does this: our MSRV is 1.70 right now (that was released over a year ago) and we test our code on CI against this version (as well as the current stable release). And we go a little further. Using the `direct-minimal-versions` (nightly only right now unfortunately) flag we downgrade our dependencies to the minimal ones we've specified in our `Cargo.toml` and test against those dependencies as well, as well as the latest dependencies specified in `Cargo.lock` which we update regularly. This allows us to at least partially verify that we still work against old versions of our dependencies, allowing our upstream packagers to more easily match their packages against our own. Of course we all should update to newer versions whenever possible, but sometimes that is hard to do (especially for package maintainers in distributions such as Fedora and Debian, who have to struggle with so many packages at the same time) and we shouldn't create unnecessary work when its not needed. Hopefully this is our way of helping the ecosystem slow down a little and focus more on security and functionality and less on redoing the same thing all over again every year because of some shiny new feature.

I'm afraid this is a pretty common sentiment. NTS has been out for several years already and is implemented in several implementations (including our ntpd-rs implementation, and others like chrony and ntpsec). Yet its usage is low and meanwhile the fully unsecured and easily spoofable NTP remains the default, in effect allowing anyone to manipulate your clock almost trivially (see our blog post about this: https://tweedegolf.nl/en/blog/121/hacking-time). Hopefully we can get NTS to the masses more quickly in the coming years and slowly start to decrease our dependency on unsigned NTP traffic, just as we did with unencrypted HTTP traffic.

Our project also includes a PTP implementation, statime (https://github.com/pendulum-project/statime/), that includes a Linux daemon. Our implementation should work as well or even better than what linuxptp does, but it's still early days. One thing to note though is that NTP can be made to be just as precise (if not more precise), given the right access to hardware (unfortunately most hardware that does timestamping only does so for PTP packets). The reason for this precision is simple: NTP can use multiple sources of time, whereas PTP by design only uses a single source. This gives NTP more information about the current time and thus allows it to more precisely estimate what the current time is. The thing with relying purely on GNSS is that those signals can be (and are in practice) disrupted relatively easily. This is why time synchronization over the internet makes sense, even for large data centers. And doing secure time synchronization over the internet is only practically possible using NTP/NTS at this time. But there is no one size fits all solution for time synchonization in general.

I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock.

Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start.

Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.

I agree that amplification and reflection definitely are worries, which is why we are working towards NTS becoming a default on the internet. NTS would prevent responses by a server from a spoofed packet and at the same time would make sure that NTP clients can finally start trusting their time instead of hoping that there are no malicious actors anywhere near them. You can read about it on our blog as well: https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-s...

One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload.

As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!

I don’t think our dependency tree is perfect, but I think our dependencies are reasonable overall. We use JSON for transferring metrics data from our NTP daemon to our prometheus metrics daemon. We’ve made this split for security reasons, why have all the attack surface of a HTTP server in your NTP daemon? That didn’t make sense to us. Which is why we added a readonly unix socket to our NTP daemon that on connecting dumps a JSON blob and then closes the connection (i.e. doing as little as possible), which is then usable by our client tool and by our prometheus metrics daemon. That data transfer uses json, but could have used any data format. We’d be happy to accept pull requests to replace this data format with something else, but given budget and time constraints, I think what we came up with is pretty reasonable.

I would encourage you to take a look at some of our testing data and an explanation of our algorithm in our repository (https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a...). I think we are very much in spitting distance of Chrony in terms of synchronization performance, sometimes even beating Chrony. But we’d love for more people to try our algorithm in their infrastructure and report back. The more data the better.

In our internal testing we are very close to Chrony with our synchronization performance, some of our testing data and an explanation of our algorithm is published in our repository: https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a...

Given the amount of testing we (and other parties) have done, and given the strong theoretical foundation of our algorithm I’m pretty confident we’d do well in many production environments. If you do find any performance issues though, we’d love to hear about them!

This only works on Linux of course since the Linux kernel and libc are not tightly coupled, on any other OS dynamically linking against libc is a necessity. Also, I've never seen anyone statically linking against glibc? Is that even something people do? I'd consider your system provided libc being broken is a similar situation to an unbootable kernel: you just need a rescue stick/partition to fix it, or reinstall your OS.

I'd like to respond to a few things. I think using dependency count as a metric is a bad idea, that metric could easily be lowered by just copying all the code over to your project. As you rightfully say, the logic has to exist one way or another. Our approach definitely wasn't that though: it was and is never our goal to have no dependencies, but we do think that dependencies should be part of the safety story, i.e. is a dependency better or worse than what you would write yourself based on your specific use case. This needle will fall much much quicker towards writing it yourself for something like sudo, but still, at least the considerations you make should stay the same, even if the decision ends up being different. Considerations such as: is the communication with the dependency team worth it for the amount of code we save, are their goals aligned with ours, is the number of transitive dependencies that I take within my codebase with this dependency small enough, how much code am I actually saving, would I even be able to do this myself, could I help the wider community with my contributing back to that dependency, etc. I do feel that right now more often than not dependencies are just bolted onto a project as needed, and no consideration is given to any burden such a dependency might have. Aside from that, I think at least a much better metric would be something like 'teams', 'groups' or 'projects' needed to keep your project working. Still not perfect, nor is any other metric, but sometimes it helps to quantify things.

Some responses to your notes:

- The trouble is that we had to re-implement an existing CLI, and as you might expect with something that evolved over a period of some 30 years, there are quite a few weird behaviors in sudo. We initially had a mostly working implementation based on clap, but could not get some parts of the CLI to parse nicely, i.e. the code just looked hacky, and had to do all kinds of post-processing to complete the parsing of the CLI, resulting in lots of additional code. Maybe we should have looked at something like lexopt, but we just went ahead and did it ourselves initially just to see how it would go, we kind of liked the result and never looked at any alternative implementations. I do believe we looked at clap alternatives for a little while to see if something would make our parsing a little easier, but lexopt didn't surface at that time for whatever reason. We're not perfect either. I do think our parsing is pretty decent though.

- We did think about contributing back, but in the end we wanted a little more control over where the password (or more precisely 'hidden input') was stored in memory, and needed some specific parts for handling TTYs (given our setuid context) resulting in us quickly deconstructing rpassword until almost nothing of it was left. I think it's a little hard to contribute those things back, but as a side project I'd love to contribute some of the changes we made back to rpassword, there just wasn't the time to do it at that time as it would be quite a bit of work.

- Glob is a hard one, as the Rust crate is not entirely compatible with how the original sudo works. But the logic has to be there one way or another and if we have to decide between libc (i.e. probably C code) and Rust, we'd prefer to go with Rust of course. That already resulted in an issue being opened for incompatibilities of course, but it's a hard one: I'd prefer to keep the Rust code, so I hope that someone who manages glob at least agrees that it should be as compatible as possible. But I can't and don't have the expectation that their team has the same priorities, and thus we are back at one of the reasons why a dependency might not always be worth it. There's always choices to be made. For now though, we'll keep the Rust crate dependency, as it works well enough!

- Thiserror is great for prototyping, but loses its value quickly once you know what kind of errors you have, it just takes a few lines of extra code. But, thinking about teams etc: given that it is not that big, and is created and maintained by dtolnay, whose code you probably already use in multiple ways in nearly any other project, it's not the worst either. For sudo-rs though, I still think it was the better choice to remove it.

- All the sudo-* packages were mostly removed because we didn't want to expose any public API for all that internal stuff. Our initial goal is to get sudo the CLI application working, not to provide all the building blocks while the API is still in flux. We initially put it all in separate crates because of compilation time worries, but in the end those worries were unfounded. It's one of those things where Rust is still somewhat limited: we can't specify these sort of semi-private dependencies in the crates ecosystem right now, if we would have been able to specify 'nobody but us can use these as a dependency' they would have probably stayed as separate crates.

BTW: I'd like to thank you for continuing to work on Clap! There might have been a time I would have been a little worried about all the breaking changes and churn happening, but since that has stabilized I couldn't be happier! I don't think there's anyone on the sudo-rs team that had anything against clap, and I did not want to single out clap in our post specifically, so I hope you don't consider it an attack against clap. At least personally I use clap in basically every other project with a CLI.

I can only thank you for the work you've done in creating sudo, I think it's an invaluable tool in the general day to day use for so many people. As someone working on sudo-rs, our goal with creating it never was to invalidate any of the work previously done, and we are very much aware that our implementation will not be bug free, especially not at the start.

For me personally, creating this Rust version allowed me to work on something that I would normally not be able to work on, given how I would not rate my confidence in writing relatively safe C code very high. If nothing else, at least we already found a few bugs in the original sudo because of this work. Despite the 43 years of bugfixing, such a piece of software is unlikely to ever be free of bugs, even if just for the changing surroundings.

Other than that, having some alternatives can never hurt, as long as we keep cooperating and trying to learn from each others work (and from each others mistakes).

They don’t have to be though, a yubikey can be used as a passkey as well.

We did indeed look at and are still looking at the list of CVEs where sudo takes part right now. However, many of the bugs are through one way or another not applicable for our implementation (yet). For example CVE-2014-0106 is only applicable when env_reset is disabled, but we purposely do not support that mode. Or take for example CVE-2023-28487 which concerns sudoreplay, but right now we have no plans to add that functionality.

Sudo does have a suite of regression tests, but they mostly test specific implementation details which makes it hard to port to our codebase. Our test suite tests against the integrated artifact, which allows the easy switching between the two implementations (and also allowed us to find a few bugs in the original sudo).

I’m on the sudo-rs team: we are actually very mindful of the dependencies we use. Our current main branch already uses significantly fewer dependencies than a few months ago. Aside from security it also really helps with adoption since it makes packaging way easier. A large part of the output you got is due to a tempfile dependency, which is a dev only dependency that is not touched when compiling a release/dev binary, only during testing a small part of this would be used. I say a small part because most of it is related to tempfile running on windows, which is irrelevant for sudo-rs since we don’t support or intend to support windows.

Just for good measure though and to prevent any further discussion about this I’ve removed the tempfile dependency from the main sudo crate as our usage of it could easily be replaced by a simple timestamp/pid combination. But again, this really only affected testing, I think the size of the code that ends up in the final binary is very reasonable.

As for people suggesting supply chain attacks via dev dependencies: I doubt we would be the final target of such an attack: i.e. what an attacker would really want is access on all/some machines that have sudo-rs installed. The only way to do that would be to change the release artifacts, which dev dependencies do not have the ability to change, at least not directly, so such an attack would only be the first step in a chain of attacks. I have a feeling that there are way easier and less detectable ways of manipulating us than by using modified dev dependencies. Of course that doesn’t mean we should ignore the risks.

We’d love to get a report for this! While we are sure we will make bugs (which is also why we have the notice up on the repository, and you have to go through some hoops in order to use it), we also think that we can try and come up with an implementation that is safer in other ways as well. And if nothing else then at least some good attention will hopefully be drawn to sudo, because it can use a little care and attention.

I’m on the team that is creating sudo-rs. I agree that just assuming that memory safety will get rid of all the bugs is just plain wrong, it will prevent some particular classes of bugs, but that is all it does.

But we do think that sudo in particular suffers from being a really old codebase, and that especially on that point we can make progress. Sudo has lots of features no longer relevant. It has a plugin interface, but that interface uses almost exclusively strings in `key=value` format, and the plugin boundaries aren’t perfect either. It has behaviors that probably almost nobody relies on and yet add complexity and increase the attack surface. If we wanted to do something about that it would mean rewriting large parts of sudo. So if we’re re-implementing sudo anyway, why not start at the basics and build it a little better.

Aside from the memory safety I think that is one of our main goals: make a simpler sudo that still has lots of the expressiveness. I hope you agree that cleaning up sudo is a good thing, and if we are doing that, we might as well take some memory safety along for the ride.

I think your expectations are too high. During our initial exploration we actually managed to talk to Todd Miller, the maintainer of sudo. In our (brief) interactions with him he did not sound cavalier like this at all. Instead I think that a lot of the issues with sudo are more about it being a thirty+ year old program and codebase, and sometimes features turn into bugs and security issues all on their own in such time periods. But then again, C just cannot offer the kinds of protections that Rust can, and mistakes will be made eventually by every human, better to have some protection from your mistakes than none at all.

Definitely one of the things that is on our radar! Lots of features in sudo are just basically legacy features, remnants from the 90s or 00s that are no longer necessary in a modern setup, but nevertheless exist because in past times they used to be the default or only option. We are particularly aware of how much power a suid binary has, so we do try to be cautious!

I’m on the team that is implementing sudo-rs, and we are definitely aware of doas and other similar utilities. Doas does definitely have its usages, especially with the ‘just let me run this as root’ use case in mind. That said, there is also something to say for the more extensive configurability of sudo, and doas does sometimes have an execution model that might be a little too simple. And having another alternative is never a bad thing in my mind.

I’m one of the people working on this. While I do think that the memory safety aspect is an important one, I think that one thing where we can win even more is that Rust has a type system that is much more robust than what C offers. If set up right, we can prevent lots of these kinds of bugs right from the start. We can also simplify things in lots of cases by removing unnecessary details no longer relevant in a modern implementation.

I wasn't arguing that official distro repositories are unsafe, I was actually saying user provided repos are almost as bad (or even worse in some ways, given that give the feeling of being way more secure) as `curl|bash`. Even if they are signed (such as AUR and PPA) most people will blindly add signing keys for people or organisations they do not know, giving them the feeling that they have secured themselves, but have they really?

I guess detecting attacks is easier if all files have to be uploaded to a central service, which does allow everyone to see the personalized attack (I mean adding `if (targetUser()) attackTarget()` isn't that hard, but it would be visible for everyone compared to doing that server-side). But then if I'm a sophisticated attacker I'd be sure to make that way less obvious in my code. My feeling is that it would be detected later rather than sooner if hidden well enough. And that is excluding things like non-official apt repositories.

I wonder why there is such a focus on this `curl|bash` pattern. Meanwhile most of us are downloading hundreds of thousands of lines of code using all kinds of package managers and I don't see many inspecting all those downloaded files, especially not manually. I don't think anyone would ever get to doing anything other than checking if you really want to verify everything.

I'm not saying that downloading something from your official OS package repositories is the same as downloading a random URL from the internet. The thing I'm more thinking about is language specific package managers such as NPM, Composer and Cargo. Or user repositories, things like AUR, PPAs and non-official apt repositories, where any random person can put something up. The thing for those is that they almost look like they are something official and something to be trusted. Often times they are displayed on an official site, you download them from a trusted URL and they look like they are really secure, even with hashing and things like that built-in. Lots of package managers don't support any way of verifying the identity of the one uploading the files, and even if they do we often import signing keys into our chain of trust without a moment of thought or we don't use the signing mechanism at all.

And with something like NPM packages you are likely to download another few dozen of other packages which you didn't even intent on downloading. You will probably run a lot of code there that could be doing all kinds of horrible things.

At least with `curl|bash` I get some feedback of where the code is originating from, what URL will I be downloading something from and is that some place that I can trust. At least I get somewhat of an identity verification (albeit very very weak) as long as I trust the owner of the site to protect it adequately from preventing unauthorized uploads.