Announcing Zstandard in Rust

https://trifectatech.org/blog/announcing-zstandard-in-rust/

by jmillikin • 2 months ago

63 8 2 months ago

trifectatech.org

ikawe• 2 months ago

It’d be nice if they at least mentioned the existing pure rust implementation.

https://github.com/KillingSpark/zstd-rs

And better yet if they compared it explicitly in their “why” section. As it is, I’m left guessing.

merlincorey• 2 months ago

The "Why" section exclusively opens with a reference to zstd-rs:

Why though?
Using zstd in Rust is already supported via the zstd crate, so why bother with a whole new implementation?

Separately, I'd say their end goal seems to be replacing the C library dependency in Rust's zstd-rs:

We have our own fork of the zstd that uses libzstd-rs-sys instead of the C library. We'd like to upstream this at some point.

soulbadguy• 2 months ago

They seems to be referering to https://github.com/gyscos/zstd-rs which looks like a wrapper around the C library. https://github.com/KillingSpark/zstd-rs. is a pure rust implementation

zamalek• 2 months ago

That is an absolutely massive amount of unsafe, is the plan to reduce it over time? The benefits of the Rust build system are reason alone to use this (as a direct competitor to the C version), but claims about memory safety are suspect no matter how confident the authors are about correctness.

mswphd• 2 months ago

they're using c2rust. It transpiles c -> rust "directly", e.g. ints map to libc::c_int, pointers map to pointers, array access map to dereferencing pointers at an offset, etc.

Typically the suggested way to use it is to obtain "bad" rust directly from it (but which still works etc), and then make that "bad" rust more idiomatic. I don't remember any publicly visible success stories for this though. The current Bun rewrite is arguably of this form, but not exactly a "success story" in producing idiomatic, safe rust. Other large rewrites that were well-publicized/docoument, e.g. the fish shell from c++ -> Rust, took a different approach (factor into modules/rewrite each module by hand).

https://fishshell.com/blog/rustport/

tnelsond42• 2 months ago

I was using the original zstd rust crate for my web app, but I couldn't get my module any smaller than 150kb, but when I switched to C for web app I was able to do a whole lot of optimizing and culling until I got it down to 39kb.

I wanted to like Rust, but I was using unsafe code for performance and such anyway that it made more sense to switch to C.

elevation• 2 months ago

I got it down to 39kb.

If you're counting KBs and you have to include zstd yourself (a reverse proxy isn't handling it for you) does that mean you're targeting an embedded device? Are you supporting TLS at that size?

tnelsond42• 1 month ago

39kb includes the zstd decoder which is statically compiled into it. I'm not targeting microcontrollers, but I am targeting smartphones in areas with bad internet connections, so every kilobyte saved is nice. I also just like to optimize things and make them minimal while still being powerful. I don't handle tls, GitHub pages handles all that, I'm just using it to host a static pwa.

Announcing Zstandard in Rust

Related Stories