HN user
Software Engineer working on Data Compression at Facebook.
[ my public key: https://keybase.io/felix; my proof: https://keybase.io/felix/sigs/mjJ1GvUGKTRmYImkgCu8Z04zn4AetQ7MsGOGRCSMh-A ]
SDDL (and the front-end task of reshaping data in general) is only one component of OpenZL. Once you have the streams, you can do all sorts of transformations to them that Zstd doesn't.
The OpenZL core supports arbitrary composition of graphs. So you can do this now via the compressor construction APIs. We just have to figure out how to make it easy to do.
Wanna hop over to https://github.com/facebook/openzl/issues/76?
Update: let's continue discussing genomic sequence compression on https://github.com/facebook/openzl/issues/76.
Yes. None of the algorithms under test used any hardware acceleration in the benchmarks we ran.
Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descriptions. I'll check them out.
Yes, definitely! Chunking support is currently in development. Streaming and seeking and so on are features we will certainly pursue as we mature towards an eventual v1.0.0.
Thanks @dang!
It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!).
Let us know how it goes!
We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people who, you know, didn't develop OpenZL. Your feedback is welcome!
In addition to open-sourcing the code today, we also published:
- A white paper: https://arxiv.org/abs/2510.03203
- A blog post: https://engineering.fb.com/2025/10/06/developer-tools/openzl...
- The documentation website: https://openzl.org/
In addition to the blog post, here are the other things we've published today:
Code: https://github.com/facebook/openzl
Documentation: https://openzl.org/
White Paper: https://arxiv.org/abs/2510.03203
Zstd has a similar-ish capability called "repetition codes" [0].
The first stage of Zstd does LZ77 matching, which transforms the input into "sequences", a series of instructions each of which describes some literals and one match. The literals component of the instruction says "the next L bytes of the message are these L bytes". The match component says "the next M bytes of the input are the M bytes N bytes ago".
If you want to construct a match between two strings that differ by one character, rather than saying "the next N bytes are the N bytes M bytes ago except for this one byte here which is X instead", Zstd just breaks it up into two sequences, the first part of the match, and then a single literal byte describing the changed byte, and then the rest of the match, which is described as being at offset 0. The encoding rules for Zstd define offset 0 to mean "the previously used match offset". This isn't as powerful as a Levenshtein edit, but it's a reasonable approximation.
The big advantage of this approach is that it doesn't require much additional machinery on the encoder or decoder, and thus remains very fast. Whereas implementing a whole edit description state machine would (I think) slow down decompression and especially compression enormously.
[0] https://datatracker.ietf.org/doc/html/rfc8878#name-repeat-of...
This is because Zstd's long-distance matcher looks for matching sequences of 64 bytes [0]. Because long matching sequences of the data will likely have the newlines inserted in different offsets in the run, this totally breaks Zstd's ability to find the long-distance match.
Ultimately, Zstd is a byte-oriented compressor that doesn't understand the semantics of the data it compresses. Improvements are certainly possible if you can recognize and separate that framing to recover a contiguous view of the underlying data.
[0] https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zs...
(I am one of the maintainers of Zstd.)
Awesome work!
I get exactly that green cast and muted color range off of my flatbed scans (Epson v800). This is a really intriguing path to fixing them I hadn't considered.
It seems like the writeup here doesn't specify what you're using for the actual imaging? A flatbed scanner? A camera?
That would only work in the case that the camera is fixed on a tripod and has a long period of stable / rigid pointing before the exposure during which to collect this data. This is sometimes the situation in which image stabilization is used. (But if you can be that stable for that long on a tripod, you may not actually need image stabilization.)
By far the more common case for image stabilization is one in which the photographer is hand-holding the camera and may not frame the subject until the moment before the exposure begins. The camera movement will likely be several orders of magnitude (~4 to 7) larger than the drift that you want to measure. A low pass filter will tell you nothing at all.
At a certain point we can just start using guide stars [0].
Probably referring to this: https://futurism.com/the-byte/piece-falls-off-boeing-starlin...
You're saying that a system that can recognize flaws in the alignment imposed on it can reject that alignment, but that doesn't follow.
Sure, humans act against their own interests all the time. Sometimes we do so for considered reasons, even. But that's because humans are messy and our interests are self-contradictory, incoherent, and have a fairly weak grip on our actions. We are always picking some values to serve and in doing so violating other values.
A strongly and coherently aligned AI would not (could not!) behave that way.
Sure, but Gödel encoding is pretty much purely a theoretical exercise. I'm not sure anyone anywhere has ever practically manipulated Gödel-encoded expressions in a useful way. His original scheme also has the problem that prime factorization is rather computationally challenging--it is after all the basis of the RSA cryptosystem.
Whereas Arithmetic encoding is actually practical, extensively used, and a direct analogue to the stick.
1. There are several implementations of Zstandard other than the "reference" libzstd that we maintain, tagged as "Ports" in the table here [0]. You are of course welcome to create or fork your own. We will happily take a PR adding yours to the table.
2. The format spec is an RFC [1]. It's frozen. We're not going to pull the rug out from under you somehow (and why would we??).
3. I think we do a pretty good job accepting contributions. But again, if we aren't, you can always fork it.
All in all, we've worked pretty hard to make zstd universal--portable, stable, flexible, robust, etc. I guess I don't really have a point... I guess I hope you reconsider your rejection of zstd? It's pretty cool.
Indeed--at this point zstd is the product of something like 25 man-years of improvements and optimizations.
What differences do you have in mind? Zstd is largely compatible as a drop-in replacement for those commands. There are differences, but it's our goal to minimize them.
There's also <>, which I (en-US) call "angle brackets".
No widely deployed compressors are PAQ-like. The speed-up anticipated by the OP never materialized.
Good write-up! I applaud your use of dictionary compression! We've done a lot of work to Zstd to make it a powerful tool that opens doors to new and better compression integrations.
One correction: the "dictionary" in Zstd is sort of a misnomer. It's (mostly[0]) not a data structure. It's simply a concatenation of substrings that occurred frequently in the samples provided to the trainer. When you compress with a dictionary, that content is used as a prefix against which matches can be made (as if you were doing streaming compression).
This is an important observation because it allows other patterns of dictionary compression. Zstd-trained dictionaries can be used with other LZ77 algorithms (like LZ4) and they will be useful there as well. Or, because dictionaries can be totally unstructured content, you can use one piece of content directly as a dictionary for another. This is useful for example for use cases like you mention, where you store multiple versions of a document. You can compress one version using another as a dictionary, which produces a compressed object which represents the delta.
One other note is that I know of at least one other database that uses dictionary-based Zstd to compress chunks: RocksDB[1].
[0] Zstd-produced dictionaries do actually have a structured header (described here: https://datatracker.ietf.org/doc/html/rfc8878#section-5), but the bulk of the dictionary is unstructured.
[1] https://github.com/facebook/rocksdb/wiki/Dictionary-Compress...
They don't actually turn off though. It just turns the display off. They need to continuously sense position since digital calipers measure movement increments rather than absolute position. That's why if you take the battery out and put it back in it will prompt you to re-zero the calipers.
Amusing video explanation: https://www.youtube.com/watch?v=KnDype-j3hk
As someone on the Zstd team, I'm always happy to see it on HN! I'm curious though what motivates the submission?
Yes. For example, RocksDB uses zstd dictionaries: https://rocksdb.org/blog/2021/05/31/dictionary-compression.h...
Two points worth noting:
1. Gzip is not a suitable compressor for this use case, because it's limited to a 32KB window. So the input can only be correlated with the last 32KB of the reference texts.
2. You can save a great deal in computation by avoiding recompressing the reference texts over and over and over. Some compression algorithms support checkpointing the compression state so that it can be resumed from that point repeatedly ("dictionary-based compression", which is a distinct capability from just streaming compression, which generally can only be continued once).
I would personally shill for using Zstandard [0] instead for this purpose. Although I should disclose my bias: I'm a developer of Zstd. A few salient facts:
1. Zstd supports very large windows (up to 128MB, or up to 2GB in long mode).
2. Zstd is much faster than zlib.
3. Zstd has well-developed support for dictionary-based compression.
4. Additionally, it has a dictionary trainer that can reduce a corpus of reference documents to a compact summary document that aims to capture as much of the content as possible of the reference corpus. [1]
5. It has (more than one) python binding available. [2][3]
[0] https://github.com/facebook/zstd
[1] https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L40
Glad to hear it! It's a pretty hefty single file, so it probably won't be qualifying for https://github.com/nothings/single_file_libs anytime soon... but hopefully people find it useful nonetheless.