Xz format inadequate for long-term archiving (2016)

http://lzip.nongnu.org/xz_inadequate.html

by UkiahSmith • 7 years ago

92 58 7 years ago

lzip.nongnu.org

brianpgordon• 7 years ago

Previous discussions:

https://news.ycombinator.com/item?id=12768425

https://news.ycombinator.com/item?id=16884832

LinuxBender2• 7 years ago

If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic.

    tar cvfJ ./files.tar.xz /some/dir

zamalek• 7 years ago

You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term) can easily render the entire xz file corrupt.

This has nothing to do with xattrs/etc.

LinuxBender2• 7 years ago

Yes, I am totally on auto-pilot today. I'm used to a different article that gets re-posted often about xz and my browser blocks non-https sites so I assumed it was that other article.

That said, I use xz in automation that compresses files on one end and decompresses on the other. I've not had any file corruption thus far. checksums always match. Hopefully the author has submitted bug reports and ways to reproduce.

imiric• 7 years ago

I've been using tar+pixz+par2 for backups for a while now, but this article still worries me.

What can I replace pixz with that compresses as well and keeps the indexing functionality? I'd like to avoid zstd because Facebook.

SEJeff• 7 years ago

This should have (2016) in the title.

Thoreandan• 7 years ago

and "from the author of lzip, a competing lzma library that never went viral".

Welcome to the Better Technology that Shoulda Made It bench. Your seat's over there next to OS/2, BeOS, and OpenGenera.

nkoren• 7 years ago

Amiga forever!!!!!!

microcolonel2• 7 years ago

"3 Then, why some free software projects use xz?"

Because the files are usually smaller than gzip, with faster decompression than bzip2, and the library is available on most systems.

h1d• 7 years ago

Archiving for distribution and backups are very different things. You don't care if some app distribution compressed file gets corrupted, you just compress again but your compressed backup files usually don't have much source of reference.

I wouldn't use any unreliable format for backups. I picked bzip2 for stability and compression rate.

microcolonel2• 7 years ago

In my opinion, the compressor is not the right place to add data integrity mechanisms, especially since data integrity mechanisms only really apply to particular media. Data on hard drives don't get corrupted in the same way as data on TLC SSDs, and generally on the latter you're better off with redundancy and diversification, than with inline error correcting codes.

Honestly, I don't see why xz should have any of its own data integrity mechanisms whatsoever, except maybe a whole-archive CRC32 or similar.

Adamantcheese• 7 years ago

How about something like ZPAQ instead for archiving? Especially if you're doing backups and not a lot of the information is changing.

jordigh• 7 years ago

This guy used to go around GNU mailing lists (and others) trying to get us to use lzip.

https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html

https://lists.debian.org/debian-devel/2017/06/msg00433.html

It was a bit bizarre when he hit the Octave mailing list.

Eventually, people just wanted xz back:

http://octave.1599824.n4.nabble.com/opinion-bring-back-Octav...

AdmiralAsshat• 7 years ago

Everyone's gotta have their white whale, I guess.

esaym2• 7 years ago

Interestingly, since "recovery" is mentioned several times, I decided to test myself.

I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte.

The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and all).

Uncompressing the corrupt gzip jpeg file via zcat at all times resulted in an image file the same size as the original and could be opened with any image viewer although the colors were clearly off.

I never could recover the image compressed with bzip2. Trying to extract all the recovered blocks made by bzip2recover via bzcat would just choke on the single corrupted block. And the smallest you can make a block is 100K (vs 32K for gzip?). Obviously pulling 100K out of a jpeg will not work.

Though I'm still confused as to how the corrupted gzip file extracted to a file of the same size as the original. I guess gzip writes out the corrupted data as well instead of choking on it? I guess gzip is the winner here. Having a file with a corrupted byte is much better than having a file with 100K of data missing...

wereHamster• 7 years ago

Whether recovery leads to (almost) useable data depends on what byte you modify. It's entirely possible that a single corrupt byte in the compressed data leads to a single corrupt byte when uncompressed. When you are dealing with images you may not even notice that a single pixel is wrong. But it's also possible that you completely destroy the data such that the decompression algorithm can't even deal with it and has to give up.

chasil• 7 years ago

A decade and a half ago, I wrote an Oracle archived log that I had compressed with bzip2 to a DLT40 tape.

I recovered and uncompressed (without error) the log, then tried to apply it to a database recovery which rejected it as corrupt.

After several attempts to read the tape (amounting to dozens of hours), I finally put it in the original drive that wrote it and pulled the file to the remote recovery system - this worked.

I immediately began including PAR2 files on the tapes, so the restored contents could be verified and corrected.

I have my doubts that bzip2 is as sensitive to corruption as the author of asserts, but perhaps there have been improvements to the code since my misfortune.

gmueckl• 7 years ago

Your method is clearly flawed. Altering a single byte once is insufficient as a test unless you analyzed the structure of the compressed file first to see where the really important information is stored. It may well be that you just modified a verbatim string from the source data in the gzip case, but corrupted a bit of metadata about how the compressed data is structured in the bzip2 case. If you tried a different random bytes, the results might be reversed.

The proper test would be to iterate over every bit in the compressed file, flip it and try to recover. Then compute number of successful recoveries against the number of bits tested. Compression algorithms that perform similarly should gmhave similar likelyhoods that a single bit flip corrupts the entirety of the data.

esaym2• 7 years ago

I thought about that as well. I tried it three different times all with the same results.

asveikau3• 7 years ago

I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.

StavrosK2• 7 years ago

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

kipari• 7 years ago

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

asveikau3• 7 years ago

Seems easier to increase the amount of disks and address it at a low layer than to re-engineer all layers, all file formats, for corruption.

xoa2• 7 years ago

Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the principle of layers and graceful failure modes it's better if even sub-archives can handle some level of corruption without total failure, and from a long term perspective of implementation independence simpler/better specified is preferable, but that still doesn't seem to substitute for just having enough parity built in to both notice corruption and fully recover from it to fairly extreme levels.

matt-attack3• 7 years ago

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original executable is either not available or not able to be executed.

xoa2• 7 years ago

I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM.

Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?

ltbarcly34• 7 years ago

Why would they need to recreate it solely based on 'documentation'? It is open source, the source code is the documentation. It seems just as likely that the source would survive as it is likely that some complete technical documentation would survive. Maybe they wouldn't be able to compile it (probably they would be able to compile it, I don't see why they wouldn't have some kind of computer emulator available), but it's better than any other kind of documentation you could provide.

The src/ tree of xz is 335k (compressed with gzip). If you are worried future digital historians won't be able to figure out the xz format, throw a copy of the gzip'd source onto every drive you store archives on, it would basically be free and would almost guarantee they would have a complete copy of exactly what they would need to decompress the files.

shmerl• 7 years ago

xz-utils should implement parallel decompression already. pixz is doing it, but stock xz is not. Most end users benefit from faster decompression.

ltbarcly34• 7 years ago

No file format is perfect, I've been using xz for years and I can't think of a single issue I have had. The compression rate is dramatically better than gzip or bzip2 for many types of archives (especially when there is a large redundancy, for example when compressing spidered web pages from the same site you can get well over 99% size reduction compared to 70% reduction for gzip, which means using less than one 30th of the disk space).

Lately I have been using zstd for some things since it gives good compression and is much faster than xz.

This criticism of xz just seems nit picky and impractical, especially if you are compressing tar archives and/or storing the archives on some kind of raid which can correct some read errors (such as raid5).

mkj• 7 years ago

A bit of speculation here, but perhaps xz won over lzip because it has a real manpage?

lzip has the usual infuriating short summary of options with a "run info lzip for the complete manual". Also the source code repository doesn't even seem linked directly from the lzip homepage - technical considerations aren't the only thing that determines if software is "better", it also has to be well presented.

Xz format inadequate for long-term archiving (2016)

Related Stories