7-Zip for Windows can now use more than 64 CPU threads for compression

by doener • 1 year ago

287 212 1 year ago

avidiax• 1 year ago

Why was there a limitation on Windows? I can't find any such limit for Linux.

lmm• 1 year ago

Seems like this is a general Windows thing per https://learn.microsoft.com/en-us/windows/win32/procthread/p... - applications that want to run on more than 64 CPUs need to be written with dedicated support for doing so.

lofties• 1 year ago

Windows has a concept of processor groups, that can have up to 64 (hardware) threads. I assume they updated 7zip to support multiple processor groups.

okanat• 1 year ago

Every operating system, that's relevant in 2025, needs that concept. It is called NUMA. At some point you cannot model the system ignoring the memory affinity / closeness to the cores.

Modern AMD CPUs are literally consist of core groups on chiplets. It is better for an OS to make decisions / expose APIs for cores that are physically so far away from each other that moving data back-and-forth over the RAM, system bus or interconnect has significant time penalties.

dwattttt2• 1 year ago

The linked Processor Group documentation also says:

Applications that do not call any functions that use processor affinity masks or processor numbers will operate correctly on all systems, regardless of the number of processors.

I suspect the limitation 7zip encountered was in how it checked how many logical processors a system has, to determine how many threads to spawn. GetActiveProcessorCount can tell you how many logical processors are on the system if you pass ALL_PROCESSOR_GROUPS, but that API was only added in Windows 7 (that said, that was more than 15 years ago, they probably could've found a moment to add and test a conditional call to it).

dspillett• 1 year ago

It isn't just detecting the extra logical processors, you have to do work to utilise them. From the linked text:

"If there are more than one processor group in Windows (on systems with more than 64 cpu threads), 7-Zip distributes running CPU threads across different processor groups."

The OS does not do that for you under Windows. Other OSs handle that many cores differently.

more than 15 years ago, they probably could've found a moment to add and test a conditional call to it

I suspect it hasn't been an issue much at all until recently. Any single block of data worth spinning up that many threads for compressing is going to be very large, you don't want to split something into too small chunks for compression or you lose some benefit of the dynamic compression dictionary (sharing that between threads would add a lot of inter-thread coordination work, killing any performance gain even if the threads are running local enough on the CPU to share cache). Compression is not an inherently parallelizable task, at least not “embarrassingly” so like some processes.

Even when you do have something to compress that would benefit for more than 64 separate tasks in theory, unless it is all in RAM (or on an incredibly quick & low latency drive/array) the process is likely to be IO starved long before it is compute starved, when you have that much compute resource to hand.

Recent improvements in storage options & CPUs (and the bandwidth between them) have presumably pushed the occurrences of this being worthwhile (outside of artificial tests) from “practically zero” to “near zero, but it happens”, hence the change has been made.

Note that two or more 7-zip instances working on different data could always use more than 64 threads between them, if enough cores to make that useful were available.

silon42• 1 year ago

Maybe WaitForMultipleObjects limit of 64 (MAXIMUM_WAIT_OBJECTS) applies?

An ugly limitation on an API that initially looks superior to Linux equivalents.

monocasa11• 1 year ago

A lot of synchronization primitives in the NT kernel are based on a register width bit mask of a CPU set, so each collection of 64 hardware threads on 64 bit systems kind of runs in its own instance of the scheduler. It's also unfortunately part of the driver ABI since these ops were implemented as macros and inline functions.

Because of that, transitioning a software thread to another processor group is a manual process that has to be managed by user space.

zik• 1 year ago

Wow. That's surprisingly lame.

whalesalad• 1 year ago

Windows is a terrible operating system.

xxs6• 1 year ago

WaitForMultipleObjects is limited to 64... since forever.

d332• 1 year ago

I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.

sammy2255• 1 year ago

https://github.com/mcmilk/7-Zip-zstd

abhinavk• 1 year ago

https://github.com/M2Team/NanaZip

It includes the above patches as well as few QoL features.

d332• 1 year ago

Thanks! Any ideas why it didn't get merged? Clearly 7-Zip has some development activity going on and so does this fork...

rf15• 1 year ago

Not that many people care about zstd; I would assume most 7-zip users care about the convenience of the gui.

jorvi2• 1 year ago

.. but 7-zip has a pretty terrible GUI?

Hence why PeaZip is so popular, and J-Zip used to be before it was stuffed with adware.

yapyap• 1 year ago

if by gui u mean the ability to right click a .zip file and unzip it just through the little window that pops up ur totally right. At least that + the unzipping progress bar is what I appreciate 7zip for

Beretta_Vexee3• 1 year ago

I just hope that the recipient will be able to open the file without too much difficulty. I am willing to sacrifice a few megabytes if necessary.

arp2422• 1 year ago

It's been a long time since I used Windows, but back in the day I used 7-Zip exactly because it could open more or less $anything. That's also why we installed it on many customer computers.

On Linux bsdtar/libarchive gives a similar experience: "tar xf file" works on most things.

quickthrowman• 1 year ago

I use the right click context menu to run 7zip, why would you open a GUI?

KronisLV• 1 year ago

That's basically me! I really like 7-Zip because it opens most archive formats I have to work with and also the .7z format has pretty good compression for the stuff I want to store longer term.

snickerdoodle12• 1 year ago

That's why 7zip should support it. People care about the convenience of the GUI and we all benefit from better compression being accessible with a nice GUI.

cm2187• 1 year ago

in fact this is the first time I even hear about it, and I am semi-IT litterate. The prevalence of a compression standard is about how ubiquitous it is. For that one, I would vote "not even on the radar yet".

m-schuetz2• 1 year ago

Being a bit faster or efficient won't make most people switch. 7z offers great UX (convenient GUI and support for many formats) that keeps people around.

rat9988• 1 year ago

If anything, the gui and ux is terrible compared to winrar.

dikei2• 1 year ago

I use ZSTD a ton in my programming work where efficiency matters.

But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

cesarb2• 1 year ago

Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification).

Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the ZIP format features. Which is why formats which use ZIP as a base (Java JAR files, OpenDocument files, new Office files) standardize such a subset; but for general-purpose ZIP files, there's no such standard.

(I have encountered such ZIP files in the wild; "unzip" can't decompress them, though p7zip worked for these particular ZIP files.)

notepad0x903• 1 year ago

I don't know about, had a dicey situation recently where powershell's compress-archive couldn't handle archives >4GB and had to use 7zip. it is more reliable and you can ship 7za.exe or create self-extracting archives (wish those were more of a thing outside of the windows world).

sidewndr469• 1 year ago

What are you compressing with zstd? I had to do this recently and the "xz" utility still blows it away in terms of compression ratio. In terms of memory and CPU usage, zstd wins by a large margin. But in my case I only really cared about compression ratio

jart• 1 year ago

Use the pigz command for parallel gzip. Mark Adler also has an example floating around somewhere about how to implement basically the same thing using Z_BLOCK.

mrWiz2• 1 year ago

My main use case for 7z is bypassing corporate filters that block ZIPs from being sent.

psyclobe• 1 year ago

zip is such a shit standard, hell there are parts of it that are still undocumented and sharing documents between system zip implementations across mac and windows sometimes fails.

Beretta_Vexee3• 1 year ago

You are looking for 7-Zip Zstd: https://github.com/mcmilk/7-Zip-Zstd

I don't know what your use case is, but it seems to be quite a niche.

zx2c4• 1 year ago

I was curious upon seeing this and found the thread where its inclusion was turned down: https://sourceforge.net/p/sevenzip/discussion/45797/thread/a...

birksherty4• 1 year ago

I use this one too. This is the best one out of all the alternative 7zip with zstd. UI is same, has all the options and faster than those for zstd in my comparisons.

pjmlp2• 1 year ago

As long as it does a better job than whatever Windows team packs into the OS, they're safe.

Even on latest Windows 11 takes minutes to do what 7-Zip does in seconds.

Goes to show how good all those leetcode interviews turn out.

conkeisterdoor• 1 year ago

Glad I'm not the only one who feels this way. WinZip is a slow and bloated abomination, especially compared to 7-Zip. The right-click menu context entry for 7-Zip is very convenient and runs lightning fast. WinZip can't compete at all.

jccalhoun• 1 year ago

Since Windows 11 incorporated libarchive back in October 2023 there is less reason to use 7-zip on windows. I would be surprised if any of my friends even know what a zip file is let alone zstd.

rs186• 1 year ago

If you ever try to extract an archive file of several gigabyte size with hundreds of thousands of files (I know, it's rare), the built-in one is as slow as a turtle compared to 7z.

privatelypublic• 1 year ago

It already has- look up nanazip

xxs6• 1 year ago

There are lots of 7zip alike with zstd support (it's a plugin effectively). On [corporate] Windows NanaZip would be my choice as it's available in Windows store.

on anything else - either directly zstd or tar

Night_Thastus4• 1 year ago

7-zip is the de-facto tool on Windows and has been for a long time. It's more than fast and compressed enough for 99% of peoples use cases.

It's not going anywhere anytime soon.

The more likely thing to eat into its relevance is now that Windows has built-in basic support for zipping/unzipping EDIT: other formats*, which relegates 7-zip to more niche uses.

izzydata2• 1 year ago

Is there something different about the built in zip context menu functionality now than before? I'm pretty sure you could convert something to a zip file since forever ago by right clicking any file.

malfist2• 1 year ago

Windows has had built in zip/unzip since vista. 7zip is far superior (and the install base proves that)

Bender• 1 year ago

7-zip is the de-facto tool on Windows and has been for a long time.

Agreed. The only thing I think it has been missing is PAR support. I think they should consider incorporating one of the par2cmdline forks and porting that code to Windows as well so that it has recovery options similar to WinRAR. It's not used by everyone but that should deprecate any use cases for WinRAR in my opinion.

_zagj• 1 year ago

7-zip, through its .7z format, also supports AES encryption. I'd argue it's probably the easiest way to encrypt individual file archives that you need to access on both Windows and Linux. I have a script I periodically run that makes an encrypted .7z archive of all of my projects, which I then upload for off-site backup. (On-site, I don't bother encrypting.)

marcellus23• 1 year ago

Why are they not adopting ztsd?

aquir3• 1 year ago

7-zip is one of the software that I miss since I’ve moved to macOS

DeepSeaTortoise• 1 year ago

How about PeaZip?

aquir3• 1 year ago

I've used PeaZip in the past but only on Windows, I was not aware that a MacOS version exists! I'll give it a try. Cheers

MYEUHD• 1 year ago

If you're talking about the program you use in the terminal, you can install it via homebrew

immibis2• 1 year ago

No, the GUI. 7-zip integrates well with the shell: select a group of files, right click -> make zip file, and so on. Or right-click a zip file and select extract. If you're accustomed to Linux you might not know what they're talking about.

TortoiseGit (and TortoiseSVN) are similarly convenient. Right click a folder with an SVN repo checked out, and select "SVN update". Right-click an empty space, and select "SVN checkout". SVN was the main distribution method for some modding communities before things like Steam Workshop and Github, specifically because TortoiseSVN made it so convenient. Checkout into your addons folder, and periodically update. What could be simpler?

portaltonowhere• 1 year ago

Keka is also really nice!

https://www.keka.io/

aquir3• 1 year ago

Never heard of it, I'll give it a try!

yeah879846• 1 year ago

Imagine voluntarily moving to mac.

ltbarcly35• 1 year ago

Wow, a program that doesn't matter anymore has been very very minimally enhanced on a platform that doesn't matter anymore, benefitting the 7 users that have more than 64 real cores with Windoes and are regularly compressing archives so large that it doesn't drastically reduce the compression ratio to split it into more thsn 64 sections.

Posting this link to hn has consumed more human potential than the thing it is describing will save up to the end of time.

tobinc2• 1 year ago

A 1% speed improvement for 1% of 7zip users is several times more productive than your comment.

ltbarcly35• 1 year ago

This will not deliver your stated threshold.

starkrights• 1 year ago

a program that doesn’t matter anymore

The rest of this comment has, though gratuitously snarky, a point, but I don’t think claiming that 7zip is irrelevant as an independent statement is even remotely coherent.

ltbarcly35• 1 year ago

tar.zstd is superior in basically every way, open, and portable

esafak• 1 year ago

It's funny 'coz it's true! But no slight against 7zip; it was good for its time.

marcodiego• 1 year ago

https://xkcd.com/619/

lihaciudaniel• 1 year ago

7zip has been the greatest usage for limbo x86 on mobile.

You just termux qemu-utils convert your qcow2 partitions to IMG and 7zip can read IMG file

Try yourself to see you can extract from your emulated windows

leecarraher• 1 year ago

I've used pbzip2 which takes the same parallel blocked compression approach 7zip seems to be taking (using AI's analysis of the changes). Theoretically the compression is less efficient, but i haven't noticed a difference in practice.

izzydata2• 1 year ago

This may or may not be a relevant question, but does the terminology of "zip" have the same origin as the zip disk drive?

malfist2• 1 year ago

No. Zip format significantly predates the zip disk.

vpShane• 1 year ago

The ZIP file format was created in 1989 by Phil Katz for his PKZIP software.

The name 'ZIP' was chosen to convey the idea of compressing files into a smaller size, making them 'zip' together for easier storage and transfer. The term suggests speed and efficiency in compressing and uncompressing files.

Like zipping something up in a bag. You unzip the bag and have all of its contents, zip it up and all of its contents are in one bag (file).

7zip has always been the defacto-best, I'm sure there's other decent compression algorithms but 7z has always taken the cake for me.

ninjis• 1 year ago

I had initially migrated to NanaZip, but with Windows natively supporting the 7z format now, I'm not sure it's needing anymore.

fabiensanglard• 1 year ago

How does that work? You cannot write to disk before you know the compressed size. Or if you do you can use a data descriptor but you cannot write concurrently.

I guess they buffer the compressed stream to RAM before writing to zip. If they want to keep their zip stable (always the same output given the same input), they also need to keep it a bit longer than necessary in RAM.

1W6MIC49CYX9GAP• 1 year ago

I think you get different compressed files depending on how many threads you use to compress

amelius• 1 year ago

Maybe Windows allows a file to be constructed quickly from parts that are not necessarily multiples of the blocksize. Maybe they have a fast api for writing multiple files and then turning it into a single file. POSIX doesn't allow that, but it is quite old.

7-Zip for Windows can now use more than 64 CPU threads for compression

Related Stories