Should you remove duplicate files?

https://eclecticlight.co/2023/04/22/should-you-remove-duplicate-files/

by ingve • 3 years ago

62 61 3 years ago

iamnotsure• 3 years ago

I would prefer not to.

ggm• 3 years ago

If your fs implements block dedupe, no need. But, it's expensive to implement and from what I see no longer recommended on ZFS.

If your fs implements hard links, no need inside one mount point. But you can't hard link cross mount.

Copies can be good or bad. It depends.

ajsnigrutin• 3 years ago

Dedup on FS works, but hard links are useful only if you (and everyone else using the PC) is aware of the hard link (and never forgets about that). Otherwise it's "I can just mess with this file, I have a copy in that other folder for backup".. until you notice it's not an actual copy.

matheusmoreira3• 3 years ago

It's better to assume all files are potentially links that must be explicitly copied before any destructive operation. Software should be designed to always produce new copies by default and overwrite files only when explicitly requested.

counttheforks2• 3 years ago

I'd like copy-on-write hardlinks, or deduplicated blocks, that are made automatically on a schedule as to not incur the constant overhead of dedup, but still reap some of the benefits for archival.

Freaky• 3 years ago

Reflinks are basically CoW hardlinks — supported on Linux by btrfs, XFS, and I've seen some effort to bring it to ZFS.

DragonFlyBSD's HAMMER filesystem supports on-demand/scheduled dedup using the hammer(8) command, with configurable memory and runtime limits.

the84722• 3 years ago

https://manpages.debian.org/testing/duperemove/duperemove.8....

tatersolid• 3 years ago

NTFS deduplication works this way on Windows server since 2012, but I don’t think any of the Linux NTFS drivers support NTFS dedupe correctly.

soneil• 3 years ago

The behaviour is very different to hardlinks. The clone is a new file that uses the same blocks. When you modify the clone, writes are written to new blocks rather than the old ones.

So if I have a disk image that takes 1,000 blocks, and clone it - I now have two images that total 1,000 blocks. And if I modify one block in one image, I now have two different images that total 1,001 blocks. With a hardlink the modification would affect both images.

The way I visualise the difference, is that a directory entry points to a file entry, and a file entry points to a list of blocks. A hardlink is a new directory entry that points to an existing file entry. a linked clone is a new file entry that points to the same list of blocks.

the84722• 3 years ago

But, it's expensive to implement and from what I see no longer recommended on ZFS.

Btrfs (and more recently also XFS) supports on-demand, batch dedup which doesn't have any of the downsides of ZFS's always-active dedup. This is an implementation issue, not a categorical problem of block (or extent) dedup.

matheusmoreira3• 3 years ago

Btrfs is like an almost perfect file system. If only it had proper stable support for software RAID. It's weird that it still doesn't have it, I thought enterprises cared about that.

kevincox• 3 years ago

The ideal situation here would probably be to update any identical but not reflinked files to be reflinks. But if there is no API like you suggested then it may not be easy. Also ideally there would be an API to do this atomically please share the storage of these files assuming they are the same)

If Time Machine is tracking file contents for deduplication it would be really cool if it did this to the source files whenever it noticed that they were identical.

lizknope• 3 years ago

There are many programs out there that will find duplicates files. I use this one. It gives you the option of replacing the duplicates with either hard or symbolic links.

https://github.com/qarmin/czkawka

ukuina• 3 years ago

I have been looking for something like this, thanks for sharing!

From the project:

This name is not as bad as it seems, because I was also thinking about using words like żółć, gżegżółka or żołądź, but I gave up on these ideas because they contained Polish characters, *which would cause difficulty in searching for the project.*

Really dodged a bullet there.

Eisenstein• 3 years ago

Remember how it is pronounced and you can remember the spelling: Czkawka == Ch-kah-v-kah. Cz is 'ch' like Czech, and w is v. It sounds like a sneeze when you say it.

bloaf2• 3 years ago

I've used fdupes and refund for exact duplicate finding.

They work shockingly well on large libraries, I've managed to check millions of files consuming several TBs in a matter of hours.

synergy20• 3 years ago

I use fdupes and it worked well, but what is 'refund' here?

Szpadel• 3 years ago

I use btrfs + bees (https://github.com/Zygo/bees) for such cases

binkHN• 3 years ago

Back in the day I used to do this on file servers at the block level. Was valuable as storage was more limited, IT would often have duplicated large deployment images and users would often have duplicate copies of the same files. Multiple one user by ~1000 and you have significant space savings.

vinaypai• 3 years ago

"Finding and removing duplicate files has long been one of the mainstays of Mac housekeeping."

What are Mac users doing that makes finding and removing duplicate files such a regular task? I don't use a Mac, but I can honestly say I've never given second thought to duplicate files. I'm sure there are some here and there but my hard drive isn't filling up with duplicates of large files.

nopakos• 3 years ago

Space is precious! Soldered SSDs can't be replaced, so we are stuck with the size we could afford at the time we bought our MacBooks.

graypegg2• 3 years ago

If I had to guess, it might have to do with the Mac standard of “save on close window” which is a vestige of an old apple desktop metaphor where every window represents a file somewhere.

So you end up with a lot of duplicates from either trying to edit a read only document in some apps (which spawns an editable copy) or accidentally hitting duplicate (generate a copy of the current file in the same directory) which is the mac equivalent of “Save As”. It’s on you the user to move the new file where you want it.

dahart2• 3 years ago

There’s nothing special about Macs or Mac users, this article is just in a Mac-focused blog, so it’s not implying anything about other OSes. Deduping is a mainstay of computer housekeeping on any OS, for some people, though maybe not you. There are lots of workflows that end up with duplicates, but certainly not everyone uses them.

Here are just a few ways I’ve ended up with enough dupes to want to dedupe, which has happened on Windows and Linux for me as frequently as on Mac: Backing up mobile phones, using version control for multiple repositories that share files, losing access to repository servers and archiving or backup up the files, forking an old archived folder and making modifications to it that I want to keep, not to mention homegrown schemes & scripts for archiving and backing up when I either don’t want to pay for backup software or can’t even find anything appropriate to all my needs (cross platform, saving to a NAS, shared with my family, etc.)

actuallyalys• 3 years ago

Yeah, as someone who sometimes uses macOS, I was confused by this initially but I believe you have the right interpretation.

epistasis4• 3 years ago

I do use a Mac. We spend a lot of time on maintenance. Here's my Saturday schedule:

* 7:00 AM find and remove duplicate files

* 8:00 AM reorganize the 2D layout of file icons within folders for maximum visual pleasure

* 9:00 degauss the LCD (really frustrated that you need third party software for this since the CRT days)

* 10:00 find and remove duplicate files

And so on. Finding duplicate files is just as essential as monitor degaussing, but you won't find this listed in the manual. You have to go to user forms where people will convince you of the need and get you to adopt procedures like the above to keep your machine at top performance.

graypegg2• 3 years ago

Heheh, don’t forget to reset the SMC, it collects too much entropy overnight so if you want deterministic behaviour you’ll need to make sure to dump its contents to /dev/null.

ChumpGPT• 3 years ago

Just install Gentoo

taneq• 3 years ago

How do you ever find time to record a video of yourself on a white background talking about how much better you are than PC users? ;)

lowkeyokay• 3 years ago

I think a simple explanation could be that it is slightly harder (mentally) to move than copy a file. Windows move: ctrl + x Mac: cmd + option + v In every other program, cut/paste is the same as windows. That’s confusing - at least as a former Windows user. That said, I had many duplicate files on Windows too.

DangitBobby2• 3 years ago

Dragging from one folder into another is also a copy operation on MacOS instead of a move like it is on Windows.

joshspankit• 3 years ago

One of the biggest ones from past days was media files and especially photos. They are large, users like to arrange/rearrange, compress, and manually back them up (or “back them up” depending on the knowledge level of the user). They are also rarely modified after being initially tweaked.

And on the other side: Apple computers typically don’t have upgradeable hard drives so for anyone who takes photos/video it’s an inevitable countdown to zero free space.

ComodoHacker• 3 years ago

Finally, why don’t some housekeeping utilities recognise clones? That’s because there’s no official way to tell them apart.

Is it intentional?

tacostakohashi• 3 years ago

If there were any official way to tell them apart, would they still be clones?

dahart2• 3 years ago

I used to dedupe a lot to save space, on any OS, Mac, Windows, and Linux. But one of the main reasons I end up with many dupes is various archiving and backup schemes I use, and I’ve realized deduplicating reduces backup safety. If the source file gets somehow corrupted for any reason, all copies are lost. I’m sure a deduping file system is slightly better here than a deduping utility and/or my own scripts, but still having only one copy of the bits is more precarious than saving multiple copies.

js2• 3 years ago

In my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterprise 4500s, to NetApp NFS clusters, to a TrueNAS box in my basement with terabytes of space.

I've been a sysadmin managing SunOS and Linux systems at scale.

I've had personal machines running a variety of OSs: FreeBSD, Linux, Windows, Mac OS, OS X, to say nothing off all the different filesystems involved across those OSes.

In all that time, I can't think of a single time I've ever bothered to de-duplicate files.

Once upon a time, you used to occasionally need to defragment some file systems. But it's been more than two decades since I've even thought about defragmenting. Every modern filesystem just handles that automatically.

What even is the use case where folks end up with enough duplicate files that it's worth worrying about?

dghlsakjg• 3 years ago

I can think of exactly one case!

In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users).

I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive.

Then you would dedupe. That’s the last time I thought about deduping

jeroenhd• 3 years ago

I never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders.

It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).

Strom• 3 years ago

I'll give you two use cases, one for non-technical and one for technical people.

Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs.

Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.

BrandoElFollito• 3 years ago

For the second one I would love to have some kind of centralized node_modules used across my projects. I am pretty sure that a large part is duplicated, just for the fact that I use the same framework everywhere

jamesu• 3 years ago

One time I had to consolidate a late family members personal files spread across several types of media which ended up totalling several TBs. None of it was organized, so there were duplicates everywhere. Probably only 20% of the data was non-duplicated stuff I genuinely wanted to keep.

firstlink• 3 years ago

Every modern filesystem just handles that automatically.

Do they? I always assumed they just try to avoid fragments but ignore them once created, and rely on bigger faster disks to hide any problems this may cause.

didgetmaster• 3 years ago

So if you change one byte in a clone, then it will just update a single block in the changed file while leaving all the other clone's data streams unchanged. But what happens if you add (or delete) a byte near the front of a 10GB file? Does that cause the entire data stream to be written, since the rest of the data must be shifted up or down?

jerf• 3 years ago

Yes, but precisely because of that there is a bit of resistance to people using that operation on large files, so it is not a common thing. It wasn't that long ago that rewriting a 10GB file from top to bottom could easily take over three minutes (50MB/s write speeds to a spinning rust hard drive = 200 seconds, and if we account for also reading the file it will be even slower), so people tend to learn to program without asking for that operation.

While there are many people who weirdly (to me at least) grotesquely overestimate the value of deduplication, it is legitimately useful because we tend not to want to completely rewrite large files very often like that, so even modifications to that file are very likely to either be appends or modifications of something in the middle without shifting the entire file.

CharlesW• 3 years ago

For folks who know they have duplicate photos, but also have a photo library large enough that it's hard to manually de-dupe them, I can strongly recommend PhotoSweeper (no affiliation). https://overmacs.com/

What's unique about PhotoSweeper is that it gives you deep control over what you mean by "dupe" and supports pro image formats like RAW and DNG. For example, I was able to automatically de-dupe byte-for-byte equivalents, then manually evaluate dupes with the same image data but slightly different metadata. Of course, it can do things like find dupes at different resolutions and keep the highest-quality version, etc. It saved me a ton of time.

euroderf• 3 years ago

I would like to know if there are similar utilities for audio files.

Should you remove duplicate files?

Related Stories