Jennifer Aniston and Friends Cost Us 377GB and Broke Ext4 Hardlinks

https://blog.discourse.org/2026/04/how-jennifer-aniston-and-friends-cost-us-377gb-and-broke-ext4-hardlinks/

by speckx • 3 months ago

47 31 3 months ago

blog.discourse.org

replooda• 3 months ago

In short: Deduplication efforts frustrated by hardlink limits per inode — and a solution compatible with different file systems.

UltraSane7• 3 months ago

The real problem is they aren't deduplicating at the filesystem level like sane people do.

otterley4• 3 months ago

From the article:

[W]e shipped an optimization. Detect duplicate files by their content hash, use hardlinks instead of downloading each copy.

dj_rock• 3 months ago

We were on a break...of your filesystem!

bravetraveler3• 3 months ago

As is always the case, short vs long term... but I think I'd put effort into migrating to a filesystem that is aware of duplication instead of trying to recreate one with links [while retaining duplicates, just fewer].

Effectiveness is debatable, this approach still has duplication. An insignificant amount, I'll admit. The filesystem handling this at the block level is probably less problematic/prone to rework and more efficient.

edit: Eh, ignore me. I see this is preparing for [whatever filesystem hosts chose] thanks to 'ameliaquining' below. Originally thought this was all Discourse-proper, processing data they had.

ameliaquining2• 3 months ago

Discourse is self-hostable; they can't require their users to use a filesystem that supports deduplication. (Or, well, they could, but it would greatly complicate installation and maintenance and whatnot, and also there would need to be some kind of story for existing installations.)

bravetraveler3• 3 months ago

Fair, I am/was confused by the hosting model and presentation. This is a nice User-preparation/consideration, I guess. I still maintain a backup filesystem unaware of duplication at the block level is a mistake.

I completely overlooked the shipping-of-tarballs. Links make sense, here. I had 'unpacked' and relatively-local data in mind. Absolutely would not go as far to suggest their scheme pick up 'zfs {send,receive}'/equivalent, lol.

uticus• 3 months ago

And I thought this was a reference to a Win95 problem https://www.slashgear.com/1414245/jennifer-aniston-matthew-p...

mingus88• 3 months ago

Yeah Block level dedupe has been an industry standard for decades. Tracking file hashes? Why?

And I see above that this is a self-hosted platform and I still don’t get it. I was running terabytes of ZFS with dedupe=on on cheap supermicro gear in 2012

zulux• 3 months ago

File hashes are great to get two systems to work together to dedupe themselves. I have a Windows backup that sends hashes to a backup server, so we don't back up crud we already have.

UltraSane7• 3 months ago

This makes them look rather incompetent. Storing the exact same file 246,173 times is just stupid. Dedupe at the filesystem level and make your life easier.

trixn86• 3 months ago

The Problem. The fix. The Limit.

Is it just me or is everybody else just as fed up with always the same AI tropes?

I've reached a point where I just close the tab the moment I read a headline "The problem". At least use tropes.fyi please

colejohnson66• 3 months ago

Doesn’t read like AI to me

snickerbockers• 3 months ago

Let that sink in.

otterley4• 3 months ago

Another reason to use XFS -- it doesn't have per-inode hard link limits.

(Some say ZFS as well, but it's not nearly as easy to use, and its license is still not GPL-friendly.)

burnt-resistor3• 3 months ago

xfs on mdraid is what I use on my homelab NAS across several giant RAID arrays. While it lacks some integrity and CoW features, it's really, really stable. I had ZoL ZFS troubles that the maintainers shrugged off requiring transferring everything to another volume.. so I won't ever use or recommend ZFS unless it's Sun-Oracle.

niobe• 3 months ago

Completely Claude written FWIW. I recongise the style.

Jennifer Aniston and Friends Cost Us 377GB and Broke Ext4 Hardlinks

Related Stories