Hard problems in social media archiving

https://alexwlchan.net/2025/hard-problems-in-social-media-archiving/

by surprisetalk • 5 months ago

54 11 5 months ago

zoobab• 5 months ago

HTTP is not designed for mirroring.

FTP was easy to mirror with "lftp> mirror -p".

Easy mirroring and archive level maintenance (let's say the network always maintain 3 copies at minimum) should be built-in the "social media" protocols.

binarykult• 5 months ago

Well if only we still can archive Instagram full-profiles, for example ...

pjc50• 5 months ago

Good that they actually raise the question of users not wanting to be archived. I think the semi-ephemerality of channel based systems like Discord is increasingly popular partly because of various sorts of "cancel wars", well- or ill-intentioned capture and use of posts out of context.

digiown2• 5 months ago

Semi-ephemeral is the wrong way to view Discord et al. It's only that way if you actively remove your messages after a while, but they have made it as difficult as possible.

Though the fact that it's hard to archive too, perhaps make it more ephemeral on the whole, since few people will have a backup when do you get around to remove it.

Korici• 5 months ago

Agreed. This is mostly mitigated by using a Bot to auto-delete all messages in a channel based on time since the post. I've switched to this method to be more ephemeral and have less concern of retention in perpetuity by Discord's egregious privacy policy.

Here is the bot I am leveraging: https://eazyautodelete.xyz/en

CGMthrowaway• 5 months ago

TLDR The actual (formally) hard problems:

  Defining archive boundaries in a dense social graph (graph traversal + stopping criteria without exploding scope)
  Entity resolution across pseudonymous accounts 
  Reconstructing opaque ranking algorithms from outputs

garethsprice• 5 months ago

Would it make sense to archive every word every person ever speaks? At what point does archiving everything people do constrain their ability to live freely in the present?

Despite being in written form (decreasingly so), social media feels more like a private conversation in a public space - and like all such conversations, it deserves the right to decay, so that we do not all become prisoners of the dumbest thing we ever said.

The transformative work of curation - choosing which pieces to save, to turn into books, diary entries, or blog posts that record context for posterity - is a valid part of how archivists build the corpus of history. Harvesting all the raw data simply because we can is a dangerous road.

nancyminusone• 5 months ago

social media feels more like a private conversation in a public space

Then it wasn't very smart of you to post something publicly then.

You should be able to choose what things you want public or private, but if your intention is for public, I don't think you should be able to delete it. You can make amendments to what you say, but you shouldn't be granted the ability to choose what other people remember. Otherwise, private conversations or just saying nothing are alternatives.

At what point does archiving everything people do constrain their ability to live freely in the present?

The point at which information you intended to post privately is made public without your permission.

drnick1• 5 months ago

Why would you want to archive social media? It's always been slop and now it's increasingly AI slop.

We should really to back to people hosting their own websites when they want to share something publicly. Just plain HTML like in 1995.

gs17• 5 months ago

For better or for worse, there are a lot of "important" discussions that happen on Twitter, and a lot of "documentation" that has unfortunately been assembled in Discord channels. You can put your content on your website, but that's not going to make everyone else stop taking the easy way out.

Hard problems in social media archiving

Related Stories