Anna’s Archive approaching 1 petabyte

https://annas-archive.org/blog/critical-window.html

by popcalc • 2 years ago

87 16 2 years ago

This project is now all the more interesting given what we now know about LLM scaling laws. It will take trillions of high quality, factual tokens to train the next generation of LLMs.

Copyright is also appearing to be beside the point. A petabyte of copyrighted data could be run through an LLM and rephrased/summarized in a way that retains the essential knowledge but creates a public domain dataset for further training.

shrimp_emoji• 2 years ago

It will take trillions of high quality, factual tokens to train the next generation of LLMs.

This is like saying it will take trillions of man-hours to chisel through a mounting with a spoon.

You might want to think outside of that box.

pilimi_anna• 2 years ago

Anna here. In some sense copyright is appearing to be beside the point, but for the exact same reason copyright holders are becoming more defensive, and trying to expand the scope of copyright. That's one of the reasons why this is such a critical window for preservation!

glimshe• 2 years ago

You probably hear this a lot, but thank you for fighting the good fight and working towards advancing humanity. Your work is incredibly important.

dtx1• 2 years ago

I am immensely curious about everything AA is doing but I guess all my questions better remain unanswered for the sake of the project. But perhaps, if you are interested, you might chose to answer some of them.

How are you holding up? Do you notice the persecutocutorial pressure on a day to day basis?

How is the Project being financed? How is the financing being protected from the legal pressures? Just having one petabyte of data somewhere is expensive. Are KYC an issue for you?

I would think that any large enough law enforcement would be able to take down not just your domains but also your servers. Why is it that enforcement seems to focus on domains not on server seizures?

What protective measures and backup plans do you have in place? Any Opsec tips to share for small time pirates?

dotancohen• 2 years ago

Thank you for all that you do.

In the fine article, "10,000" is mistakenly written as "1,0000".

boramalper• 2 years ago

May want to change its title to the original: The critical window of shadow libraries

edflsafoiewq• 2 years ago

Yes please. The content has vastly larger scope than the current title implies.

throwaway2562• 2 years ago

Agreed

carlosjobim• 2 years ago

Other interesting news from the link: They expect improvements in OCR to make it possible for them in the coming years to apply it to their entire library. This would liberate an enormous amount of knowledge for easier access.

moonchild• 2 years ago

they are talking about treating ocr as lossy. i wonder about making a lossless compression algorithm for text scans based on an ocr; in effect, use the ocr to predict which text will show up and how, and then encode the pixel-level differences on top of that

eulgro• 2 years ago

DjVu does this to some extent, identifying identifical glyph bitmaps and reusing them for compression. See https://en.m.wikipedia.org/wiki/DjVu#Compression

benxh• 2 years ago

I am assuming this will be solved this year.

RecycledEle• 2 years ago

A lot of their content is duplicated.

De-duping similar uploads of the same version of the same book would cut petabyte down quite a bit.

ffsm8• 2 years ago

A lot of seemingly duplicate content isn't actually. There are often multiple book publications/revisions that are hard to tell apart as the name/cover stays the same. You only find out by reading the small print at the start were the publisher info/ISBN is

Anna’s Archive approaching 1 petabyte

Related Stories