Are 99% of LLMs based on web scraping?

https://news.ycombinator.com/item?id=37284136

by squatrito23 • 3 years ago

11 8 3 years ago

news.ycombinator.com

I just get this impression when reading news on recent lawsuits involving Google bot "fair use" of IP protected scraped data, OpenAI, and Midjourney

throwaway888abc• 3 years ago

I will counter with question:

How and where do you get most of human knowledge nowadays ?

cyanydeez• 3 years ago

Usenet?

birdyrooster• 3 years ago

Ayahuasca ceremonies, duh

FetusP• 3 years ago

With Burning Man about to happen, LLMs are going to hit new levels

tamimio• 3 years ago

Books, a lot of books aren’t available online, especially non English ones.

jgalt212• 3 years ago

Somewhat related. Has anyone else notices the spiders out of control lately? We have seen 5-10X the spider traffic Aug 23 vs Aug 22.

jmpman• 3 years ago

CommonCrawl…

simne• 3 years ago

Sure, no.

As I estimate, there are few terabytes of books in library of Congress, and this considered largest books collection in whole world.

Fortunately, many books in LoC already public domain (copyright expired), so could use without limitations. For others, things are complicated.

In overall, scientists allowed to use any book at their work, but this mean automatically, that derived content could not be used commercially, only for research or for arts.

On other side, people tried to use synthetic datasets, generated by algorithms, or even got real texts and just dilute with noise words. But unfortunately this leads to hallucinations.

So, if you need more data then in PD books to train model, you have to scrape web, or to make some tricks, like use PD videos (that's why GPT-4 become multimodal), or hire lot of people and pay them to talk with model (this is just very expensive).

I hear, some big corporations feed models with their own internal talks, for example used records of support talking with customers, but this is too synthetic approach and very limited.

Are 99% of LLMs based on web scraping?

Related Stories