The Pile: An 800GB Dataset of Diverse Text for Language Modeling

http://pile.eleuther.ai/

by leogao • 6 years ago

223 59 6 years ago

pile.eleuther.ai

leogao4• 6 years ago

Twitter thread: https://twitter.com/nabla_theta/status/1345130412532645888

bratao• 6 years ago

I´m super excited by this dataset. The EleuterAI team is stellar and many great things are coming soon from them!

legatus• 6 years ago

I think it's worth noting that EleutherAI is a grassroots collection of researchers, which distinguishes it from academia/industry labs.

As part of their work on democratizing AI, they're now hoping to replicate GPT-3 and release it for free (unlike OpenAI's API).

I would encourage everyone interested to join their discord server (https://discord.gg/BK2v3EJ) -- they're extremely friendly and I think it's a project worth contributing to.

joe_the_user• 6 years ago

Any idea how large dataset used to train GPT-3 was?

arugulum• 6 years ago

570GB of Common Crawl post-filtering, but only 40% of CC data was seen even once during training, though CC is only 60% of the training data. You could work through the math to find the rough size of GPT-3's training data, but it sounds like The Pile is of comparable size.

odnes• 6 years ago

How are they sourcing/funding the compute to train these massive models?

luto2• 6 years ago

they'll probably run it on scientific clusters of various universities, or on collections of idle lab desktop machines. Both of these tend to sit idle a lot of the time, based on my experience at uni in Europe.

Alvion_Bleeds• 6 years ago

Connor Leahy, who I think is a sort of BDFL figure for ElutherAI, mentioned in a Slatestarcodex online meetup I attended that Google donated millions of dollars worth of preemptable TPU credits to the project. There is a video of the meetup on YouTube somewhere. Struck me as a really smart kid with a lot of passion.

stellaathena6• 6 years ago

TFRC, a program that lets you borrow Google's TPUs when they're not being used. You can apply here: https://www.tensorflow.org/tfrc

pizza• 6 years ago

Related: is it possible for me to download/try out a pretrained GPT-Neo?

stellaathena6• 6 years ago

Not currently. The code is on GitHub, but we do not have a public-facing model. We felt that the world doesn't need another GPT-2 replica, but if there's interest we can look into doing so though. We are planning on making our GPT-3 replica public facing though.

https://github.com/EleutherAI/gpt-neo

thereticent• 6 years ago

Love the name. It's what we called our shared FTP-served collection of mp3s during college 15-19 years ago. Yours is a MUCH more impressive amount of information!

eutectic2• 6 years ago

I'm kind of surprised there's 800GB of text in the world.

luto2• 6 years ago

heh, yeah! for comparison: the English Wikipedia is around 40 GB of text.

https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...

pbourke• 6 years ago

It’s not that much, actually. The average English word is 4.7 letters long. Let’s round up to 5 and add 1 for a space to make 6 characters. Novels are around 90,000 words long. So 800G of pure text represents 800G / 6 char per word / 90,000 words per book =~ 1.5M books. The Library of Congress has 39M books, not to mention all the text produced that’s exclusively online.

d33• 6 years ago

That makes me wonder what's the upper bound of storage required to contain all of the text humans had ever written. 1k times as much? 10k? Either way, I have a feeling that all of the drives would probably fit in a regular apartment room.

Crazy how we went from first computers spanning across entire floors to fitting all of human thoughts in such tight space.

stellaathena6• 6 years ago

There's more than 800GB of fanfiction in the world.

eutectic2• 6 years ago

Well, that's truly scary!

leogao4• 6 years ago

Oh, there's a lot more... no promises atm, but we plan on going much bigger for our next dataset.

jan_Inkepa• 6 years ago

Cool project! Minor note: It would be nice if ye said what language the dataset was for on the website :)

mrconter12• 6 years ago

Would you mind sharing some hints? What new approaches are you taking compared to this one?

lallysingh• 6 years ago

I wonder when this will be a benchmark number for a desktop GPU/NPU...

jcims• 6 years ago

I feel like YouTube is going to be a major source of language data in the future. The last statistic I saw was 500 hours of video uploaded every minute. If only 10% of those videos have original speaking in them and those average 40 words a minute, that’s almost 300 GB of transcribed speech per year.

wongarsu• 6 years ago

YouTube would be a great source for spoken language. But only a tiny portion of YouTube has subtitles, and it doesn't yet feel like automatic transcription is at a level where you would want to use its output to train something else. That day will surely come though

lovelearning• 6 years ago

It's not that big even as a text dataset. Common Crawl weighs in at some 250+TB. Even if we assume just 1% of that web data is usable text (it's likely much more), it's still 2.5+TB.

https://en.wikipedia.org/wiki/Common_Crawl

sillysaurusx2• 6 years ago

I'm sad they dropped Literotica from the dataset. It's available in the old "preliminary components" archive, if anyone wants to train a Literotica-Enron-Email GPT:

https://the-eye.eu/public/AI/pile_preliminary_components/

I think The Pile is probably one of the most important AI projects of the last year or so, at least for lone wolf researchers like me. Gathering training data at scale can be excruciatingly difficult. (Perhaps someone will do something similar for GAN training one day: a large dataset for image modeling would help a lot.)

By the way, consider contributing to The Eye: https://the-eye.eu/

Without them, I’m not sure any of us would have been able to host the datasets we gathered — or organized torrent seeds, or fielded DMCA complaints, etc. So I feel The Eye deserves an explicit shoutout for being such an asset to the AI community, along with TFRC and Eleuther.

6gvONxR4sf7o5• 6 years ago

I feel the opposite way about the eye. The data hosted by the eye isn’t theirs to give away. Their approach feels disingenuous.

sillysaurusx2• 6 years ago

I struggled with similar feelings. But, what convinced me, is that they take great care to remain compliant with DMCA. Pirates don’t.

The difference with The Eye is that they thoroughly vet the legitimacy of each DMCA claim. The claimant is required to show proof that they are the legal copyright holder. And The Eye seems willing to call bluffs: on more than one occasion they have dealt with bogus DMCAs in ways where normal companies would simply give up.

Ultimately, their actions are legal. And for us in the AI community, it was something like the hand of God reaching down to bless us with a guardian angel. The reproducibility situation is getting worse each week, and much of that is due to the fact that realistic datasets can’t be distributed without fear of reprisals.

That said, I respect your feelings on the matter too. I think it’s equally valid to feel uncomfortable. I just take solace in the fact that it’s legal.

w1nk• 6 years ago

Will this dataset be able to produce a model that's actually better than GPT-3? One of the things mentioned in the paper is that the dataset had some buggy filtering applied to it (https://arxiv.org/pdf/2005.14165.pdf, page 9), that minimally impacted benchmarking some, there's a whole section on how they tried to deal with it. The gist though is that a cleaner training run, even on slightly more data, may (should?) actually produce something a bit better.

Does anyone know if OpenAI has retrained/updated gpt-3 yet?

forgingahead• 6 years ago

This is a great effort, and it's important to have datasets like this available to democratise ML learning and work.

One small comment: It would be great for this (and other) datasets to give a quick "sample data" file - preferably one that doesn't need to be downloaded to be viewed. Even a screenshot of some of the data would be useful for people browsing to get a quick understanding of the actual content, and how it is formatted. Downloading gigabytes of data just to have a look isn't practical.

remexre2• 6 years ago

The test and validation datasets are under half a gig; I agree it'd be nice if there were a few example items and perhaps a JSON Schema [0] for it, though.

I'll reproduce a line here:

    {"text": "Roman Catholic Diocese of Tambacounda\n\nThe Roman Catholic Diocese of Tambacounda () is a diocese located in the city of Tambacounda in the Ecclesiastical province of Dakar in Senegal.\n\nHistory\n August 13, 1970: Established as Apostolic Prefecture of Tambacounda from the Diocese of Kaolack and Diocese of Saint-Louis du S\u00e9n\u00e9gal\n April 17, 1989: Promoted as Diocese of Tambacounda\n\nSpecial churches\n The cathedral is Cath\u00e9drale Marie Reine de l\u2019Univers in Tambacounda, which is located in the Medina Coura neighborhood of the town.\n\nLeadership\n Bishops of Tambacounda (Roman rite)\n Bishop Jean-No\u00ebl Diouf (since 1989.04.17)\n Prefects Apostolic of Tambacounda (Roman rite) \n Fr. Cl\u00e9ment Cailleau, C.S.Sp. (1970.08.13 \u2013 1986.04.24)\n\nSee also\nRoman Catholicism in Senegal\n\nReferences\n\nExternal links\n GCatholic.org\n Catholic Hierarchy \n\nCategory:Roman Catholic dioceses in Senegal\nCategory:Tambacounda\nCategory:Christian organizations established in 1970\nCategory:Roman Catholic dioceses and prelatures established in the 20th century", "meta": {"pile_set_name": "Wikipedia (en)"}}

[0]: https://json-schema.org/

gwern• 6 years ago

I believe the paper includes random samples at the end.

mrconter12• 6 years ago

It includes two example snippets from each dataset.

floatingatoll• 6 years ago

To clarify, they mean diverse as in “unconnected datasets”.

durnygbur• 6 years ago

Someone explain me how this will not stir the wrath of copyright echelons, especially American ones? Or once I get PhD I can torrent movies again?

unixhero• 6 years ago

How can I start working on this dataset? Are there any Python GPT libraries?

mrfusion• 6 years ago

Why not throw Wikipedia into this collection?

ValueNull• 6 years ago

Hopefully not the Scots version

stellaathena6• 6 years ago

... is it in the collection?

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Related Stories