Announcing GPT-NeoX-20B

https://blog.eleuther.ai/announcing-20b/

by jscob • 4 years ago

200 70 4 years ago

blog.eleuther.ai

nefitty3• 4 years ago

Thank you to everyone who has worked on this. EleutherAI has become a touchstone in my mind on what is possible in open data and code. In creating alternatives to closed gardens they have shown me new possible paths. I know Linux has done the same for others.

Huggingface has also made playing with this stuff super accessible. They've made me super curious about rust and AI/ML research which has influenced my personal engineering goals for the future. I am on your team Roko's Basilisk.

rsync• 4 years ago

I came to this thread looking for comments that I would suspect were machine generated.

I was not disappointed.

coolspot• 4 years ago

Good bot

monkeydust• 4 years ago

Shout out to Huggingface. As a business user it has allowed me to explore use cases around text summarisation very easily and provided ideas for future work. I clearly need to check out EleutherAI as well.

fpgaminer2• 4 years ago

So excited for this release. In the wake of AI Dungeon's downfall, having GPT-Neo to fallback on has been a saving grace. While the 6B model is nowhere near as good as the original AI Dungeon, which used OpenAI's 175B model, it was at least serviceable unlike the "gentled" AI Dungeon. And you could run it locally or through Colab, which was really cool. I ended up using it through NovelAI, since they've spent a lot of time fine-tuning the model and adding a plethora of features that end up improving the overall output. (NovelAI's interface is like AI Dungeon on steroids!) But there is a vibrant community of Colab notebooks and other tools for DIYers surrounding the GPT-Neo model.

That said, besides being overall "dumber" than 175B GPT-3, the 6B model was missing a critical feature: prompting. 175B GPT-3 could be "prompted" to write things. For example, you could give it "Write a story about cyberpunk gnomes:" and it would go on to do just that, all on its own. GPT-Neo didn't really have that capability in my experience. The only way to get it to reliably write such a story is to begin writing it yourself, at which point GPT-Neo could help to continue the story.

So I'm excited to see not just how much "smarter" Eleuther's new 20B model is, but also if it has attained that coveted prompting ability. Given the non-linear relationship between parameters and loss, my hopes are high.

P.S. NovelAI recently added the Fairseq 13B model to their repertoire. I haven't had a chance to try it personally, but I've seen positive things about it. My bet is on GPT-NeoX-20B being better still.

f38zf5vdt2• 4 years ago

You can try the model out for free at goose.ai after making an account and going to the sandbox section.

qlm• 4 years ago

What was AI Dungeon's downfall? Can't find much about it.

minimaxir• 4 years ago

tl;dr AI Dungeon was required to add additional content filters after it went too off the rails, which caused community backlash.

https://www.wired.com/story/ai-fueled-dungeon-game-got-much-...

benjismith2• 4 years ago

The "prompting" ability you're referring to is called "instruction following", and here are some descriptions of it.

https://openai.com/blog/instruction-following/

I think the differences are more in the training data used, than in the nature of the model itself. So you could probably train your own instruction-following model on top of this raw 20B model.

f38zf5vdt2• 4 years ago

Right on, they're closing in on "Open"AI's best models. Can this still be run on a GPU, or does it require a lot more VRAM?

stellaathena5• 4 years ago

It can be run on an A40 or A6000, as well as the largest A100s. But other than that, no.

djoldman• 4 years ago

How much VRAM does it use during inference?

bm-rf• 4 years ago

You could use Microsoft's DeepSpeed to run the model for inference on multiple GPUS, see https://www.deepspeed.ai/tutorials/inference-tutorial/

dqpb• 4 years ago

EleutherAI is the real open ai.

guidovranken• 4 years ago

  GPT-NeoX-20B will be publicly downloadable from The Eye on the 9th of February.

The Eye as in the-eye.eu? That site has been down for a long time.

stellaathena5• 4 years ago

There is a mirror at https://mystic.the-eye.eu/ that has been up for a long time.

drusepth• 4 years ago

Thanks for this. When the-eye.eu went down it broke a ton of my Colab notebooks and it was impossible to find a mirror.

Jowsey• 4 years ago

From what I very briefly read, apparently their server hardware broke and they didn't have the funds to repair everything at once? Was quite shocked when I first saw it was down.

dash2• 4 years ago

Does anyone know whether the spammy websites that sit at the top of search engine results are already generated by this kind of model?

nefitty3• 4 years ago

That's a use case. I don't see why anyone would go out of their way to make intelligible content for spam. Google is so broken right now that SEO hacks are easy to generate. Not to overstress the tangent, but without search operstors, I have to sift through pointless Gitlab/Github/Stackoverflow/Wikipedia clones all the time.

ChefboyOG3• 4 years ago

By and large, no.

That's not to say that those sites are not generated programmatically--without a doubt, most of them are--but not by a cutting edge transformer model. The fact is, generating words has never been the bottleneck for blackhat SEO types. Generally, those sites are generating their content through some kind of scraping, or in rarer cases, paying pennies for nonsense articles. The page itself is structured for search (targeted H1s, metadata, etc.) and some kind of private blog network is used to create a pyramid of backlinks.

daniel-cussen• 4 years ago

The fact of generating words programmatically is hard to mask. You end up with all kinds of computational biases humans don't have.

btdmaster• 4 years ago

Awesome! Any chance for an online demo (like https://6b.eleuther.ai/)?

stellaathena5• 4 years ago

Coming soon!

schleck8• 4 years ago

Awesome, thanks for your work Stella & team!

iggldiggl• 4 years ago

One wish for both playgrounds: Would it be possible to have a specific error message for the "maximum context window size exceeded" error? At the moment you can only deduce it indirectly from the fact that you're immediately and repeatedly getting an "Unable to connect to the model. Please try again."-error as soon as you hit the prompt button. (And very occasionally it turns out to have been something else after all, because after one more try it surprisingly successfully generates one more round of completions)

Related question: Does GPT-NeoX-20B have the same context window size as GPT-J-6B?

terafo• 4 years ago

The best there is right now is a playground on https://goose.ai/

stavros• 4 years ago

Which unfortunately doesn't work properly on Firefox (spaces are removed).

gibsonf17• 4 years ago

How is pattern matching ever inference when there is no reference to the underlying computational model of what the words mean in spacetime?

How is it helpful to see what word might come next when the word sequence is just based on statistics with no reference at all to meaning?

ChefboyOG3• 4 years ago

Have you ever used any sort of autocomplete?

gibsonf17• 4 years ago

Yes, and very much like it when quickly selecting from a scope of valid selections.

This is not that. It is all A with no I.

drxzcl• 4 years ago

Humans assign a lot of, well, meaning to meaning. It turns out that you can get a really good score on tasks that superficially you would think require actual understanding without programming any of that in.

Does this mean the neural network has learned about meaning? Does that mean that it has just gotten really good at faking it? Does is mean that meaning itself doesn't really exist, and it's just a shorthand for advanced pattern matching? Does it matter?

Honestly, we don't know. But we've been thinking about it for a very long time. See for example the famous Chinese Room thought experiment:

https://en.wikipedia.org/wiki/Chinese_room

gibsonf17• 4 years ago

Try driving a car around without both conceptual and causal systems understanding of the world - meaning matters for survival.

neatze• 4 years ago

Does it matter?

As long at you don't make reckless assumptions then it not for some application, unklike (not going to name here) build a cult a like that GPT like models in near future will perform most if not all tasks better then humans.

Where it really matters is for mission critical application for example; in Windows or Linux terminal would you allow GPT to run terminals commands based of events in automated way ?

benjismith2• 4 years ago

I'm super excited about this!

I'm on the cusp of releasing a model into production that was fine-tuned upon your 6B model, and the results are quite excellent. I'd be very curious to try out the 20B model the next time we retrain.

Are there any other differences in this release (number of layers, number of attention heads, etc) compared with the 6B model, or does it simply scale-up the number of parameters?

trasz• 4 years ago

So, what does it do?

29athrowaway3• 4 years ago

Please test it using Idiocracy's IQ test question:

    If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have?

cahaya• 4 years ago

It replies with:

1st try: As the word question already gives you the answer to the question, "are you able to?" the rest of the question

2nd try: A. three B. four C. eleven D. five fewer.

29athrowaway3• 4 years ago

I think GPT-3 has better answers.

Announcing GPT-NeoX-20B

Related Stories