Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

https://github.com/karpathy/llm.c/discussions/677

by alecco • 2 years ago

182 58 2 years ago

github.com

alecco2• 2 years ago

Also https://x.com/karpathy/status/1811467135279104217#m

iforiq• 2 years ago

How much did gpt2 training cost when it came out in 2019?

ozr• 2 years ago

About $50,000.

withinboredom• 2 years ago

They probably spent more on the training data, to be honest. They had to get it the hard way.

jamestimmins2• 2 years ago

Anyone have an idea if this is feasible to do on a Macbook with a built-in GPU?

arthurcolle• 2 years ago

Will take more hours

michaelmior• 2 years ago

Probably not with the same amount of training time, but I'd imagine a recent MBP GPU could handle GPT-2 training. The biggest challenge is that the training would need to be reimplemented for Metal instead of CUDA.

jamestimmins2• 2 years ago

Ah so I couldn't just run this on my laptop for ~48 hours? That's too bad.

rty32• 2 years ago

Slightly off topic -- I just saw people saying how Mac's unified memory makes it a strength to train models on Macs: https://www.macrumors.com/2024/07/10/apple-leads-global-pc-g..., and how energy efficient they are etc. But what I am seeing is that people don't often even touch Macs at all -- they write code with CUDA and that's it. I find this kind of conversation fascinating.

tomalaci• 2 years ago

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years.

What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications.

For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI).

talldayo• 2 years ago

I expect this will cost maybe few dozen dollars and train in few hours within next few years.

I wouldn't count on it. Nvidia's been cleaning up shop, but their best option for expanding right now is through parallelization (bigger clusters, basically). Now that Blackwell is on TSMC, Nvidia is alongside Apple waiting for new and denser nodes to upgrade to. A real "generational leap" in training cost is going to require some form of efficiency gain that we're not seeing right now. It's possible that Nvidia has something up their sleeve, but I'm not holding my breath.

What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models.

What's funny is, you basically already can. The problem is becoming integration, and in the case of video games, giving the AI a meaningful role to fill. With today's finest technology, you can enjoy an AI-generated roguelike that is nigh-incomprehensible: https://store.steampowered.com/app/1889620/AI_Roguelite/

As time goes on, I really think developers are just going to not use AI for video games. Maybe I'm missing the "minecraft moment" for procedurally-generated stories here, but the sort of constraints needed to tell a story of create an interactive experience don't exist within LLMs. It's a stochastic nightmare of potential softlocks, contradictions or outright offensive requests. The majority of places I've seen AI applied today isn't for content creation, but instead automated moderation.

up2isomorphism• 2 years ago

For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI).

To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.

MisterBastahrd2• 2 years ago

You don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.

bick_nyers• 2 years ago

What if the LLM that a specific NPC utilizes was handcrafted/fine-tuned by a human?

chongli2• 2 years ago

Yeah. I have played a bunch of the roguelike Caves of Qud [1] and it has both hand-written text and procedurally generated text. The former is quite interesting and relevant to both gameplay and plot. The latter is mostly uninteresting and irrelevant, though it does work as "filler." This is similar to how procedurally-generated grass can give a more natural look to a hill than you'd get with tiles (which are incredibly easy to spot unless a ton of work is put into hiding the seams and repeating patterns).

I still long for the day when we can have procedurally-generated stories and quests that are actually interesting to play through. I have no idea how that is going to work though!

[1] https://www.cavesofqud.com

forrestthewoods5• 2 years ago

For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI).

Current AI isn't even close to good enough for video game NPCs and related. We're several breakthroughs away from that being possible at any cost. Those breakthroughs might happen in 3 years, or they might not happen in 10. Hard to predict.

Tiberium4• 2 years ago

Are you sure? Models like Claude 3.5 Sonnet are both good at writing and instructions, as long as you set some guardrails for the model, they can be great NPCs.

swatcoder4• 2 years ago

For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI).

This'll be a niche for a long, long time.

Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one for many game designers -- especially given the current/near state of the technology. There will be games and experiments that explore it, for sure, but the vast majority of games just don't have any need for it.

ImHereToVote• 2 years ago

I don't have needs. I have wants. You see, I don't need. I want.

headcanon• 2 years ago

I do think there is a big opportunity for widely supported hardware-accelerated matrix algebra in games. Currently most of that is geared towards graphics (naturally) but being able to easily encode arbitrary models and have them run on-device would open up a lot of opportunities for games (like deep simulation) that weren't possible before. Its currently possible of course but requires custom tooling and (relatively) niche hardware like a high-end graphics card.

I see the development energy around LLMs as a way to open up support for that.

jetrink• 2 years ago

Those don't have to be mutually exclusive though. To take AI out of it, think of those murder mystery parties where actors interact with attendees. Actors have roles to play and things they must do to move the story forward, but they improvise their dialog when talking with the players and sometimes each other. Or if you've ever played D&D, you have experienced talking with NPCs that are controlled by your DM. I think video game AI could be a lot like that, where NPCs use natural language instead of rigid dialog trees, but otherwise, they behave a lot like they do today.

123yawaworht456• 2 years ago

for narrative/dialogue, yes, text generation is currently useless. censorship, slop, extreme positivity bias. even jailbroken Opus is shit.

but audio generation we already have is pretty much good enough, and this is big. it's not AAA tier yet, sure, but still lightyears better half-assing it with mediocre voice actors. it is now an option to only use real voice actors for a few key characters, and even that won't be necessary within the next decade. even indie video games will be fully-voiced soon.

opyate• 2 years ago

Perhaps Llama are good enough now to add flavour NPCs to the world, i.e. NPCs not on your critical path, but maybe on the town square where you can hear them talk about mundane life stuff.

robbomacrae• 2 years ago

Whilst I agree with the reservations of the other replies I think you were implying in the future and I'm sure the LLM's will be more trustworthy and up to the task at some point.

What I would really like to see now is all the new TTS models being used more widespread. There are still so many games that have text only output. My kid love Alba: A wildlife Adventure but the eldest still isn't quite ready to read all the text so I have to sit with them reading out all the lines.

If anyone has a way of applying universal mods / accessibility features to existing games I'd love to see someone solve this and happy to help with the TTS!

charlescurt123• 2 years ago

I imagine we could do this now but not the way you think.

have a human created story and text as a guideline.

With that have genAI make the text per stage, you would get different statements every time and would stay on track.

Would be interesting to play a game where all players say the same information in slightly different ways every single playthrough.

HPsquared• 2 years ago

Similar scaling to genome sequencing. First genome was a huge undertaking, now routine after a few Moore-esque cycles.

alecco2• 2 years ago

It will be interesting to see this with today's FlashAttention 3 for H100.

rurban• 2 years ago

Would be free for us because we have those H100`s, but currently it's way too hot now. They will reach 70°C, even watercooled.

Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

Related Stories