Building LLMs from the Ground Up: A 3-Hour Coding Workshop

https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up

by mdp2021 • 2 years ago

970 135 2 years ago

magazine.sebastianraschka.com

abusaidm• 2 years ago

Nice write up Sebastian, looking forward to the book. There are lots of details on the LLM and how it’s composed, would be great if you can expand on how Llama and OpenAI could be cleaning and structuring their training data given it seems this is where the battle is heading in the long run.

rakahn• 2 years ago

Yes. Would love to read that.

rahimnathwani2• 2 years ago

  how Llama and OpenAI could be cleaning and structuring their training data

If you're interested in this, there are several sections in the Llama paper you will likely enjoy:

https://ai.meta.com/research/publications/the-llama-3-herd-o...

kbrkbr• 2 years ago

But isn't it the beauty of llm's that they need comparably little preparation (unstructured text as input) and pick the features on their own so to say?

edit: grammar

atum47• 2 years ago

Excuse my ignorance, is this different from Andrej Karpathy https://www.youtube.com/watch?v=kCc8FmEb1nY

Anyway I will watch it tonight before bed. Thank you for sharing.

BaculumMeumEst7• 2 years ago

Andrej's series is excellent, Sebastian's book + this video are excellent. There's a lot of overlap but they go into more detail on different topics or focus on different things. Andrej's entire series is absolutely worth watching, his upcoming Eureka Labs stuff is looking extremely good too. Sebastian's blog and book are definitely worth the time and money IMO.

brcmthrowaway• 2 years ago

what book

bilsbie• 2 years ago

What are the main differences?

eclectic29• 2 years ago

This is excellent. Thanks for sharing. It's always good to go back to the fundamentals. There's another resource that is also quite good: https://jaykmody.com/blog/gpt-from-scratch/

_giorgio_5• 2 years ago

Not true.

Your resource is really bad.

"We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text."

skinner_2• 2 years ago

Your resource is really bad.

What a bad take. That resource is awesome. Sure, it is about inference, not training, but why is that a bad thing?

adultSwim• 2 years ago

This page is just a container for a youtube video. I suggest updating this HN link to point to the video directly, which contains the same links as the page in its description.

yebyen• 2 years ago

Why not support the author's own website? It looks like a nice website

_giorgio_5• 2 years ago

He shares a ton of videos and code. His material is really valuable. Just support him?

mdp20215• 2 years ago

On the contrary, I saved you that extra step of looking for Sebastian Raschka's repository of writings.

bschmidt114• 2 years ago

Love stuff like this. Tangentially I'm working on useful language models without taking the LLM approach:

Next-token prediction: https://github.com/bennyschmidt/next-token-prediction

Good for auto-complete, spellcheck, etc.

AI chatbot: https://github.com/bennyschmidt/llimo

Good for domain-specific conversational chat with instant responses that doesn't hallucinate.

p1esk5• 2 years ago

Why do you call your language model “transformer”?

bschmidt114• 2 years ago

Language is the language model that extends Transformer. Transformer is a base model for any kind of token (words, pixels, etc.).

However, currently there is some language-specific stuff in Transformer that should be moved to Language :) I'm focusing first on language models, and getting into image generation next.

vunderba• 2 years ago

I took a very cursory look at the code, and it looks like this is just a standard Markov chain. Is it doing something different?

bschmidt114• 2 years ago

I get this question only on Hacker News, and am baffled as to why (and also the question "isn't this just n-grams, nothing more?").

https://github.com/bennyschmidt/next-token-prediction

^ If you look at this GitHub repo, should be obvious it's a token prediction library - the video of the browser demo shown there clearly shows it being used with an <input /> to autocomplete text based on your domain-specific data. Is THAT a Markov chain, nothing more? What a strange question, the answer is an obvious "No" - it's a front-end library for predicting text and pixels (AKA tokens).

https://github.com/bennyschmidt/llimo

This project, which uses the aforementioned library is a chat bot. There's an added NLP layer that uses parts-of-speech analysis to transform your inputs into a cursor that is completed (AKA "answered"). See the video where I am chatting with the bot about Paris? Is that nothing more than a standard Markov chain? Nothing else going on? Again the answer is an obvious "No" it's a chat bot - what about the NLP work, or the chat interface, etc. makes you ask if it's nothing more than a standard [insert vague philosophical idea]?

To me, your question is like when people were asking if jQuery "is just a monad"? I don't understand the significance of the question - jQuery is a library for web development. Maybe there are some similarities to this philosophical concept "monad"? See: https://stackoverflow.com/questions/10496932/is-jquery-a-mon...

It's like saying "I looked at your website and have concluded it is nothing more than an Array."

kgeist5• 2 years ago

Simpler take on embeddings (just bigrams stored in JSON format)

So Markov chains

bschmidt114• 2 years ago

See https://news.ycombinator.com/item?id=41419329

karmakaze• 2 years ago

This is great. Just yesterday I was wondering how exactly transformers/attention and LLMs work. I'd worked through how back-propagation works in a deep RNN a long while ago and thought it would be interesting to see the rest.

aDyslecticCrow2• 2 years ago

For some intuition, 3b1b has some videos that explain it nicely. But he doesn't go that far into the nitty gritty.

ein0p3• 2 years ago

I’m not sure why you’d want to build an LLM these days - you won’t be able to train it anyway. It’d make a lot of sense to teach people how to build stuff with LLMs, not LLMs themselves.

ckok• 2 years ago

This has been said about pretty much every subject. Writing your own Browsers, compilers, cryptography, etc. But at least for me even if nothing comes of it just knowing how it really works, What steps are involved are part of using things properly. Some people are perfectly happy using a black box, but without kowning how its made, how do we know the limits? How will the next generation of llms happen if nobody can get excited about the internal workings?

ein0p3• 2 years ago

You don’t need to write your own LLM to know how it works. And unlike, say, a browser it doesn’t really do anything even remotely impressive unless you have at least a few tens of thousands of dollars to spend on training. Source: my day job is to do precisely what I’m telling you not to bother doing, but I do have access to a large pool of GPUs. If I didn’t, I’d be doing what I suggest above.

kgeist5• 2 years ago

It's possible to train useful LLMs on affordable harwdare. It depends on what kind of LLM you want. Sure you won't build the next ChatGPT, but not every language task requires a universal general-purpose LLM with billions of parameters.

BaculumMeumEst7• 2 years ago

It's so fun! And for me at least, it sparks a lot of curiosity to learn the theory behind them, so I would imagine it is similar for others. And some of that theory will likely cross over to the next AI breakthrough. So I think this is a fun and interesting vehicle for a lot of useful knowledge. It's not like building compilers is still super relevant for most of us, but many people still learn to do it!

alok-g• 2 years ago

This is great! Hope it works on a Windows 11 machine too (I often find that when Windows isn't explicitly mentioned, the code isn't tested on it and usually fails to work due to random issues).

sidkshatriya• 2 years ago

When it does not work on Windows 11 -- what about trying it out on WSL (Windows Subsystem for Linux ) ?

politelemon• 2 years ago

This should work perfectly fine in WSL2 as it has access to a GPU. Do remember to install the Cuda toolkit, NVidia has one for WSL2 specifically.

https://developer.nvidia.com/cuda-downloads?target_os=Linux&...

paradite2• 2 years ago

I wrote a practical guide on how to train nanoGPT from scratch on Azure a while ago. It's pretty hands-on and easy to follow:

https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guid...

firesteelrain• 2 years ago

Did it really only cost $200?

What sort of things could you do with it? How do you train it on current events?

paradite2• 2 years ago

Yes. I checked the Azure usage after training.

Beyond learning how it all works and demo, there is not much practical usage. You can train it on current events if you feed that corpus during training instead of just OpenWebText. Shouldn't be hard.

1zael• 2 years ago

Sebastian, you are a god among mortals. Thank you.

alecco3• 2 years ago

Using PyTorch is not "LLMs from the ground up".

It's a fine PyTorch tutorial but let's not pretend it's something low level.

menzoic• 2 years ago

Is this a joke? Can’t tell. OpenAI uses PyTorch to build LLMs

jnhl• 2 years ago

You could always go deeper and from some points of view, it's not "from the ground up" enough unless you build your own autograd and tensors from plain numpy arrays.

TZubiri• 2 years ago

Source please?

leobg• 2 years ago

People think of the Karpathy tutorials which do indeed build LLMs from the ground up, starting with Python dictionaries.

atoav4• 2 years ago

No it is not. From scratch has a meaning. To me it means: in a way that letxs you undrrstand the important details, e.g. using a programming language without major dependencies.

Calling that from scratch is like saying "Just go to the store and tell them what you want" in a series called: "How to make sausage from scratch".

When I want to know how to do X from scratch I am not interested in "how to get X the fastest way possible", to be frank I am not even interested in "How to get X in the way others typically get it", what I am interested in is learning how to do all the stuff that is normally hidden away in dependencies or frameworks myself — or, you know, from scratch. And considering the comments here I am not alone in that reading.

SirSegWit• 2 years ago

I'm still waiting for an assembly language model tutorial, but apparently there are no real engineers out there anymore, only torch script kiddies /s

sigmoid10• 2 years ago

Pfft. Assembly. I'm waiting for the real low level tutorial based on quantum electrodynamics.

oaw-bct-ar-bamf• 2 years ago

Automotive actually uses ML in plain c with some inline assembly sprinkled on top run run models in embedded devices.

It’s definitely out there and in productive use.

wredue3• 2 years ago

Ironically, slippery slope argumentation is a favourite style of kids.

Unfortunately, your argument is a well known fallacy and carries no weight.

botverse• 2 years ago

#378

alecco3• 2 years ago

I'll write a guide "no-code LLMs in CUDA".

jb19913• 2 years ago

Learn to play Bach: start with making your own piano.

defrost• 2 years ago

Bach (Johann Sebastian .. there were many musical Bach's in the family) owned and wrote for harpsichords, lute-harpsichords, violin, viola, cellos, a viola da gamba, lute and spinet.

Never had a piano, not even a fortepiano .. though reportedly he played one once.

jahdgOI• 2 years ago

Pianos are not proprietary in that they all have the same interface. This is like a web development tutorial in ColdFusion.

atoav4• 2 years ago

Wanted to say the same thing. As an educator who once gave a course on a similar topic for non-programmers you need to start way, way earlier.

E.g.

1. Programming basics

2. How to manipulate text using programs (reading, writing, tokenization, counting words, randomization, case conversion, ...)

3. How to extract statistical properties from texts (ngrams, etc, ...)

4. How to generate crude text using markov chains

5. Improving on markov chains and thinking about/trying out different topologies

Etc.

Sure markov chains are not exactly LLMS, but they are a good starting point to byild a intuition how programs can extract statistical properties from text and generate new text based on that. Also it gives you a feeling how programes can work on text.

If you start directly with a framework there is some essential understanding missing.

BaculumMeumEst7• 2 years ago

I really like Sebastian's content but I do agree with you. I didn't get into deep learning until starting with Karpathy's series, which starts by creating an autograd engine from scratch. Before that I tried learning with fast.ai, which dives immediately into building networks with Pytorch, but I noped out of there quickly. It felt about as fun as learning Java in high school. I need to understand what I'm working with!

krmboya2• 2 years ago

Maybe it's just different learning styles. Some people, me included, like to start getting some immediate real world results to keep it relevant and form some kind of intuition, then start peeling back the layers to understand the underlying principles. With fastAI you are already doing this by the 3rd lecture.

Like driving a car, you don't need to understand what's under the hood you start driving, but eventually understanding it makes you a better driver.

jph002• 2 years ago

fast.ai also does autograd from scratch - and goes further than Karpathy since it even does matrix multiplication from scratch.

But it doesn’t start there. It uses top-down pedagogy, instead of bottom up.

delano• 2 years ago

If you want to make an apple pie from scratch, first you have to invent the universe.

CamperBob2• 2 years ago

After watching the Karpathy videos on the subject, of course.

_giorgio_5• 2 years ago

Your comment is one of the most pompous that I've ever read.

NVDIA value lies only in pytorch and cuda optimizations with respect with pure c implementation, so saying that you need go lower level than cuda or pytorch means simply reinventing Nvidia. Good luck with that

alecco3• 2 years ago

1. I only said the meaning of the title is wrong, and I praised the content

2. I didn't say CUDA wouldn't be ground up or low level (please re-read) (I say in another comment about a no-code guide with CUDA, but it's obviously a joke)

3. And finally, I think your comment comes out as holier than thou and finger pointing and making a huge deal out of a minor semantic observation.

nerdponx4• 2 years ago

Low level by what standards? Is writing an IRC client in Python using only the socket API also not "from scratch"?

badsectoracula4• 2 years ago

Considering i seem to be the minority here based on all the other responses the message you replied to, the answer i'd give is "by mine, i guess".

At least when i saw the "Building LLMs from the Ground Up" what i expected was someone to open vim, emacs or their favorite text editor and start writing some C code (or something around that level) to implement, well, everything from the "ground" (the operating system's user space which in most OSes is around the overall level of C) and "up".

bvrmn2• 2 years ago

In LLMs context pytorch.nn is low level. It's a crucial thing in education to not operate on too many abstraction levels.

cruffle_duffle• 2 years ago

Not just education but real life engineering too.

theanonymousone2• 2 years ago

It may be unreasonable, but I have a default negativity toward anything that uses the word "coding" instead of programming or development.

xanderlewis• 2 years ago

Probably now an unpopular view (as is any opinion perceived as 'judgemental' or 'gatekeeping'), but I agree.

smartmic• 2 years ago

I fully agree. We had a discussion about this one year ago: https://news.ycombinator.com/item?id=36924239

ljlolel• 2 years ago

This is more a European thing

atoav4• 2 years ago

I am from Europe and I am not completely sure about that to be honest. I also prefer programming.

I also dislike software development as it reminds me of developing a photograhic negative – like "oh let's check out how the software we developed came out".

It should be software engineering and it should be held to a similar standard as other engineering fields if it isn't done in a non-professional context.

SkiFire13• 2 years ago

As an European: my language doesn't even have a proper equivalent to "coding", only a direct translation to "programming"

mdp20215• 2 years ago

Quite a cry, in a submission page from one of the most language "obsessed" in this community.

Now: "code" is something you establish - as the content of the codex medium (see https://en.wikipedia.org/wiki/Codex for its history); from the field of law, a set of rules, exported in use to other domains since at least the mid XVI century in English.

"Program" is something you publish, with the implied content of a set of intentions ("first we play Bach then Mozart" - the use postdates "code"-as-"set of rules" by centuries).

"Develop" is something you unfold - good, but it does not imply "rules" or "[sequential] process" like the other two terms.

theanonymousone2• 2 years ago

I did my research, but as a non-native speaker, couldn't figure out whether "quite a cry" means you agree or disagree with me.

ChatGPT doesn't know either.

k1tanaka• 2 years ago

I am from Brazil and I find this funny because in my circle of friends/co-wroekers we mostly use "coding" when speaking English, or "codar" (code as a Portuguese verb) with other Brazilians. I am not sure why, but I think it is because "program" has a strong association with prostitution in Brazilian Portuguese.

cpill• 2 years ago

yeah really valuable stuff. so we know how the ginormous model that we can't train or host works (putting practice there are so many hacks and optimizations that none of them work like this). great.

leopoldj• 2 years ago

This is the exact level of details I was looking for. I'm fairly experienced with deep learning and pytorch and don't want to see them built from scratch. I found Andrej's materials too low level and I tend to get lost in the weeds. This is not a criticism but just a comment for someone in a similar situation as I'm.

Building LLMs from the Ground Up: A 3-Hour Coding Workshop

Related Stories