Outperforming larger language models with less training data and smaller models

https://blog.research.google/2023/09/distilling-step-by-step-outperforming.html

by atg_abhishek • 3 years ago

320 123 3 years ago

blog.research.google

xnx• 3 years ago

The amount of activity and progress in the LLM/ML/AI spaces is truly fantastic. Optimizations like this are particularly valuable when hardware (e.g. Nvidia) is so expensive.

ziofill• 3 years ago

Is it that a lot of capacity is unused in those behemoth LLMs, or that the smaller language model just mimics the reasoning task? (Mimics the mimicking?)

famouswaffles• 3 years ago

There is no actual distinction between the "real" thing and "mimicking".

The datasets behemoth LLMs are trained on include a lot of noise that derail progress. They also just contain a lot of irrelevant knowledge that the LLM has to learn or memorize so an obscene amount of parameters is required.

When you're not trying to teach a language model the sum total of human knowledge and you provide a high quality curated dataset, the scale barrier is much lower.

https://arxiv.org/abs/2305.07759

behnamoh7• 3 years ago

Why don't people just train LLMs on pure Wikipedia, arXiv, and other scientific websites? That would reduce the noise and improve hallucinations, no?

fulafel• 3 years ago

That question seems synonymous with "are those [current] behemoth LLMs close to optimal?" and it seems obvious that they are not.

I wonder what ideas there are about how one could estimate the optimal size.

sp3322• 3 years ago

The large models generalize better. The small models are easier to train for specific tasks.

sinuhe69• 3 years ago

Why the amount of the training data for LLM is less than for the distilled and task-specific models (in the first figure)?

Or did the authors count the amount of training data for the LLMs to the required training data for the destined/task-specific models?

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj...

swsieber• 3 years ago

Or did the authors count the amount of training data for the LLMs to the required training data for the destined/task-specific models?

Yes.

They are counting the amount of data you need to collect to solve your problem. I can grab a pretrained LLM, and the data I have to collect in that instance is what I need to fine tune it.

threeseed• 3 years ago

For instance, serving a single 175 billion LLM requires at least 350GB of GPU memory using specialized infrastructure

Apple ships the Mac Studio which support up to 144GB of usable GPU memory.

Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space.

rubyn00bie2• 3 years ago

Apple's hardware advantage is something I'm hoping to see really unleashed with the M3 generation. The fact the A17 Pro has ray tracing support is giving me hope they can catch up with the incumbents quickly. It's honestly the one thing that has kept me away from their newer hardware-- I primarily use my computer at a desk and PC hardware (GPU primarily) is leagues ahead of what their very best can do. I honestly can't justify spending nearly 4k when Linux works great for everything I do professionally AND when I'm done with work I can still play a game.

behnamoh7• 3 years ago

3 most important things for Apple are:

- HW design

- UI

- UX

That Macs come with awesome VRAM is only because #3. Apple has no explicit incentive to make LLM-ready machines.

stevenhuang• 3 years ago

That number is not quantized either. Quantize 175B param at 4 bits then it'd fit in ~120GB VRAM (one can fit 34B models at 4 bit quantization in a single RTX3090 24GB VRAM)

brucethemoose24• 3 years ago

Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space

Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet.

Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment.

Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.

astrange4• 3 years ago

It's coming to llama.cpp soon.

https://github.com/ggerganov/llama.cpp/pull/3228

oezi2• 3 years ago

I am wondering who will be the first to raise RAM capacity of their hardware offerings dramatically to win the LLM crowd. Seems like the way to win market share.

marcyb5st• 3 years ago

Agreed. I would love to have something with the FLOPs of a 3070/4060 and 80 gigs of even slower VRAM (not necessarily HBM/GDDR6x) that can run the XXB models.

greatpostman• 3 years ago

Still waiting for google to release a model that matches gpt4. Until then, I’m assuming their presumed ai supremacy is marketing

behnamoh7• 3 years ago

There's AI-research supremacy and there's AI-product supremacy. Google has many brains (pun intended) on the former, but lacks incentive/structure for the latter.

6D794163636F756• 3 years ago

They're often also slow to market. They weren't the first search engine, they weren't the first email platform, they weren't the first video site, but now they're the biggest in those categories. I think Bard's growth will be very interesting to watch given that track record.

Or it'll get dropped within a year.

bobmaxup• 3 years ago

Bard is getting decent in search results. They adopted some patterns Phind took in early interfaces with citations and made it much better in terms of interaction and response time compared to what Phind is doing today.

Also, with more recent changes I can get decent summaries much easier (and at no cost to me) for every document in a google drive.

In my day to day, those features can be pretty powerful tool, even if its not at GPT4 level 6 months after GPT4 and Bard were released in March.

To my knowledge, no one is really integrating search at the level that Google is with generative language models.

Palmik• 3 years ago

Bing has citations, Perplexity has citations, etc.

I would not be surprised if ChatGPT moves some of the plugins to the free tier as well, to stay competitive over the longer term.

druskacik• 3 years ago

The article is just a fine-tuning concept that could have been published by anyone, using any large LLM and small LLM in combination. This has nothing to do with marketing.

carlossouza• 3 years ago

Their researchers are great, irrespective of how successful their gazillion products might be.

Without their papers, there wouldn't be GPTs.

kromem• 3 years ago

The title refers to them outperforming their previous SotA model with one a tenth the size by building off one of last year's most important research papers in generating additional synthetic data which effectively exports domain knowledge from the larger model to the smaller.

All this AI angst going on right now is ridiculous.

Yeah, every technology gets a hype cycle these days.

But not every technology has research cycles this fast.

In fact, I can't think of anything that's had research cycles this fast.

pedrovhb• 3 years ago

Interesting that they use T5 for the distilled model. I was under the impression that encoder-decoder architectures were on the way of the Dodo, but it seems they may still be relevant after all.

Also interesting is that this isn't an inconceivably clever and out of the box idea. It shows there's still a lot of low hanging fruit to explore, and the future of LLMs isn't set in stone yet. Could be that the real deal is a mixture of experts trained in this style. It's exciting that it feels the holy grail is close to being achievable if only the right combination of ideas is tried.

behnamoh7• 3 years ago

I was under the impression that encoder-decoder architectures were on the way of the Dodo,...

Why is that?

janalsncm• 3 years ago

Decoder-only is pretty good on some tasks. Translation for example was a classic encoded-decoder task but it seems decoders can handle it.

joemoon• 3 years ago

It's definitely a very similar method but fundamentally different in that the 'Distilling step-by-step' approach is a multi-task model.

As I understand it, rather that training the smaller model to produce the CoT/rationale as a part of the (decoded) response, it actually has two output layers. One output is for the label and the other is for the rationale. The other layers are shared, which is how/why the model is able to have an improved "understanding" of which nuances matter in the labeling task.

euclaise• 3 years ago

They actually have a performance edge, but they aren't well suited to chat models because you can't do caching of past states like with decoder-only models

rolisz• 3 years ago

T5 family is awesome. FastChat-T5 has amazing text generation quality (eg for RAG chatbots) and it can easily run on CPU fast enough to do a live conversation.

isaacfung3• 3 years ago

The paper mentioned was submitted in May. I think encoder-decoder architectures still make a lot of sense for multimodal models.

There are still many low hanging fruits. I have probably seen dozens of variations of chain-of-thoughts, tree-of-thoughts, graph-of-thoughts, self-ask, self-critique, self-plan, self-reflect, etc.

shanusmagnus2• 3 years ago

Is there a paper, or some other source, that synthesizes this grab-bag of techniques? As an interested outsider, it's hard to keep up with all the action. Would appreciate any pointers that try to bring some of the pieces together.

p-e-w2• 3 years ago

given the input question “Sammy wanted to go to where the people are. Where might he go? Answer Choices: (a) populated areas, (b) race track, (c) desert, (d) apartment, (e) roadblock”, distilling step-by-step provides the correct answer to the question, “(a) populated areas”

Huh? My answer as a human would have been "race track", as that is probably "where the people are" (during a race).

Did I fail? Am I a poor language model? Or is the whole thing just tea leaf reading to begin with?

smeagull• 3 years ago

A race track or an apartment are likely to be populated areas as well. "A" is a broader answer that encompasses more possibilities.

Zuiii• 3 years ago

A depending on the size of a "populated area", the density can be far lower than a crowded race track.

p-e-w2• 3 years ago

"Where the people are" doesn't simply mean "where a large number of people live". It also has connotations of "where it's happening", as in "where people are coming together (right now)".

But my point is really that speaking of the correct answer with a question as vague and open to interpretation as this one is absurd.

Reubend• 3 years ago

Did I fail? Am I a poor language model? Or is the whole thing just tea leaf reading to begin with?

No offense to you, but I never would have picked "race track". That answer doesn't make much sense since the original prompt doesn't mention anything about racing, and the definition of a populated area fits well with the question being asked.

stevepike• 3 years ago

Fascinating. I would 100% pick race track. Maybe because populated areas is plural? Or because the question gives a named subject and nobody would ever respond to "where are you going?" with "populated areas"?

diziet• 3 years ago

A google image search for "race track" has 0 people visible in the 17 images above the fold.

zarzavat• 3 years ago

But if I watch F1 on TV there are a lot of people there. A race track is like a stadium, it’s a place constructed for a spectator sport. It may be empty most of the time but when there is a race on then there will be crowds there.

“Populated areas” does not mean much. Montana is a populated area. It’s not a highly populated area but it’s populated alright.

ilaksh• 3 years ago

I am not a researcher, but it always seemed intuitive to me that the most effective models would be multimodal and trained with a core carefully tailored curriculum.

I would want to ensure that the system gains and retains the fundamental structures and skills that you know it needs to effectively and accurately generalize. While maintaining those things you then feed it lots of diverse data to learn the exceptions and ways the skills can be combined. But somehow you need to ensure those core skills and knowledge throughout. Maybe you could do that just by including outputting those understandings or manipulations in addition to the final answer. Similar to what the paper does.

For example, a code generation model might be required to output a state machine simulation of the requested program.

eru6• 3 years ago

I agree that multimodal seems like the way to go, but it's not at all intuitive why we should expect the curriculum needing to be carefully tailored. Compare https://gwern.net/scaling-hypothesis

mcmoor• 3 years ago

Yeah so far looking at the progression since 70s, domain-specific attempts stagnant while just-increase-the-input-and-core attempts get us where we are now

isaacfung3• 3 years ago

LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206

AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701

Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models https://arxiv.org/abs/2307.14430

Aerbil313• 3 years ago

Pondering on the idea of school curriculum. I wonder whether the order the training data is fed will make a difference. Whether it’s fed starting from simple things to more complex or vice versa. The gradient descent surely can end up in a different, better or worse, local minimum?

avereveard• 3 years ago

So this https://arxiv.org/abs/2212.08410 but one year later

sp3322• 3 years ago

Impressive gains, but 22% on GSM8K is maybe not an attention-grabbing end result.

fbnbr• 3 years ago

I think smaller expert models will dominate the majority of applications. there is an optimum and fine balance to strike when it comes to size and usability. There will be many mechanisms like demonstrated in the post to find that optimum and realize it.

dtx1• 3 years ago

And for a large general model just have multiple small expert models and a relay model that decides which domain specific model to ask

sandpaper263• 3 years ago

It's not a likely solution given how loss functions work, but in theory a single model could learn to perform exactly the function you describe. When you say "just do X" where X is any function (in this case, a piecewise function), a large enough model could do it.

basch2• 3 years ago

It would get more complex with cross domain questions. Part of what makes LLMs feel magical is their ability to synthesize such disparate topics and information into coherent “thoughts.”

I would suspect what you have instead is a single model attached to data sources, where the model doesn’t have to have so much compressed fact, and instead can rely on higher level summary.

treprinum2• 3 years ago

This approach would remove one of the main benefits - the ability to run multi-task one-shot prompts where a single LLM call returns answers to multiple NLP tasks.

avyfain• 3 years ago

I wrote about this a few months ago, right before OpenAI released ther Plugin feature: https://faingezicht.com/articles/2023/03/02/federated-langua...

TLDR, the essay explores how LLM could evolve into front-end routers that connect users with specialized tools, leading to a future where federated models determine the best-suited system to answer specific queries. Not too different from today's federated search approaches.

bradknowles• 3 years ago

I did a senior reading/research class in AI back in 1989, as my final class I needed to graduate with a BSCS. The idea originally was to do a survey of the methods known at the time and determine which was the best. This included things like e-mycin, the first generalized expert system toolkit.

I ended up with the premise that each of them had their relative strengths and weaknesses, and it would actually be best to use all of them, but only in their own areas of strength. Then have them use something akin to a shared blackboard where they could all read the results from the other systems and write their results as well. That the sum of all the available algorithms working together, each in the areas in which they were best, would result in better outcomes than any one algorithm could achieve on its own.

My professor was not impressed. I only got a C.

Now, the story of how my dad had to work his ass off to finally get the College of Engineering to force the professor to actually give me a grade that he owed me, when all the professor really wanted to do was focus on his new job at one of the big airlines -- well, that's a story for another time.

sourabh03agr• 3 years ago

Interesting! Do you think RLHF would be a necessity for smaller models to perform as par as state-of-the-art LLMs? In my view, instruction tuning will resolve any isssues related to output structure, tonality or the domain understanding but will it be enough to improve the reasoning capabilities of the smaller model?

scotty79• 3 years ago

I wonder if facebook could train LLM on all chat histories of all of its users.

Outperforming larger language models with less training data and smaller models

Related Stories