GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

https://www.threads.net/@ethan_mollick/post/C46AfItO8RS

by CharlesW • 2 years ago

161 127 2 years ago

threads.net

chintler• 2 years ago

$10 Million(M), not $10 Billion(B).

CharlesW3• 2 years ago

Thank you, fixed! Also, direct link to paper: https://arxiv.org/pdf/2305.05862.pdf

affgrff2• 2 years ago

Not looking forward for the times when an AI costs as much as an aircraft carrier.

moffkalast• 2 years ago

At least with an aircraft carrier you can make your money back by holding a small country for ransom, har har.

MuffinFlavored3• 2 years ago

Does anything currently beat GPT-4?

I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

monsieurbanana• 2 years ago

Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.

MuffinFlavored3• 2 years ago

The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)

rubymamis3• 2 years ago

A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax)

Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.

fragmede2• 2 years ago

Is there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.

cosmojg3• 2 years ago

Do you prefer Mistral-Large or Claude-Opus?

geor9e• 2 years ago

I don't know what system and user prompt you are testing with, but as one anecdote, Claude 3 Opus (and only Opus) consistently gives me better coding answers than GPT-4. Maybe it's the type of stuff I am doing or how I phrase things, who knows. I was using GPT-4 since the day it came out but haven't felt like going back so far.

jxdxbx• 2 years ago

Claude is better than GPT 4 for my uses, and was able to help me do some simple coding things that GPT 4 could not. It’s worth trying at least.

treprinum2• 2 years ago

Claude-2 in some tasks albeit it's a bit slower, Mistral on some tasks and it's a bit faster.

yieldcrv• 2 years ago

The benchmark is Sora or whatever Open AI is working on right now or next, not trying to beat the model released a year ago and still failing

so when looking at it that way, the real question is what do you need? all I need is Mixtral 7x8B Q5 in an 8,000 token context window, at the moment

I think there are plenty of other people that can design their applications and problems around lower fidelity tools, or just pursue something else

jasonjmcghee• 2 years ago

Claude Opus (largest v3 model) consistently outperforms GPT-4 for me. Better at following prompts, _feels_ much better.

thorum3• 2 years ago

I don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it.

Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.

simonw8• 2 years ago

I've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time.

As always, your results will vary based on your personal prompting style. My style apparently works great with Opus.

Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32...

Claude 3 Opus with the same prompt got it right the first time: https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83...

drexlspivey• 2 years ago

According to Chatbot Arena where people vote on responses blindly and an ELO rating is determined for each LLM, gpt4 is on top slightly ahead of Claude 3 Opus

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

marviel2• 2 years ago

Claude is excellent for brainstorming, being a thought partner, general knowledge acquisition tasks, and creative writing.

The one mixed-bag weak spot I've found is in coding -- It tends to make more "d'oh" mistakes while coding, but comes up with more creative solutions at the same time ¯\_(ツ)_/¯

dragonwriter• 2 years ago

Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

GPT-4 didn't figure that out, either; that’s just tooling built around the model, not something the model “figures out”.

jebarker3• 2 years ago

How do they know GPT-4 received no specialized financial training?

CharlesW3• 2 years ago

Meaning, they used the same generalized foundation model that all of us have access to, with no special fine-tuning, no retrieval-augmented generation, etc.

jebarker3• 2 years ago

I don't understand your point. To me GPT-4 is not a foundation model, it's been highly tuned for the chat task. Nobody outside of OpenAI knows what that fine-tuning really involved. So it's impossible to say how much finance specific data it was trained on (in pre-training or fine-tuning) or whether finance specific tasks were involved in fine-tuning.

__loam2• 2 years ago

GPT-4 cost like $100m so I don't think this is surprising?

rafaelero• 2 years ago

A lot of organizations still think they should have their own [finetuned] model to provide a custom experience to their users, so that may come as a surprise for them.

ShamelessC• 2 years ago

Scaling laws basically guarantee that a sufficiently larger general model will usually beat a smaller specialist model. The misunderstanding is perhaps acceptable but the headline here is essentially restating a well known property of deep learning.

minimaxir4• 2 years ago

Extremely hot LLM take: You will often get better results with few-shot prompting (with good examples) on a modern LLM than with a finetuned LLM.

Finetuning was the best option for weaker LLMs with lower context windows (e.g. the original GPT-3): both problems have been solved nowadays.

The cost economics are much better with few-shot prompting to modern LLMs too: input tokens are super cheap (especially with the recently-released Claude Haiku), so giving a lot of examples per call will still end up cheaper than finetuning.

Meanwhile, a finetuned ChatGPT costs 4-6x of normal ChatGPT usage.

viksit2• 2 years ago

can you give a few pointers on articles or examples of this?

minimaxir4• 2 years ago

A low-tech example to create a good blog post title for submission to Hacker News would be a system prompt like:

    You are an expert copywriter. Write five distinct blog post titles optimized for high clickthrough for Hacker News for the article the user provides.

    Your response must follow the style of these titles:
      - The ü/ü Conundrum
      - Why isn't preprint review being adopted?
      - Majority of web apps could just run on a single server
      - Weather Planning for Eclipse Day
      - PSChess – A chess engine in PostScript

Then provide the blog post as the user message input.

I just ran one of my blog posts (https://news.ycombinator.com/item?id=39495476) with the workflow through Claude Haiku and got this:

    Here are five distinct blog post titles optimized for high clickthrough on Hacker News for the article provided:

    1. Tipping ChatGPT: Does Offering Monetary Incentives Improve AI Text Generation?

    2. Quantifying the Impact of Incentives on Large Language Model Performance

    3. Carrot or Stick? Exploring the Effects of Positive and Negative Prompts on ChatGPT

    4. Gamifying AI: Using "Generation Golf" to Test ChatGPT's Ability to Follow Length Constraints

    5. The Curious Case of ChatGPT's Motivations: Can an AI Be Incentivized Like Humans?

Not bad titles, although more verbose than the 5 input examples I gave. I only gave 5 for simplicity: my main point is that you can give it a lot more than five and/or be more aggressive with constraints, like the blog post linked incidentially.

netdur• 2 years ago

Use Gemini 1.5 Pro, which has 1.5 million tokens. Prompt it with a logical question and observe it struggling to answer. Then, upload a book on logical thinking in PDF format and ask the same question again. Notice how it can now answer the question effectively.

thorum3• 2 years ago

That might be true for finetuning ChatGPT 3.5, but if you can finetune a small model (7B or less) to perform on par with GPT-4, while being faster and private, that’s a different story.

smallnamespace2• 2 years ago

You definitely can't in the general case (for example, your 7B model is never going to be able to help much with coding, fine tuning or no).

It can make sense if you have a particularly simple use case.

zapperdulchen• 2 years ago

Seems like the bitter lesson is still right: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Solvency2• 2 years ago

Whoa this guy says "computation" and not grammatically bastardized techbrospeak "compute" like some neckbeard equivalent of a caveman!

For that alone I commend him.

nialse• 2 years ago

For those who were oblivious to it, like myself, the bitter lesson is written by Richard S Sutton who invented reinforcement learning a long, long time ago.

tomrod• 2 years ago

This is an earth-shattering read.

pbronez• 2 years ago

I can’t access the article there… SSL error and then timeout. Here’s a link to the most recent WayBackMachine snapshot:

https://web.archive.org/web/20240321091803/https://www.incom...

marviel2• 2 years ago

Several MSFT AI/ML friends actively dissuaded me and my team from fine-tuning. They said that it's pretty clear in all their internal tests that it "lobotomizes" the general reasoning capabilities of the model, unless you're really careful.

"All work and no play makes GPT a very dull AI"

FrustratedMonky2• 2 years ago

"bitter lesson that building in how we think we think does not work in the long run"

Guess. Stop trying to shape the NN. And let it learn on its own.

Ambix• 2 years ago

Yes, that's what I've seen from a lot of my experiments with fine-tuning. One should be really careful to not "lobotomize" already capable model and achieve better results at the end. It's trickier than seems from multiple of tutorials.

But I believe that most of the data stored in foundation models are just useless for some particular domain. So it's better to forget something, getting really useful info instead.

jna_sh• 2 years ago

“Modern” is an extremely funny delineation given the small temporal window of this whole thing

ldjkfkdsjnv3• 2 years ago

Fine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Anyone starting an LLM application startup is arguably wasting their time, wait until the next iteration is out. Then you will know whats possible.

minimaxir4• 2 years ago

Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out.

Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.

ldjkfkdsjnv3• 2 years ago

Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.

treprinum2• 2 years ago

OpenAI-related startups are likely using GPT-5 already. Waiting it out won't help other startups, they will be too far behind.

rkagerer• 2 years ago

"Don't buy a computer today, because the faster one is coming out tomorrow"

simonw8• 2 years ago

Don't buy a computer today with a six month delivery lead time, because there's a company that releases computers with a same-day lead time with several improved models coming out next week.

hulium• 2 years ago

There is also the open source FinGPT, that is claimed to beat GPT4 in some benchmarks at a fine tuning cost of $17.25.

https://github.com/AI4Finance-Foundation/FinGPT

potatoman222• 2 years ago

One major advantage of FinGPT or Bloomberg's LLM is that the embeddings produced by the model can be used for downstream prediction tasks. GPT-4 does not expose its embeddings so it cannot be used for this.

bernawil• 2 years ago

sorry, noob here trying to make sense of this: you mean you can extract embeddings from the model file or that the embeddings are available in the repo and you can just use those files?

LASR• 2 years ago

I lead AI teams at my company. I've advised leadership against any kind of training / fine-tuning anything.

We're not in the business of training models. We will never be as good as OpenAI / Anthropic etc.

Where the real value in applications is smarter prompting techniques and RAG. There is a lot of room at the bottom in doing "dumb" things and simply feeding models with the right context to deliver customer value.

elforce002• 2 years ago

This. I work in a startup and told upper management we need to keep focusing on ML models that bring tangible benefits to our customers and then, try to integrate LLMs into their current flow instead of pivoting completely to LLMs. It seems they valued the input and now we're going for a hybrid approach.

jnwatson2• 2 years ago

It is trivial to fine tune these days. RAG is already irrelevant with large context windows.

Xenoamorphous3• 2 years ago

RAG is already irrelevant with large context windows

Just last Friday I took the contents of the 2024 folder of one of the teams at the company I work for, for which we use RAG at the moment. I dumped the text index, concatenated it and used Google’s API to return the token count, to see if it would fit in Gemini’s 1M context window; turned out it was 5.7M tokens. And that’s less than 3 months worth of documents for that team.

So yeah RAG is not dead yet, although I do question its usefulness, but that’s a separate topic.

simonw8• 2 years ago

Citation needed on "trivial to fine tune".

SgtBastard• 2 years ago

A remarkable comment in that it is clear, confident and wrong.

Fine-tunes lead to catastrophic forgetting.

RAG is only irrelevant if you’re completely disinterested in cost and latency.

We also don’t have enough data to gauge performance of models >200k context window size when reasoning over inputs of that size, much of which will be irrelevant to any particular user. Multiple random needles in haystack tests work flawlessly, but rarely applies to real world activity.

seydor• 2 years ago

Your advise is based on what?

throwaway74432• 2 years ago

Hear hear. I know a 3 person startup that has a "lead AI researcher" who is trying to train and fine-tune models. That's not their startup's purpose though... they have an actual product. So wtf are they doing? The lead AI guy thinks he's going to compete with these big companies and it's total fantasy.

LLMs are a commodity

qeternity2• 2 years ago

That does indeed sound crazy. But finetuning is also a commodity these days. You can train a good Mistral LoRA in under 24 hours on a single consumer GPU. We’re talking about $10 of compute.

You can run a dozen of these LoRAs atop the same base model on the same infrastructure for a dozen specific use cases.

The inference quality, performance and cost can all be substantially better than GPT4 with prompting.

VirusNewbie• 2 years ago

Doesn't it entirely depend on how specialized the training data for a given fine tuned model might be?

redox994• 2 years ago

That's a pretty odd stance. I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt.

You have to know when to RAG, finetune, or RAG+finetune.

kossTKR• 2 years ago

How narrow is the dataset to be outperforming greatly?

Just curious about what the usecase is for a 7b model in a business context - ie. what does it do?

singularity2001• 2 years ago

    greatly outperform GPT4 *for* just a prompt

your overfitting to training data convinces no-one that you created a "better GPT4"

simonw8• 2 years ago

"I've finetuned llama/mistral models that greatly outperform GPT4 with just a prompt"

If you write about your experiments with that in detail I guarantee you'll get a lot of interest. The community is crying out for good, well documented, replicable examples of this kind of thing.

thorum3• 2 years ago

Note that the benchmarks used for comparison are basically measuring the model’s ability to understand financial content. In other words, reading comprehension for English, just in a specific domain. It shouldn’t really be surprising that a strong generalist model performs well here.

On the other hand, GPT-4 actually did worse on the NER task - labelling and tagging terms used in the text - vs their finetuned model. I assume the finetuned model was better at using the specific labels they were targeting.

Rustwerks• 2 years ago

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

hallqv2• 2 years ago

This discussion is so dumb - finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token.

What Bloomberg did for $10M was not finetuning..

simonw8• 2 years ago

"finetuning a base model costs ~$1 with LORA/QLORA and can yield same performance as gpt-4, but at 1/100 of the cost per token"

That's a big claim - can you back that up with any examples?

Implicated• 2 years ago

I had opened a new tab back when this comment was just a few minutes old in hopes that when I came back there was some really great blog post linked with the details on the sorcery.

hallqv2• 2 years ago

https://arxiv.org/pdf/2402.00841.pdf

krasin2• 2 years ago

Finetuning LLMs is currently the most promising way for next-gen robotics. One of such works (PaLM-e) among other things measured the impact of finetuning on general purpose tasks: https://twitter.com/DannyDriess/status/1632904698108821505

In short, an 8B model could degrade almost 10x after being finetuned on robotics tasks, while 500B model experiences a very minor degradation (~4%) and there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning).

What I am saying is that while GPT-4 could beat a finetuned GPT-3.5 class model, I predict good things about finetuned GPT-4 class models, when they become practical outside of OpenAI/Google.

TMWNN2• 2 years ago

there's a hope that with a larger-sized model, it could become either zero or even negative (adding more experience improves general purpose reasoning).

The best single work of fiction ever created about LLMs' capabilities (and, perhaps, dangers) is Colossus by Jones. Although I think the film is even better than the book, only the latter mentions how, despite being created specifically for US national defense, Colossus is also fed unrelated data including Shakespeare's sonnets, because its creators do not know if it could be important.

paulmd• 2 years ago

What good is a revolution without dancing?

gradascent• 2 years ago

Interesting! I would like to learn more about how AI is being applied to robotics. Do you have any suggestions for how to keep up with developments/ideas in this field?

krasin2• 2 years ago

These two links could be a good start:

ALOHA-2: https://aloha-2.github.io/

RT-X: https://robotics-transformer-x.github.io/

hlfshell2• 2 years ago

In October I wrote a blogpost on this subject: https://hlfshell.ai/posts/llms-and-robotics-papers-2023/

..and plan to do an updated version soon for much of what's been released since. I've also done work related to LLM and robotics integration, also on that site.

Happy to chat about it.

atleastoptimal• 2 years ago

Yeah, the equivalent is: would it be better for a quant firm to spend 200 thousand dollars giving a first-class specialist education to a guy with an IQ of 95, or just hiring a guy with an IQ of 150 straight out of college.

kcorbitt• 2 years ago

IMO it's possible to over-generalize from this datapoint (lol). While it's true that creating a general "finance" model that's stronger than GPT-4 is hard, training a task-specific model is much easier. Eg. "a model that's better than GPT-4 at answering finance-related questions": very hard. "A model that's better than GPT-4 at extracting forward-looking financial projections in a standard format": very easy.

And in practice, most tasks people are using GPT-4 for in production are more like the latter than the former.

(Disclaimer: building https://openpipe.ai, which makes it super easy to productize this workflow).

jonplackett• 2 years ago

FYI it was GPT-3.5 Class not GPT3.5.

A lot of models claim to be GPT3.5 class that clearly are not in the first place.

GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

Related Stories