Fine-tuning now available for GPT-4o

https://openai.com/index/gpt-4o-fine-tuning/

by davidbarker • 2 years ago

216 87 2 years ago

openai.com

hubraumhugo• 2 years ago

Curious to hear some real-world use cases for fine-tuning and what your setup looks like in terms of:

- frameworks

- hardware

- foundation models (OS vs 3rd party)

- evaluation & training data

Did the performance gains justify the investment for fine-tuning?

darksaints2• 2 years ago

I think vision models have a lot bigger ROI from fine tuning than language models. That being said, I do consider fine-tunes to be helpful in improving smaller models as long as the domain is limited in scope. In other words, fine tuning allows you to get similar performance in smaller models, but improved performance in larger models seems pretty elusive at the moment, albeit somewhat possible with very creatively engineered training datasets.

An example of this is DeepSeek-Coder, which can essentially be considered a fine-tune of a fine-tune of a Mixtral model. It performs very similarly to Claude 3.5 Sonnet, which is pretty damn impressive, but it does it at less than 1/10th the cost.

What I don't understand though is why anyone would even remotely consider fine tuning a GPT-4o model that they will never fully own, when they could spend the same resources on fine tuning a Llama3.1 model that they will own. And even if you absolutely don't care about ownership (???), why not do a fine tune of an Anthropic model which is already significantly better than GPT-4o. At this point, with the laggard performance of OpenAI and their shameless attempts at regulatory hostility to competitors, I can't imagine ever giving them any of my money, let alone owning my derivative work.

simonw8• 2 years ago

"An example of this is DeepSeek-Coder, which can essentially be considered a fine-tune of a fine-tune of a Mixtral model"

I've not heard that anywhere else. My impression from https://huggingface.co/deepseek-ai/deepseek-coder-33b-base was that DeepSeek-Coder was trained from scratch.

jstummbillig3• 2 years ago

When Cosine fine tunes a model to be generally better at coding than gpt-4o that leaves me vaguely confused about the role that each part plays in building the best LLMs (and I would not be shocked to learn that this confusion was also SOTA)

everforward2• 2 years ago

I don’t think it’s all that surprising. VRAM is much smaller than all the data the bot was trained on, so some stuff gets dropped or “lossily compressed”. In the case of foundation models, this is likely a bit of everything.

Fine tunes take that and change what gets dropped or compressed. Rather than somewhat evenly forgetting things, it forgets very little of the knowledge they’re targeting (like coding) and in exchange drops much more of “everything else”.

I don’t believe it works this way, but in a sense, it’s like GPT4o is a 400B parameter model but only devotes 20B parameters to coding because the rest are taken up by Wikipedia and knowing French and what not. A 70B coding fine-tune might be able to devote more than 20B parameters to coding, in exchange for only speaking English, having very little encyclopedic knowledge, etc.

It’s kind of like CPUs vs ASICs. CPUs (GPT4o) will perform better on average at random tasks, while ASICs (finetunes) do dramatically better on the task they were built for, even if they have a lower transistor count.

jstummbillig3• 2 years ago

But is that not what MoE is supposed to solve already? Why no it bake whatever Cosine does into the coding expert?

johntb86• 2 years ago

GPT-4o has seen a lot of examples of what people write about coding on the web, what code exists, and what tasks people want to do with that. But that general data doesn't include full process of coding - get a bug report, look at a codebase beforehand, make changes to the codebase, test them and see exactly what bugs they have, and iterate. That process is what SWE-bench tests.

It's possible OpenAI did some coding fine-tuning themselves; Meta's Llama 3 paper [0], section 4.3.1 mentions what sort of work is needed. However, anything OpenAI did is based on their own tooling and set of assumptions - e.g. how is the existing code input into the LLM, what set of actions can the LLM take (e.g. look up documentation), what language is the output code being written in, etc. Cosine's LLM framework may do things differently and have different features, so you'd need to fine-tune the LLM to take maximum advantage of the framework.

It's like dropping the LLM down in front of Vim when it had only ever used or even heard of notepad (or even emacs); there needs to be some training to make it work well with the new tools it has.

[0] https://arxiv.org/pdf/2407.21783

simonw8• 2 years ago

I'm still looking forward to a fine-tuning vendor publishing an interactive real-world example of one of their fine-tuned models, accompanied by the exact training data that was used to create it.

OpenAI's examples here are artificial, and there's no mechanism to try out the results: https://platform.openai.com/docs/guides/fine-tuning/fine-tun...

Fine-tuning is expensive in terms of both time and money (more so time these days, a lot of the vendors have free trials now). Before I put that work in I want to get a much better idea of the kind of results I can expect.

bob10292• 2 years ago

Fine tuning always felt like something that only works at incredible scale with a lot of data.

Prompt engineering, RAG, etc. provide so much more uplift per unit time invested. Maybe fine tuning makes sense if you have a 9 figure budget and thousands of humans to throw at it. I don't feel this works in a typical startup ecosystem. I couldn't move the needle with the amount of data I had on hand.

btown• 2 years ago

To this point, guaranteeing that your fine-tuning doesn't actually weaken the hallucination-resistance of state-of-the-art base models, by overfitting to specific examples in your training set, is a really hard thing to measure. Unless you have robust ways of validating LLM output so that you can do real cross-validation, few-shot prompts, where you can instruct the LLM exactly how much you want it to generalize from those examples, may be far better at achieving many goals.

qudat2• 2 years ago

What about techniques like LoRA?

boredemployee• 2 years ago

perhaps my ignorance and my beginner level on the topic are biasing me, but in my experience, fine-tuning the GPT-3.5 turbo has elevated our operation to another level. When we put it into production, it became much faster, more accurate, and cheaper when compared to original gpt4. Our use case is to make unstructured data return a JSON with some key-value entities where the order matters.

satvikpendem5• 2 years ago

How are people feeling with regards to GPT-4o versus Claude 3.5 Sonnet? I recently watched this Primeagen video [0] about how, because LLMs don't actually understand anything (yes, AI Effect included [1]), one does not actually gain as much usefulness as they'd expect, especially with subtly wrong outputs. Over time, it just wastes way more time and becomes a form of learned helplessness (and yes, I do know about Socrates' dialogue, I saw it originally elsewhere on HN and had been quoting it for some time [2]).

[0] https://www.youtube.com/watch?v=1-hk3JaGlSU

[1] https://en.wikipedia.org/wiki/AI_effect

[2] https://news.ycombinator.com/item?id=40920318

carbocation• 2 years ago

I feel like this comment was made in good faith so I wanted to explain my downvote: I think it's just too far off-topic for this particular announcement. It's better as a separate discussion.

satvikpendem5• 2 years ago

Thanks, however I thought it was fairly aligned simply due to 4o receiving the same sorts of functionality as Claude had for a while.

logicchains2• 2 years ago

I'll also explain my downvote, it's because of the assertion that LLMs "don't actually understand anything", which to anyone who's actually successfully used LLMs to solve a difficult problem is clearly false, unless you use some contrived definition of the word "understand" that doesn't match how the word is actually used in normal conversation.

teaearlgraycold• 2 years ago

The bigger ones have gained a rough understanding of a few systems [1]. Which is really impressive and gives an answer to the Chinese Room experiment. In my experience they don’t understand a lot of things I ask about very well. But the fact that they understand anything at all is impressive.

1. https://danangell.com/blog/posts/gpt-understands/

satvikpendem5• 2 years ago

I am referring specifically to what was claimed in the video I quoted, so, unless you watched the actual video, what you are saying has not much bearing on what I am actually saying. Sorry to say it, and not to be harsh, but I constructed my comment to specifically point to such an instance via bracket quotes, please respond to what was said in said video.

bearjaws• 2 years ago

https://myswamp.substack.com/p/ymxb-is-hard-for-llms

I recently made this blog post showing how y=mx+b is very error prone in GPT4o, and pretty accurate (to a point) in Claude 3.5.

I haven't gone down the rabbit hole yet, but I was wondering if fine tuning could fix math errors in LLMs. My initial hunch and understanding is it will not. I'll have to give your links a read/watch.

kkukshtel2• 2 years ago

Through this same lens, using Google is a form of learned helplessness.

satvikpendem5• 2 years ago

If you follow [3], yes, even learning to read and write is a form of learned helplessness, as Thoth and Socrates conclude. Now, you might not thing that affects our day to day world, but it does; because imagine if we could not read nor write, what sorts of hypotheses we might come up with, now extend that to what we say about LLMs. To those that say those are inequal circumstances, I invite you prove why. Thoth and Socrates know a hell of a lot more than you ever did, to be frank.

viccis• 2 years ago

learned helplessness

This is an existing phrase with an explicit meaning that you are clashing with.

tines• 2 years ago

What kind of data or data formatting do you need to fine-tune GPT-4o? I'd love to throw a bunch of documentation at it and let it learn, but I don't have the resources to extract knowledge, format it as questions and answers, etc.

simonw8• 2 years ago

Fine-tuning generally isn't an effective way to add extra knowledge from things documentation - my understanding is that the vast amounts of knowledge in the original training data tend to overwhelm any extra knowledge you try to add by fine-tuning.

OpenAI's documentation has good examples of how the data should be formatted (and when it's appropriate to fine-tune): https://platform.openai.com/docs/guides/fine-tuning/fine-tun...

qudat2• 2 years ago

does the same result apply to LoRA?

floam• 2 years ago

https://platform.openai.com/docs/guides/fine-tuning

gamegoblin2• 2 years ago

Seems like their competition (Google, Anthropic) have both shipped prompt caching, which is a lot more developer-friendly and gets a lot of the same benefits as fine-tuning.

You just cache a prompt with a ton of examples that you would have otherwise fine-tuned on. You can trivially update that prompt whenever, no asynchronous fine-tuning job needed.

Wonder if OpenAI will stick with fine-tuning or go towards prompt caching (or both). Fine-tuning has uses, but prompt caching gets you 99% of the benefits for 1% the effort.

eightysixfour4• 2 years ago

They, in my experience, result in different outcomes. Fine-tuning is better at shaping the response, for example getting a correct JSON format, or making the responses shorters and concise, or use emojis, or whatever. Long prompts are better for in-context learning, fine-tuning doesn't seem to impart knowledge well, it just increases or decreases the likelihoods of existing knowledge coming out.

maeil• 2 years ago

Genuinely curious as you have experience with fine-tuning, which I don't yet. Given how simple it has become to get the correct response shape, do you feel there's still a point to it? If you need JSON and want to be very sure you can just use function calling, if you only need simple boundaries, use Claude-like XML, if it's something very complicated you can give a few shots.

From my understanding, fine-tuning allows for reducing model size, meaning less latency and cost. That seems like it would be the biggest advantage, no?

gamegoblin2• 2 years ago

I agree that fine-tuning isn't good at imparting knowledge, though in my experience, K-shot prompting is very close to as good as fine-tuning at getting output formats (for recent models, at least)

So except for cases where you are really trying to perfectly match a particular writing style and want to fine-tune on loads of some author's text, I think prompt caching mostly dominates fine-tuning for most real-world workloads.

I'd still fine-tune if I wanted my model to have a really particular "voice" though.

WanderPanda• 2 years ago

I would have expected fine-tuning to be good at imparting knowledge. Pre-training is often done for a single epoch only and models soak up the knowledge like crazy without multiple passes so why would fine-tuning be any different?

esafak• 2 years ago

Does their prompt caching work with semantically equivalent queries, or do they match lexically?

simonw8• 2 years ago

You have to mark exactly which parts of the prompt you want to cache.

The only vendor I've seen with "automatic" prompt caching so far is DeepSeek https://platform.deepseek.com/api-docs/news/news0802/

KTibow• 2 years ago

OpenAI has a mysterious reference to caching in their guide to latency.

Maximize shared prompt prefix, by putting dynamic portions (e.g. RAG results, history, etc) later in the prompt. This makes your request more KV cache-friendly (which most LLM providers use) and means fewer input tokens are processed on each request.

source: https://platform.openai.com/docs/guides/latency-optimization

okl• 2 years ago

Sounds very similar to how e.g. docker/layers work - the later you put the more dynamic stuff, the higher the change that previous layers are reused. Meaning the cache entries are likely chained, i.e., each one depends on the preceding entry.

htrp• 2 years ago

How do you know it's not prompt caching in the backend?

simonw8• 2 years ago

OpenAI haven't documented if they do this, but it should be reasonably easy to determine via experiments against their API.

Claude's prompt caching gives a very material performance improvement - they claim up to a 4x performance boost in https://www.anthropic.com/news/prompt-caching - so if OpenAI have similar techniques, even undocumented, they should become visible through sending the same prompt a bunch of times and measuring the latency.

nprateem• 2 years ago

IMO they are definitely doing something. I've had countless occasions where the first few times I run a prompt I get excellent output, then several tries later it just returns the same dumb-sounding AI shite.

impossiblefork• 2 years ago

It's also pretty much required if you want to do anything complicated with prompt networks or agent-type stuff.

Without caching it's too expensive.

thomasahle• 2 years ago

Do any vendors allow fine-tuning on images yet?

Someone1234• 2 years ago

When are they going to fix ChatGPT Classic not using Custom Instructions?

chartpath• 2 years ago

Some much focus on fine-tuning when it can actively make performance on reasoning and planning benchmarks worse (over a baseline of already worse-than-coin-toss).

Why not give us nice things for integrating with knowledge graphs and rules engines pretty please?

simultsop• 2 years ago

Just catching up the phrase of marketing "this is just the start ...". Isn't this overused?

underyx3• 2 years ago

How do y'all deploy fine-tuned models? I have separate projects for staging and prod, but it doesn't seem like a fine-tune can be shared across projects.

Am I wrong to split projects by env? Am I expected to run fine-tunes separately per env (surely not)? Am I missing an option to share fine-tunes across projects?

simonw8• 2 years ago

Are you talking about fine-tuned models that you host yourself, or fine-tuned models from a hosted provider like OpenAI?

What do you mean by an "env" here?

underyx3• 2 years ago

I mean OpenAI hosted fine-tunes (same as referenced in OP.)

I have a staging deployment and a production deployment. Ideally anything that I roll out to production, I can try on staging first — including switching from gpt-4o to a fine-tuned gpt-4o. I don't want the production API key to be accessible by my staging app, so I have two separate projects within the OpenAI dashboard. One is called my-app-prod, and the other is my-app-staging.

To illustrate the problem further, I also have infrastructure to eval models before changing what production is running. The eval infrastructure also has its own OpenAI project, so that I can set separate billing limits on running evals. Any fine-tuned model needs to be benchmarked first, but again, I'm not sure how to make the same fine-tune available to both the eval project, and the production app project.

FlamingMoe• 2 years ago

It's wild how quickly the AI landscape shifts. Just 6 months ago, I couldn't go a day without using ChatGPT. Now, it's like finding an old flip phone in a drawer. I've gone full Claude convert. Wonder where I'll be six months from now.

aantix• 2 years ago

Github's Copilot Workspace or Cursor's new composer beta feature. That's where you'll be in 6 months.

Multi-file code suggestions with an intimate understanding of the entire code base.

https://x.com/aantix/status/1819794837375263228

P.S. Or Plandex, https://plandex.ai/

rd• 2 years ago

Cursor + Sonnet feels like a cheat code right now. Feels similar to the excitement I had the first day I ever used GitHub copilot.

I saw this this morning - it really made me think about what the future of 1-man non technical founder startups will look like. https://x.com/0xluffyb/status/1825854097481736479?s=42&t=7-X...

greenchair• 2 years ago

I was going to try copilot workspace till during signup process it asked for access to my private repos then noped out of there. Why would they need that unless they wanted to use it to train off my data?

bachback• 2 years ago

how you find it better? haven't seen the benefits

machiaweliczny2• 2 years ago

I have sub to Claude now due to projects/artifacts (caching really) but aider with new gpt4o works well

ziofill• 2 years ago

I think I’m not understanding what goes into making this available. Fine-tuning is a known technique, why is it available “only” now?

HarHarVeryFunny• 2 years ago

Because this isn't you fine-tuning an open weights model and running it yourself - this is fine-tuning via OpenAI's API and them hosting the resulting model (and everybody else's differently fine-tuned ones) and running it for you.

WhitneyLand2• 2 years ago

Say I want to ask a model about a code base 500k tokens in size. The questions would not be to generate code, but rather to help understand it in a way that require the model to globally reason about everything at once.

How might performance compare between:

1.) Using a model like Gemini to load it all into context at once

2.) Using one of the various summarization systems/embedding/RAG etc.

3.) Fine tuning the whole code base into Gpt-4o

spullara• 2 years ago

1 then 2. Never 3.

WhitneyLand2• 2 years ago

Interesting why would you say that?

There’s not been much opportunity previously to easily fine tune a Gpt 4 class model. I haven’t seen anything written up on this being tried.

Liquix• 2 years ago

tools like aider [0] maintain a large context with minimal token usage by reading a repo's .git folder. impressive results

[0] https://github.com/paul-gauthier/aider

trees101• 2 years ago

also, aider has added experimental support for prompt caching: https://github.com/paul-gauthier/aider/issues/1086#issuecomm...

I really like the control you get with aider over the LLM context. You can /add or /drop source code, markdown notes. You can /clear the chat. /tokens shows you the context and the cost, you can see what each prompt will cost you.

I find aider best used in conjunction with a git diff view in VSCode, I run aider with --no-auto-commits and then manually review each time in VSCode.

I'm keen to learn any AI coding workflows if anyone has any links. I've benefitted greatly from tips such as using type hints and documentation for the LLM's benefit.

machiaweliczny2• 2 years ago

I have 5 lines shell to load git into single file and using google AI studio. Tested it to simplify dreamcoder implementation but results are ~ so far

highfrequency2• 2 years ago

Huh? It says: "Cosine's Genie achieves a SOTA score of 43.8% on the new SWE-bench Verified benchmark" with a link to https://www.swebench.com/

But the SWE-bench leaderboard (linked to in the post) doesn't show Cosine Genie at all, instead showing Amazon at the top with 38.8% accuracy.

If true, seems wild that Genie, a 10 person startup with $2.5m in funding can actually achieve SOTA results over Google, Amazon, Microsoft, Anthropic, OpenAI etc. It's not like the big players are overlooking this problem of using LLMs to automate software engineering. Anyone have more color on this? I see some speculation online that the training data can easily get contaminated with benchmark questions, but not much careful evidence.

jstummbillig3• 2 years ago

"verified" being the (rather confusing) keyword here.

https://openai.com/index/introducing-swe-bench-verified/

highfrequency2• 2 years ago

Your link doesn't actually point to the leaderboard. This link does https://www.swebench.com/ and you can click on the "Verified" tab. I don't see any entry for Genie.

hansonw• 2 years ago

It looks like they didn't want to make a public submission in order to avoid disclosing the model internals: https://cosine.sh/blog/genie-technical-report#:~:text=SWE%2D....

JamesBarney• 2 years ago

It's with an asterisk. Here's their comment from cosine's website.

Note SWE-Bench has recently modified their submission requirements, now asking for the full working process of our AI model in addition to the final results -their condition to have us appear on the offical leaderboard. This change poses a significant challenge for us, as our proprietary methodology is evident in these internal processes. Publicly sharing this information would essentially open-source our approach, undermining the competitive advantage we’ve worked hard to develop. For now, we’ve decided to keep our model’s internal workings confidential. However we’ve made the model’s final outputs publicly available on GitHub for independent verification. These outputs clearly demonstrate our model’s 30% success rate on the SWE-Bench tasks.

Der_Einzige• 2 years ago

There seems to be no information on how this fine-tuning is implemented, and this matters, because parameter efficient techniques minimize catestrophic forgetting, which is most of what causes fine-tuned models to perform worse.

Until this happens, and until we have flexibility of fine-tuning techniques, I will push investors still towards PyReft, Lora, related, and the open source models where all of these techniques work.

celdon25• 2 years ago

Fine-tuned models remain entirely under your control

Also on the very next paragraph:

We’ve also implemented layered safety mitigations for fine-tuned models to ensure they aren’t being misused.

Well done, Sam

bionhoward• 2 years ago

still has a customer noncompete so you’re still paying to get wrecked long term

Fine-tuning now available for GPT-4o

Related Stories