Function calling and other API updates

https://openai.com/blog/function-calling-and-other-api-updates

by staranjeet • 3 years ago

377 159 3 years ago

openai.com

eminence32• 3 years ago

The changes/improvements to `gpt-3.5-turbo` are very welcome, considering how hard it is to get access to gpt-4

ignoramous• 3 years ago

The blog post indicates that GPT-4 (0613) will be generally available soon.

  gpt-4-32k-0613 includes an extended context length for better comprehension of larger texts.
  
  With these updates, we'll be inviting many more people from the waitlist to try GPT-4 over the coming weeks, with the intent to remove the waitlist entirely with this model.

wilg2• 3 years ago

LLMs orchestrating, pulling data from, and coordinating between existing systems in this manner seems very powerful. I feel like we haven't even really seen many of the possibilities there.

voiper1• 3 years ago

... or the hacking. They mention exploring the security implications.

wilg2• 3 years ago

I'm not living in fear of that.

demarq• 3 years ago

20 pages of text in a single request.

Yessssssss!

aarkay• 3 years ago

Function calling is a great feature. I've been using LLMs for function calls for the past few months and gpt-4 has worked great for this out of the box. Awesome to see both the models specifically trained for this.

minimaxir• 3 years ago

The big feature here is the function calls, as this is effectively a replacement for the "Tools" feature of Agents popularized by LangChain, except in theory much more efficient since it may not require an extra call to the API. In the case of LangChain which selects Tools and their functional outputs through JSON Markdown shennanigans (which often fails and causes ParsingErrors), this variant of ChatGPT appears to be finetuned for it so perhaps it'll be more reliable.

While developing a more-simple LangChain alternative (https://github.com/minimaxir/simpleaichat) I discovered a neat trick for allowing ChatGPT to select tools from a list reliably: put the list of tools into a numbered list, and force the model to return only a single number by using the logit_bias parameter: https://github.com/minimaxir/simpleaichat/blob/main/PROMPTS....

The slight price drop for ChatGPT inputs is of course welcome, since inputs are the bulk of the costs for longer conversations. A 4x context window at 2x the price is a good value too. The notes for the updated ChatGPT also say "more reliable steerability via the system message" which will also be huge if it works as advertised.

misterdata• 3 years ago

As they are accepting a JSON schema for the function calls, it is likely they are using token biasing based on the schema (using some kind of state machine that follows along with the tokens and only allows the next token to be a valid one given the grammar/schema). I have successfully implemented this for JSON Schema (limited subset) on llama.cpp. See also e.g. this implementation: https://github.com/1rgs/jsonformer

newhouseb2• 3 years ago

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation:

  The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters).

So sadly, it is just fine tuning. There's no hard biasing applied :(. You were so close, but so far OpenAI!

[1] https://github.com/newhouseb/clownfish

[2] https://platform.openai.com/docs/guides/gpt/function-calling

Mockapapella• 3 years ago

This is a really clever approach to tool use. I'll definitely be experimenting with this trick. Previously I had a grotesque cacophony of agents and JSON parsers. I think this will do a lot to help (both the process and my wallet)

alexp7• 3 years ago

There is also an alternative approach for running code with ChatGPT, the way Nekton(https://nekton.ai) does it. It will use ChatGPT to generate typescript code code, and then just run it in the cloud.

In the end you get similar result - AI generated automation, but you have an option to review what the code will actually do before running it.

andai• 3 years ago

While using Auto-GPT I realized that for most usecases, a simple script would have suited my needs better (faster, cheaper, deterministic). Then I realized those scripts can (a) be written by GPT, and (b) call into GPT!

singularity2001• 3 years ago

Not sure how well it scales if you need to provide a function definition for every conceivable use case of 'external data': functions": [ { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } ]

Escapado• 3 years ago

This is great. I wonder if the price decrease comes from the competition (on the side of anthropic and from local LLMs). If so, I guess we will have to wait for a general GPT-4 competitor to come along before we see price decreases there aswell. Right now it's quiet expensive. We are incorporating it in a new product in the education space and we have to be fairly conservative in rate limiting things so that the cost won't go out of hand.

I also wonder how much of an impact the new Nvidia HGX systems will have on the medium term infra cost on running these services and whether we will see some benefits from that.

ilaksh7• 3 years ago

Right this is a very exciting release but disappointing that there was no price reduction at all or rate limit increase for gpt-4. I guess it just uses a lot of GPU and RAM.

kgwgk3• 3 years ago

Why would they reduce the price when they have a waiting list?

dragonwriter2• 3 years ago

Right this is a very exciting release but disappointing that there was no price reduction at all or rate limit increase for gpt-4

They are planning on reducing the pricing from $infinite to $current-listed (or, viewed another way, to increase the quota from 0 to current-listed) by clearing the waiting list.

This, obviously, doesn’t benefit (may even, competitively, hurt) those who already have GPT-4 access, but for everyone else, its a win.

bioemerl4• 3 years ago

I hate seeing these guys succeed because everyone of their successes is a new day that AI becomes less accessible to the average person and more locked behind their APIs.

sashank_15092• 3 years ago

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference, and even then it will be a slow, unpolished experience.

On the other hand OpenAI’s API makes things blazingly fast and dirt cheap to the average consumer. It honestly does feel like they have enabled the power of AI to be accessible to anyone with a laptop. If that requires fending off competition from Behemoths like Google, Meta by not releasing model weights then so be it. This critique would be more apt to Nvidia who are artificially increasing datacenter GPU prices thus pricing out the average consumer. OpenAI is doing the opposite.

hackernewds2• 3 years ago

An "average" person is not someone who knows how to call an API. Perhaps only on HN

wellthisisgreat2• 3 years ago

LLAMA 65B model at a very slow pace

How does it compare to GPT 3.5, or 4? I mean if you ask the same questions. Is it usable at all?

I tried the models that work with 4090 and they were completely useless for anything practical (code questions, etc.). Curiosities sure, but on Eliza level.

bioemerl4• 3 years ago

. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference

Give it a decade and you might be able to, but without the model you'll never have the option.

ilaksh7• 3 years ago

I have been thinking about trying to do LoRA style fine tuning of Flacon-40b or Falcon-7b on RunPod. The new OpenAI 16k context and functions thinking made me lose the urge to get into that. Was questionable whether it could really write code consistently anyway even if very well fine tuned.

But at least that is something that can be attempted without $25k.

error9348• 3 years ago

Less accessible compared to what?

egonschiele6• 3 years ago

- The API is extremely cheap

- There are plenty of open source tools built on top if it (example list: https://github.com/heartly/awesome-writing-tools)

While I wish this work was open, they are both the best and cheapest option out there... by a mile.

bioemerl4• 3 years ago

This is exactly my problem. They are doing quite well, and closing the door behind them. Open AI isn't your friend and reserves the right to screw you down the line.

TeMPOraL2• 3 years ago

Thing is, as long as the field is growing in capabilities as fast as it is, there isn't going to be any kind of "democratizing" for an average person, or even average developer. Anything you or me can come up with to do with LLM, some company or startup will do better, and they'll have people working full-time to productize it.

Maybe it's FOMO and depression, but with $dayjob and family, I don't feel like there's any chance of doing anything useful with LLMs. Not when MS is about to integrate GPT-4 with Windows itself. Not when AI models scale superlinearly with amount of money you can throw at them. I mean, it's cool that some LLAMA model can run on a PC, provided it's beefy enough. I can afford one. Joe Random Startup Developing Shitty Fully Integrated SaaS Experience can afford 100 of them, plus an equivalent of 1000 of them in the cloud. Etc.

Yeah, I guess it is FOMO and depression.

com2kid4• 3 years ago

GPT3.5 has been undergoing constant improvements, this price decrease (and context length increase) is great news!

The main problem I see with people using GPT3.5 is they try and ask it to "write a short story about aliens" and then they get back a crap boring response that sounds like it was written by an AI that was asleep at the wheel.

Good creative prompts are long and detailed, and to get the best results you really need to be able to tune temperature / top_p. Even small changes to a 3 paragraph prompt can result in a dramatic changes in the output, and unless people are willing to play around with prompting, they won't get good results.

None of the prompt guides I've seen really cover pushing GPT3.5 to its limit, I've published one of my more complicated prompts[1] but getting GPT3.5 to output good responses in just this limited sense has taken a lot of work.

As for the longer context, output length is different than following instructions, especially for a lot of use cases, pushing more input tokens is of as much interest as having more output tokens.

From what I have explored, even at 4k context length, with a detailed prompt earlier instructions in the prompt are "forgotten" (or maybe just ignored). The blog post calls out better understanding of input text, but again, I hope that isn't orthogonal to following instructions!

Finally in regards to function outputs, I wonder if it is a second layer they are running on top of the initial model output. I have always had a challenge getting the model to output parsable responses, there is a definite trade off between written creativity and well formatted responses, and to some extent having a creative AI extend out the format I specify has been really nice because it has allowed me to add features I did not think of myself!

[1] https://github.com/devlinb/arcadia/blob/main/backend/src/rou...

hackernewds2• 3 years ago

Would you mind sharing what are then good prompt structures? Seems you have a grasp

com2kid4• 3 years ago

A number of key points:

1. Give lots of examples, you can see in my shared prompt that I include plenty of different examples of things that can happen.

2. The system prompt is important, choose a style you want things written in and provide some context about what the writing will be used for

3. Restrictions create art! My prompt forces GPT to summarize almost every paragraph, which means the things that get written are things that can be summarized with a few emojis.

4. Keep playing with it, use the GPT playground to experiment with different settings.

5. Settings that allow the AI more leeway also result in prompt instructions being ignored, you need to decide where on the scale you are comfortable operating. At one point GPT3.5 was generating (good!) dialogue, which sadly wasn't what I wanted, but I could have chosen to embrace that and go with it.

6. Once you feel a good trend, keep on generating! Occasionally GPT pops out a really good story, maybe 4 or 5 out of the hundreds of stories I've seen have been truly memorable! Ideally I'd be able to prompt engineer to get more of those, but sadly the genre I am writing for (medieval fantasy drama) is right at the edge of ChatGPT's censorship rules.

At one point I actually asked GPT 4 to rewrite my GPT3.5 prompt, and the prompt it came back with resulted in much lower levels of creativity, all the generated text was of the form "A does B, resulting in C", the sentence structure just got really simplified.

Even when asking for summaries, be specific! My summary prompt (not yet pushed to GH sadly) is something like:

"After these instructions I will send you a story. Write a clickbait summary full of drama, limit the summary to 1 sentence and do not spoil the ending."

Compare that to just "summarize the following story."

An example of what output from the crafted prompt may look like:

"When the king of Arcadia fell ill, his children fought to the death to rule the kingdom."

vs the naive prompt:

"King Henry became sick and died. His two sons, John and Tim, fought over who would rule. In the end Tim killed John and became the new king."

TheCaptain4815• 3 years ago

Could you share any tips? I'm always looking to learn more on Prompting, especially with 3.5.

com2kid4• 3 years ago

See my comment at https://news.ycombinator.com/item?id=36314428 !

phillipcarter• 3 years ago

None of the prompt guides I've seen really cover pushing GPT3.5 to its limit, I've published one of my more complicated prompts[1] but getting GPT3.5 to output good responses in just this limited sense has taken a lot of work.

Completely agree. We use gpt-3.5 in our feature and it works really well! After my blog post where I detail some of the issues [0] I got a lot of people asking me questions about how we got gpt-3.5 to "work well" because they found it wasn't working for them compared to gpt-4. Almost every time the reason is that they weren't really doing good prompting and expected the magic box to do some magic. The answer is...prompt engineering is actual work, and with some elbow grease you can really get gpt-3.5 to do a lot for you.

[0] https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-...

zora_goron• 3 years ago

Definitely agree about prompts -- for MedQA [0] I ended up building up a prompt around 300 words long to get a collection of results I was aiming for. I'm still not sure about the best way to go about building a "stable" lengthy prompt that can maintain a predictable output even after adding to it; my approach was mainly via trial-and-error experimentation.

[0] https://labs.cactiml.com/medqa

avereveard3• 3 years ago

Good creative prompts are long and detailed

They don't need to be tho. You can try shotgunning in (generate 100 titles about a novel around aliens, after the gen 'pick the one most likely to resonate to a X audience, explain why')

Or you can let AI drive itself interactively (ask yourself 20 question about how to write creative alien stories, and answer yourself)

Or you can process in spirals (generate a setting for an alien story, wait answer, generate 3 protagonista and one antagonist, wait, generate motives and relationships for each of them, wait, generate a backstory, wait, then you ask for the novel)

The point is letting the ai do the work. You can always "rewrite it with more drama and some comedic relief" afterward to fix tonal issues.

gremlinsinc• 3 years ago

You can also try and convince it, that it's one of the Duffer brothers behind stranger things, and you need to create the next great series like that in book format, etc... Then steer it away from being a tit for tat, obvious rip-off as you go through chapter development.

com2kid4• 3 years ago

Or you can let AI drive itself interactively (ask yourself 20 question about how to write creative alien stories, and answer yourself)
Or you can process in spirals (generate a setting for an alien story, wait answer, generate 3 protagonista and one antagonist, wait, generate motives and relationships for each of them, wait, generate a backstory, wait, then you ask for the novel)

Both of these techniques work very well, but are not as applicable to programmatic access without wrapping things in a complicated UI flow. My focus is on public facing website so I want to avoid multiple prompts if at all possible!

JaviFesser• 3 years ago

I'm seeing that same problem. Most of the blog posts in storybot.dev suffer that problem. They are too generic.

The only interesting ones have a lot of detail in its prompts.

muzani• 3 years ago

Honestly, the good old davinci model has proven to be much better at writing for me. 3.5 feels overtrained.

You also have to give it a good sample of how to write, else it will write at the average quality of fiction that it has been fed.

Here's something by GPT-4: https://chat.openai.com/share/ab2fc479-f3f9-4bf9-b625-e2aab7...

The same prompt with GPT-3.5: https://chat.openai.com/share/be81167d-11eb-4f38-b1b4-b6f592...

The original plot was actually generated by davinci, which I think is the most creative of the three. 3.5 for price and speed, GPT-4 has rationality and experience, and davinci has his head up in the cloud.

RugnirViking• 3 years ago

I just tell it how to write a good story before asking for a story (show, don't tell. Don't list descriptions each time a new thing appears, instead let it become apparent. Withhold information from the reader to build tension, hint and further lore and be creative with your world building) etc, maybe I'll come back and publish some of my prompts in full but I'm getting great results

civilitty4• 3 years ago

Finally! I’ve been getting the shakes waiting for next OpenAI release.

16k context with 3.5-turbo is huge. It’ll make all those dime a dozen document driven assistants a lot more useful.

I’m curious to see if people will figure out ways to hack functions to get more reliable structured JSON data out out of GPT without tons of examples, giving lots more context room to play with

SkyPuncher• 3 years ago

This is awesome. We were finding a lot of frustrating with 4k context being far too short to properly chunk documents.

In a worst case scenario, you have to assume that output is going to be the same length as input. That means useful context is actually half of the total context.

Add in a bit of fixed size for chunking/overlap (maybe ~500 tokens), suddenly you're looking at only 1k to 1.5k being reliably available for input. 16k context bumps that number up to 7.5k available for input. That's massive.

Dowwie• 3 years ago

Can you provide some examples of what document driven assistants you're referring to?

N3cr0ph4g1st• 3 years ago

I'm assuming something like humata.ai

jumploops2• 3 years ago

This seems like a direct result of Plugins not hitting PMF -- rather than give API developers access to Plugins, give them the underpinnings to build Plugin-like experiences.

Love it!

andybak2• 3 years ago

I've forgotten what pmf means from the last time someone used it and someone else explained it.

(Can we please all lighten up on the acronyms a touch?)

lukaslezevicius• 3 years ago

Product market fit.

iamflimflam1• 3 years ago

Being able to give the LLM an API surface to call against is really powerful.

personjerry• 3 years ago

The 16k context window for gpt3.5 is exciting but unfortunately I think many of us were hoping for a gpt4 price drop!

JieJie• 3 years ago

It's an effective price drop for a smarter GPT-4 model, though, isn't it? A smarter and more steerable model for the same price?

sashank_15092• 3 years ago

I can't wait for vision model access and the massive new opportunities that presents!

armatav• 3 years ago

Hm, still don’t see the 32k context in the Playground

naiv3• 3 years ago

I think they never put 8k into the playground?

Edit: I can call the api now with gpt-4-32k-0613

ilaksh7• 3 years ago

So they opened 32k up to everyone?

capybara_2020• 3 years ago

Check under Mode > Complete. I see it listed there but not under Mode > Chat.

Also try

curl -X GET https://api.openai.com/v1/models \ -H "Authorization: Bearer your-api-key"

It is listed there for me.

EvgeniyZh• 3 years ago

I've seen it on playground yesterday night (European time) and it disappeared by today

braindead_in• 3 years ago

With these updates, we’ll be inviting many more people from the waitlist to try GPT-4 over the coming weeks, with the intent to remove the waitlist entirely with this model. Thank you to everyone who has been patiently waiting, we are excited to see what you build with GPT-4!

What about the rate limits? The docs say that it's 200 RPM and "We are unable to accommodate requests for rate limit increases due to capacity constraints."

janalsncm• 3 years ago

I’d assume part of that $10B is going towards getting on the short list to buy GPUs, but there’s only so many to go around.

sergiotapia• 3 years ago

What are you thoughts here regarding functions:

I have some data I can pass in CSV format to the context and ask a question against that data. "Who are my best customers?" and pass in a CSV of the top 100 customers.

I create a function that returns my best customers and call a ChatGPT function.

When would I use one or the other? The function call seems like it would be more accurate with better guardrails, but it does require me to know what questions my users will make beforehand.

Maybe that's the point, use functions when you know what kind of questions your users will make.

ilaksh7• 3 years ago

Unless it's a small CSV then I would put it in sqlite or something and tell GPT-4 to write a query given the user's question. I have done that before and worked pretty well. Even worked fairly ok with gpt-3.5. You have to give it context like schema etc.

I was even able to get it to output custom-coded embedded Chart.js charts if requested by the user.

kgumba• 3 years ago

We're[0] building a tool that helps you do what you described. We mainly advertise our ability to do this over your data warehouse, like Snowflake, but we also use DuckDB to help people query CSVs.

3.5 has worked pretty well for us in most cases. It's also a good amount faster. GPT-4 seems to really stand out for our complex joins.

- 0 = https://www.definite.app

mesmertech2• 3 years ago

Seems like I was right to wait for price drop on GPT3.5 a few months ago, was hoping for mostly a drastic drop for GPT4 but I guess 25%(for input tokens, effectively 12.5% on average) on 3.5 works as well

Fiahil• 3 years ago

Despite sharing a common prefix, GPT3.5 and GPT4 are completely different models. So, if you were hoping for a price drop on GPT4, the 3.5 drop might not be of any use to you

mesmertech2• 3 years ago

I know, I meant it'd have been better if GPT4 got a slash but for my usecase 3.5 is still cost-efficient, basically would've been happy with either of them getting a price cut

MuffinFlavored3• 3 years ago

I don't understand it. You tell the gpt-4 session a function exists and supply the signature, but I don't see any URLs being defined on where the APIs live?

1. Call the model with functions and the user’s input
2. Use the model response to call your API

I don't get this part. Where do you tell it where the API lives?

kgwgk3• 3 years ago

You don't. You take the model response and you call your API.

MuffinFlavored3• 3 years ago

Ah, I missed this part. ChatGPT response after you tell it a function call exists, it can decide whether or not to call it.

    {
      "id": "chatcmpl-123",
      ...
      "choices": [{
        "index": 0,
        "message": {
          "role": "assistant",
          "content": null,
          "function_call": {
            "name": "get_current_weather",
            "arguments": "{ \"location\": \"Boston, MA\"}"
          }
        },
        "finish_reason": "function_call"
      }]
    }

Gotcha. This makes it so that instead of English, gpt-4 can basically spit out/decide when to make function calls now, got it. Thanks.

I wonder how scary this will get if people mistakenly trust it with anything more than a GET. Imagine not being able to trust the reliability of some of the parameters of POST / PUT / DELETE on some reports/internal databases, etc.

egonschiele6• 3 years ago

OpenAI continues to impress. Function calls will make working with JSON much easier, a current pain point. Dropping the price of embeddings and increasing context length means searching through your own content should become faster and more accurate.

$0.0015 per 1K input tokens and $0.002 per 1K output tokens, which equates to roughly 700 pages per dollar.

This is such an incredible steal, especially when you consider that no open source option comes close to GPT3.5.

freediver• 3 years ago

This is such an incredible steal, especially when you consider that no open source option comes close to GPT3.5.

Orca comes close or is better.

Good explainer here: https://www.youtube.com/watch?v=Dt_UNg7Mchg

ilaksh7• 3 years ago

But isn't Orca non-commercial? Or.. do people just ignore that part.

egonschiele6• 3 years ago

Is it accessible anywhere?

jablongo• 3 years ago

This seems great, but what does this mean for fine-tuning? Will we be able to fine-tune models prompted with function calls? Should we fine-tune models prompted with function calls? Depending on the complexity of the function-call/text-query pairs I think we may still want to...

egonschiele6• 3 years ago

I'm not sure about function calls but the lower price for embeddings and longer context lengths should help with fine tuning?

MallocVoidstar• 3 years ago

They don't support fine-tuning GPT-3.5 or GPT-4. https://platform.openai.com/docs/guides/fine-tuning/what-mod...

egonschiele6• 3 years ago

Anyone know how they pick who to invite off the waitlist for GPT 4? I've been on there for a while. My project is open source and I wonder if that is getting me deprioritized.

civilitty4• 3 years ago

I sacrificed an albino goat with red eyes at midnight while chanting “sam-a sam-a sam-a” in the ancient R'lyehian language.

paulgb• 3 years ago

You can get GPT 4 access by submitting an eval if gets merged (https://github.com/openai/evals). Here's the one that got me access[1]

Although from the blog post it looks like they're planning to open up to everyone soon, so that may happen before you get through the evals backlog.

1: https://github.com/openai/evals/pull/778

gwd• 3 years ago

I didn't say I had a project at all; I think I just said I wanted to learn about it. I got access w/in 3 weeks. Maybe I was lucky and it was random? Or maybe they figured I wouldn't add that much load?

hubraumhugo• 3 years ago

Nice! Real life use case of these updates for my autonomous web scraping product:

- Bigger context window means less slicing and less calls for generating the web scrapers on the fly

- The functions will help to reliably build our data transformation steps (e.g. mapping different sources into the same structure)

- Way better unit economics

wallawe2• 3 years ago

Can this handle multiple input URLs? For example, I have 100 local business home pages, and I want you to get the email and phone number if they exists for each. Here are the 100 URLs..

I would be a happy paying customer if so.

savy91• 3 years ago

Have you tried this? https://apify.com/vdrmota/contact-info-scraper it can work with 50 input urls at a time.

weinzierl2• 3 years ago

16k context sounds exciting. The day I can throw a whole book at it and ask it arbitrary questions about it will be great. With 16k we are getting into full article realm and that is already incredibly useful.

Is there any open model with a similar context length? [I'm not talking about the dubious for long context fine-tuned LLaMA variants, I mean the real thing.]

037• 3 years ago

Not open, but with Anthropic the limit is 100k now https://news.ycombinator.com/item?id=35904773

1010112• 3 years ago

Try looking into vector databases to solve that problem.

You can chunk up a book and embed those partitions into a vector database. Then you can take a query and fuzzy match the most relevant documents in your vector database, then feed it back to open AI to resolve an answer.

It's brilliant. Postgres has an extension to support indexing the vectors, and there are some other open source and turnkey solutions in the market as well.

weinzierl2• 3 years ago

Brilliant, but not the same. I think both approaches have their place and are not mutually exclusive.

zh217• 3 years ago

Some observations:

1. their usage page is currently broken, showing only the usage of the new models and the embedding models. usage for the deprecated models are not included now.

2. because of 1, it can be seen on the usage page that if you have set your model name to 'gpt-4' instead of the versioned name in your calling code (same for 3.5), you have already been using the new models for the past two days!

anotherpaulg4• 3 years ago

I've been spending a lot of time figuring out how to make these GPT models successful at *editing* existing code. This is much more difficult than having them write brand new code.

So far, gpt-4 has been significantly better at editing code than gpt-3.5-turbo. This is for two reasons (which I previously discussed here [1]):

1. GPT-4's bigger context window lets it understand and edit larger codebases. The new 16k window for 3.5 might solve this problem.

2. GPT-4 is much better at following instructions about how to format code edits into a diff-like output format. Perhaps the improved instruction following will help 3.5 succeed here.

Essentially, 3.5 isn't capable of outputting any sort of diff-based edit format. The only thing it can reliably do is send back *all* of the code (say an entire source file) with the changes included. On the other hand, GPT-4 is capable of reliably outputting (some) diff-like formats.

So I'm very curious to see if the new 3.5 model solves both of these problems. I'll be running some benchmarks to find out!

[1] https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35

EDIT: On point (2) above... early experiments with gpt-3.5-turbo-16k seem to indicate that it is NOT able to follow system prompt instructions to use a diff-like output format. Still need to try the new functions capability.

ilaksh7• 3 years ago

What I'm thinking is to give it a function called replaceAll that just replaces text, and another one called insertAfter. Maybe also replaceBetween.

anotherpaulg4• 3 years ago

Absolutely! I will certainly be experimenting with the new functions as you suggest.

One thing which I haven't seen discussed elsewhere is the tension between the output format and the underlying task. When I ask GPT to use a simple, natural output format it does *better* at the actual code editing task. If I ask it to output using a more technical format like `diff -c` or heavily structured json formats... it "gets distracted" and does worse at the underlying coding request.

It writes worse code if you ask it to output edits in a terse, machine-readable diff format. It writes better code if you let it show you the code edits in a simple way.

For GPT3.5 this means I have to let it just type the whole source file back to me, with the edits included. GPT-4 is able to output a very simple diff-like format, but struggles with `diff -c`, etc.

So it will be interesting to see how the new function capabilities affect this tradeoff. Perhaps it is now "fluent" in function-json and so won't be "distracted" by that output formatting.

LelouBil• 3 years ago

Speaking about editing, is there a name and/or an explanations for the behaviour that LLMs have sometimes of saying they did something when they don't ?

Like :

Me : Fix this code

ChatGPT : edits it but incorrectly

Me: no, do it like .....

ChatGPT: Okay, I edited it like ... Sends back the same code as last time

Is it linked to the fact that these LLMs never or very rarely refuse a prompts even if it's something they can't do ?

Kiro• 3 years ago

Have you tried GitHub Copilot Chat? You just select the code you want to edit in your IDE and tell it what you want (the chat is in an integrated window next to your code). It will see what code is selected and use the rest of your codebase to edit it. The code it responds with even has has a "Insert at Cursor", which will replace the selected code with the new.

anotherpaulg4• 3 years ago

Ya, I would like to checkout Copilot Chat. Last time I checked it was waitlisted though.

My open source tool aider allows you to just ask for a change, without pointing it at a specific chunk of your code. It can handle changes that require multiple edits across multiple files. Ideally you just run aider inside your git repo and start asking for changes. Sometimes it helps to tell it which files to pay attention to.

Here's some example chat session transcripts that give a sense of what it's like to code this way:

https://aider.chat/examples/complex-change.html

https://aider.chat/examples/add-test.html

busseio• 3 years ago

Thank you for sharing this. I’ve been working on something for editing exiting code by using GitHub Issues and Actions for prompting, with the response from GPT as a Pull Request/Comment on the Issue[1]. I will definitely try out your project!

[1] https://github.com/busse/kodumisto

anotherpaulg4• 3 years ago

Thanks for sharing your project. I've certainly been thinking along the same lines. I've used my tool aider to solve a couple of issues filed by users:

https://github.com/paul-gauthier/aider/issues/13#issuecommen...

https://github.com/paul-gauthier/aider/issues/5#issuecomment...

cbowal• 3 years ago

Has anyone seen speed differences with the new gpt-3.5-turbo-0613 model? I've been testing for the past hour and I'm getting responses in about a quarter of the time.

saliagato• 3 years ago

pretty much the same. slightly worse at following instructions

rattray• 3 years ago

How's the quality?

JoshMandel2• 3 years ago

Interesting to see this plugin-adjacent functionality landing in the chat API.

It seems like there is no way to provide in context examples of calling functions, since they are now no longer just "assistant"-authored chat turns with text, but rather a distinct kind of output.

This can make it hard to demonstrate how to use functions effectively. I haven't played with this feature yet; maybe the model will somehow be able to leverage in context chat turn examples and use those to inform its function call outputs?

naiv3• 3 years ago

https://platform.openai.com/docs/guides/gpt/function-calling

they also have a notebook with more examples:

https://github.com/openai/openai-cookbook/blob/main/examples...

JoshMandel2• 3 years ago

I have read both of these but I did not notice any in-context examples, meaning prompts fed to the model showing it how to call a function and response to a user query, rather than docstrings telling it how to call a function.

growthwtf• 3 years ago

Just updated my little experiments to use the new models. Can confirm that this works far better. The JSON has been valid 100% of the time, but it still is a little fuzzy on string vs number in some cases for ids. Great update overall.

alpark3• 3 years ago

I wonder how much of these changes are pushed by the local LLM shift we've seen recently. I would've expected them to totally focus on GPT-4 updates, but it's nice that we're getting 3.5 improvements.

It's pretty clear that there's a large demand for much cheaper, if weaker LLMs. I'll need to test the "more reliable steerability via the system message" feature, but GPT-3.5's largely monotonic tone and lack of response to the system message was one of its largest weaknesses imo. I'm all for ggml and LLaMa, but there's almost zero need for me to invest in hardware/expensive GPUs (or /hour options) if 3.5 is this cheap. Only downsides I can see are data privacy and OpenAI's "safety" restrictions.

Function calls seem amazing, too. No need to use tokens commanding GPT about its ability to do function calls. I need to test it out though.

tedsanders• 3 years ago

Describing functions to GPT still costs extra tokens, unfortunately.

TeMPOraL2• 3 years ago

Especially given they picked just about the most verbose way of doing it, second only to XML. While this is to be tested, given the examples they give, I somehow doubt that minified description with single-letter function names will perform as well as human-readable (verbose) schemas.

benjamoon• 3 years ago

Anyone know how long it will take Azure to get this latest model in its OpenAI service?

chrisrickard• 3 years ago

"Unfortunately, the new GPT-3.5-turbo-16k model is not yet available on Azure OpenAI. So, we can't share much information with you regarding this. We don't have any ETA at this moment. Once we have anything will share it with you."

https://learn.microsoft.com/en-us/answers/questions/1305032/...

orasis• 3 years ago

This is for GPT 3.5 as well.

nlh• 3 years ago

Good catch - updated title!

ramraj07• 3 years ago

With this capability also comes potential risks. We strongly recommend building in user confirmation flows before taking actions that impact the world on behalf of users (sending an email, posting something online, making a purchase, etc).

Yeah thanks for the heads up Sam.

dumpsterdiver2• 3 years ago

"When performing actions such as sending emails or posting comments on internet forums, the language learning model may mistakenly attempt to take control of powerful weaponry and engage with humans in physical warfare. For this reason we strongly suggest building in user confirmation prompts for such functionality."

dumpsterdiver2• 3 years ago

Bonus points if the LLM replies sinisterly with something like, "As a language learning model I am not legally liable for any property damage, loss of life, or sudden regime changes that may occur due to receiving user input. Are you sure that you would like to send this email?"

skybrian• 3 years ago

I don't know what you're expecting? It's good advice. Obvious, but maybe not to everyone? People are idiots.

m3kw9• 3 years ago

How does functions work? Is it basically calling plugins?

lumost• 3 years ago

It passes arguments back to the application to call the function.

This means that it’s the applications choice to actually call the function or not. It’s not OpenAIs problem if the json does something terrible.

civilitty4• 3 years ago

Altman's Law: Somewhere out there is a webhook connected through a rube goldberg series of systems to nuclear launch codes... and OpenAI will find it. One function call at a time.

luke_s• 3 years ago

Their docs suggest you could allow the model to extract structured data by giving it the ability to call a function like `sql_query(query: string)`, which is presumably connected to your DB.

This seems wildly dangerous. I wonder how hard it would be for a user to convince the GPT to run a query like `DROP TABLE ...`

I think a good mental security model might be - if you wouldn't expose your function as an unsecured endpoint on the web, then you probably shouldn't expose it to a LLM

kerng• 3 years ago

This makes me wonder why does OpenAI not build in a mitigation by default that requires a confirmation that they control? Why leave it up to the tool developers to mitigate, many of whom never heard of confused deputy attacks?

Seems like a missed opportunity to make things a little more secure.

v3ss0n• 3 years ago

I noticed amount of spam doubled as soon as i woke up - this explains me.

valyagolev• 3 years ago

It can also decently serve as a way to output structured data, which wasn't extremely hard before but had its failure modes. This is a much more typical and understandable use-case for many apps

tmaly2• 3 years ago

note: the model may generate invalid JSON or hallucinate parameters

This seems like it could be a really big problem for this feature.

roughly• 3 years ago

The Langchain package does something similar to this and has a feature to combat that kind of thing that basically feeds the invalid input back into the bot and says “you were asked for x but gave this, please correct it” - it’s shockingly effective.

Spivak• 3 years ago

Which is surprising since jsonformer exists and the same approach works with the text-completion API just fine.

tmaly2• 3 years ago

I was looking at the models. I noticed the function calling for 3.5 is only with the 4,096 token length. The larger 16k context length for 3.5 does not mention it despite having a similar naming of 0613 suffix.

https://platform.openai.com/docs/models/gpt-3-5

gpt-3.5-turbo-0613 Snapshot of gpt-3.5-turbo from June 13th 2023 with function calling data. Unlike gpt-3.5-turbo, this model will not receive updates, and will be deprecated 3 months after a new version is released. 4,096 tokens Up to Sep 2021

gpt-3.5-turbo-16k-0613 Snapshot of gpt-3.5-turbo-16k from June 13th 2023. Unlike gpt-3.5-turbo-16k, this model will not receive updates, and will be deprecated 3 months after a new version is released. 16,384 tokens Up to Sep 2021

taf2• 3 years ago

16k is tokens on 3.5 is amazing but they also complicated the pricing from a simple flat per 1k token to two separate fees for input and output … I think it’s better maybe but token based pricing is challenging to reason with and even more so to explain to customers

wordpad25• 3 years ago

simplify the pricing and offer it as your own service = profit

visarga• 3 years ago

They should work towards making repetitive prompts be free or much cheaper because they don't have to generate it token by token and can be cached.

amitport• 3 years ago

As a customer of course I would want that, but they really shouldn't from their perspective. The customer is getting the value it wants and is willing to pay for this is where a lot of the profit will come from.

avereveard3• 3 years ago

Too bad they haven't documented how to disperse functions in the system message ourself, that would have made langchain.js crazy powerful.

(I.e create a function that answer the user question in JavaScript, you can call llm(prompt, context) to process the data into natural language and search(query) to find data for the user) - and then you let langchainjs execute the output as apart of its loop

avereveard3• 3 years ago

https://i.imgur.com/Cie0IJY.png ahah it works! even on 'older' gpt! this is amazing. and crazy.

vorticalbox3• 3 years ago

I was playing with this today made a simple add function that just takes two arguments and adds them and asked it "what is 10+5" and it called my function as you would expect. Then I made my function multiple the numbers. Weirdly gpt called my function twice then simply ignored the functions data and returned the correct answer.

Pretty impressive.

gawi• 3 years ago

Not sure I like the "I know better" attitude of the LLM in this case. What other function response the LLM is likely to discard?

vorticalbox3• 3 years ago

I updated a slack bot to support calling dalle but none of the images work because it striped the query parameters from the URLs. Very annoying

famouswaffles• 3 years ago

This one isn't new at least.

https://vgel.me/posts/tools-not-needed/

melovy• 3 years ago

openai function call + langchain for SQL query with vector search:

https://docs.myscale.com/en/sample-applications/openai-funct... https://github.com/myscale/examples/blob/main/openai_functio...

Function calling and other API updates

Related Stories