Structured Outputs in the API

https://openai.com/index/introducing-structured-outputs-in-the-api/

by davidbarker • 2 years ago

764 269 2 years ago

gamegoblin4• 2 years ago

I'm glad they gave up on their "fine-tuning is all you need" approach to structured output. It's possible fine-tuning will work in the long term, but in the short-term, people are trying to build things, and fine-tuning wasn't cutting it.

Surprised it took them so long — llama.cpp got this feature 1.5 years ago (actually an even more general version of it that allows the user to provide any context free grammar, not just JSON schema)

chhabraamit• 2 years ago

How does llama.cpp’s grammar adherence work?

Does it keep validating the predicted tokens and backtrack when it’s not valid?

gamegoblin4• 2 years ago

It's essentially an Earley Parser[0]. It maintains a set of all possible currently valid parses, and zeroes out the probability of any token that isn't valid in at least 1 of the current potential parse trees.

There are contrived grammars you can give it that will make it use exponential memory, but in practice most real-world grammars aren't like this.

[0] https://en.wikipedia.org/wiki/Earley_parser

tcdent• 2 years ago

GPT is still a language model, so at some point it's still just tokens.

Is this just a schema validation layer on their end to avoid the round trip (and cost) of repeating the call?

gamegoblin4• 2 years ago

Language models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick.

The simplest algorithm for getting good quality output is to just always pick the highest probability token.

If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods.

All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.

BoorishBears2• 2 years ago

I was surprised it took so long until I reached this line:

The model can fail to follow the schema if the model chooses to refuse an unsafe request. If it chooses to refuse, the return message will have the refusal boolean set to true to indicate this.

I'm not sure how they implemented that, maybe they've figured out a way to give the grammar a token or set of tokens that are always valid mid generation and indicate the model would rather not continue generating.

Right now JSON generation is one of the most reliable ways to get around refusals, and they managed not to introduce that weakness into their model

Der_Einzige4• 2 years ago

For many things, fine-tuning as we know of it will NEVER fully solve it, there's no hope. Even fine-tuning a model to not use the letter "e" to an overwhelming degree doesn't entirely prevent it, only reduces its chances to increasingly small amounts. Shamesless self plug, and from before the ChatGPT era too! https://paperswithcode.com/paper/most-language-models-can-be...

leetharris• 2 years ago

At the bottom:

Acknowledgements Structured Outputs takes inspiration from excellent work from the open source community: namely, the outlines, jsonformer, instructor, guidance, and lark libraries.

It is cool to see them acknowledge this, but it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation.

warkdarrior• 2 years ago

it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation

Maybe those projects were used as-is by OpenAI, so there was nothing new to contribute.

reustle• 2 years ago

I think they may be alluding to sponsorships as well as code contributions.

i.e. https://github.com/sponsors/jxnl

spencerchubb4• 2 years ago

Is offering gpt4o for free through chatgpt not enough of a contribution? They didn't release source code, but they made a product free to use

mplewis• 2 years ago

No. If it were free you'd be able to use it as a programming API. It's not free and it's not unlimited - it's a time-limited marketing tool.

talldayo• 2 years ago

Free service != Open software

notarobot123• 2 years ago

This isn't generosity, it's a well known and much used strategy for market penetration. Free until-we-decide-otherwise is very much not the same as open source.

echelon• 2 years ago

That can actually make competition from open source harder. New upstarts that are open source can't compete with free service from OpenAI and can't make money to grow their development or offerings.

OpenAI wants to kill everything that isn't OpenAI.

sirspacey• 2 years ago

You don’t anyone will use it to contribute to open source projects?

Seems like an obvious net gain for the community.

g15jv2dp3• 2 years ago

If you're unhappy about people using your work without compensating you or contributing back, don't release your work as free software. You can't have your cake and eat it too...

jjcm• 2 years ago

Interesting tidbit at the very end that's worth noting for anyone using the API today:

By switching to the new gpt-4o-2024-08-06, developers save 50% on inputs ($2.50/1M input tokens) and 33% on outputs ($10.00/1M output tokens) compared to gpt-4o-2024-05-13.

scrollop4• 2 years ago

From what I've learned from OpenAI, the "latest" "cheaper" model will perform worse than the previous model on various tasks (esp reasoning).

ralusek• 2 years ago

I don't think it's been well enough acknowledged that all of the shortcuts LLMs have been taking with ways of attempting to compress/refine/index the attention mechanism seem to result in dumber models.

GPT 4 Turbo was more like GPT 3.9, and GPT 4o is more like GPT 3.7.

scrollop4• 2 years ago

Also, is it a coincidence that at cheaper (potentially faster?) model has been released (just) before they roll out the "new" voice mode (which boasts very low latency)?

codingwagie2• 2 years ago

Its usually a distilled smaller model

samstave3• 2 years ago

Am I the only one that wants to know 1,000% *WHY* such things?

Is it a natural function of how models evolve?

Is it engineered as such? Why? Marketing/money/resources/what?

WHO makes these decisions and why?

---

I have been building a thing with Claude 3.5 pro account and its *utter fn garbage* of an experience.

It lies, hallucinates, malevolently changes code that was already told was correct, removes features - explicitly ignore project files. Has no search, no line items, so much screen real-estate is consumed with useless empty space. It ignores states style guides. get CAUGHT forgetting about a premise we were actively working on them condescendingly apologies "oh you're correct - I should have been using XYZ knowledge"

It makes things FN harder to learn.

If I had any claude engineers sitting in the room watching what a POS service it is from a project continuity point...

Its evil. It actively f's up things.

One should have the ability to CHARGE the model token credit when it Fs up so bad.

NO FN SEARCH??? And when asked for line nums in it output - its in txt...

Seriously, I practically want not just a refund, I want claude to pay me for my time correcting its mistakes.

ChatGPT does the same thing. It forgets things committed to memory - refactors successful things back out of files. ETc....

Its been a really eye opening and frustrating experience and my squinty looks are aiming that its specifically intentional:

They dont want people using a $20/month AI plan to actually be able to do any meaningful work and build a product.

campers2• 2 years ago

Thats ok from the perspective of it making room for a more capable and expensive GPT5 model to compete with Opus 3.5 when that arrives this year. The significant price drops for a small loss in quality is a reasonable tradeoff. Then GTP4o becomes the mid tier and GTP4o-mini the low tier.

There was 100 days in between Claude 3.0 Opus and Claude 3.5 Sonnet being released which gave us similar capability at a 80% price reduction. When I was using Opus I was thinking this is nice, but the cost does add up. Having Sonnet 3.5 so soon after was a nice surprise.

One more round of 80% price cuts after that combined with building out the multi-step agentic workflows should provide some decent capabilities!

minimaxir2• 2 years ago

The new price is also now reflected on the pricing page: https://openai.com/api/pricing/

It's weird that's only a footnote when it's actually a major shift.

sjnair96• 2 years ago

I also looked up the same. I wonder why. They must have a subsequent announcement regarding this I'd expect.

ComputerGuru3• 2 years ago

If you use the undecorated gpt-4o do you automatically get the latest?

daguava• 2 years ago

The un-postfixed version will point to the older model for the next 3 weeks their docs say

tedsanders5• 2 years ago

We'll update gpt-4o in 3 weeks. (We've always updated it couple weeks after launch, so no one is immediately surprised by a new model drop.)

OutOfHere4• 2 years ago

For the record, you should never use that in an application. Always explicitly note the full versioned model name. This will prevent bad surprises because not every new version is an improvement; sometimes they get worse, especially at specific tasks.

voiper1• 2 years ago

We will give a 3-week notice before updating gpt-4o to point to the new snapshot gpt-4o-2024-08-06.

Source: https://platform.openai.com/docs/models/gpt-4o

nerdjon• 2 years ago

I have a bad feeling that this is just going to introduce more shovelware apps that try to shove AI use in without really understanding what they are going to get back.

Yay I can now ensure the json object will look how I want, but lets completely disregard any concern of wether or not the data returned is valuable.

I don't understand why we are already treating these systems as general purpose AI when they are not. (Ok I do understand it, but it is frustrating).

The example given of "look up all my orders in may of last year that were fulfilled but not delivered on time".

First I have found these models incredibly dumb when it comes to handling time. But even beyond that, if you really are going to do this. I really hope you double check the data before presenting the data you get back as true. Worse that is just double checking what it gives back to you is accurate, not checking that it isn't telling you about something.

Every time I try to experiment with supplying data and asking for data back, they fall flat on their face before we even get to the json being formatted properly. That was not the issue that needed to be solved yet when it still fundamentally messes up the data. Often just returning wrong information. Sometimes it will be right though, but that is the problem. It may luck out and be right enough times that you gain confidence in it and stop double checking what it is giving back to you.

I guarantee you someone is going to have a discussion about using this, feeding it data, and then storing the response in a database.

titzer• 2 years ago

It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solving a computational problem is one of the oldest exercises around. On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. It's weird reading the kind of self-congratulating blogpost about this, like OpenAI has just discovered flint knives. On the other hand, a computer system can process natural language with extremely ambiguous, open-ended problems, compute solutions to said problems, even correct its own mistakes--and then it can format the output correctly. And then on yet another hand, it only took about 10^25 floating point operations (yeah, just ten million trillion trillion, right!?) to get this outcome.

codingwagie2• 2 years ago

I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago

ben_w2• 2 years ago

Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there.

The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models.

* that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.

srcreigh2• 2 years ago

If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.

jes5199• 2 years ago

in _general_ sure, but if you restricted each token to conform to a Kleene-star grammar you should be able to guarantee that you get something that parses according to a context-free grammar

tommica• 2 years ago

For some reason it reminds me of my civilization runs - rush to certain high level tech and then after that discovery writing :D

thruway5162• 2 years ago

I dont understand your complaint at all. If you develop a new revolutionary technology called an automobile, developing steering, brakes, starter, mufflers for it is a pretty big deal even if reins, clamps, mufflers and keys are mundane and have existed for decades. Structured outputs are a pretty big step in making this magic actually usable by developers as opposed to generating impressive cat pictures or whatever has captured the public imagination.

Bjartr• 2 years ago

I don't think it was an complaint, just a observation.

jappgar2• 2 years ago

Structured outputs are hard... but they claimed to have solved this a year ago.

They were lying, of course, and meanwhile charged output tokens for malformed JSON.

ramraj07• 2 years ago

This is like saying “we shouldn’t be celebrating a computer that can talk, my parrot can do that!”

throwawaymaths3• 2 years ago

On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification.

I highly doubt it's the model that does this... It's very likely code injected into the token picker. You could put this into any model all the way down to gpt-2.

crowcroft2• 2 years ago

I wonder if you get 90% of the way with prompt engineering, and then the last 10% is just brute force, validate output, if it fails, rerun the prompt.

My assumption is if that's all this is they would have done it a long time ago though.

scarmig• 2 years ago

I have struggled writing valid YAML before (my tokenizer doesn't handle whitespace very well). And it probably takes me a quadrillion operations on the reals to get a minimal YAML file (I think your 10^25 fp ops is an overestimate--I think it's more like 10^18-10^19).

It's kind of like an inverse Moravec's paradox.

theturtle32• 2 years ago

Relatable!!

m3kw9• 2 years ago

It’s doing more, it is allowing user to input using natural language and the output is the json format the API that is defined

raincole• 2 years ago

I don't know, it doesn't sound wild at all to me. Human languages are very imprecise, vague and error-tolerant, which is the opposite of an output format like JSON. So the a model can't do these two things well at the same time is quite an intuitive conclusion.

The wild part is that a model trained with so much human language text can still outputs mostly compilable code.

berkes2• 2 years ago

I am so often surprised by "The AI Communities" software. Often unpleasantly surprised, often just eye-rolling.

When we first started using the OpenAI API's, the first thing I reached for was some way "to be certain that the response is properly formatted". There wasn't one. A common solution was (is?) "just run the query again, untill you can parse the JSON". Really? After decades of software engineering, we still go for the "have you tried turning it off and on again" on all levels.

Then I reached for common, popular tools: everyone's using them, they ought to be good, right? But many of these tools, from langchain's to dify to promptflow are a mess (Edit: to alter the tone: I'm honestly impressed by the breadth and depth of these tools. I'm just suprised about the stability - lack thereof, of them). Nearly all of them suffer from always-outdated-documentation. Several will break right after installing it, due to today's ecosystem updates that haven't been incorporated entirely yet. Understandably: they operate in an ecosystem that changes by the hour. But after decades of software engineering, I want stuff that's stable, predictable, documented. If that means I'm running LLM models from a year ago: fine. At least it'll work. Sure, this constant-state-of-brokeness is fine for a PoC, a demo, or some early stage startup. But terrible for something that I want to ensure to still work in 12 months, or 4 years even without a weekly afternoon of upgrade-all-dependencies-and-hope-it-works-the-update-my-business-logic-code-to-match-the-now-deprecated-apis.

KoolKat23• 2 years ago

Well the simple answer is don't use it then.

In the same way people revert to older stable releases. You're welcome to revert to writing boilerplate code yourself.

The reason people are excited and use it, is because they show promise, it already offers significant benefits even if it isn't "stable".

wewtyflakes2• 2 years ago

Why would someone want `strict` to be anything other than `true`?

davidkunz• 2 years ago

Maybe if you can't precisely model your structure with (OpenAI's subset of) JSON schema.

ComputerGuru3• 2 years ago

There are many reasons, though I am not sure which they had in mind. One thing is that LLMs in general tend to do better when they can be more verbose in their output and sort of “think aloud” to reach an answer. Insisting on strict output format would rob it of the benefits (because it doesn’t just not emit but completely skips those stages, or else you’d be paying for those elided output tokens).

wewtyflakes2• 2 years ago

But then why would someone specify that the response has to be in a given JSON schema (by presence of the schema itself), but then also not care if it is actually using that schema (by specifying `strict` as `false`)? That is the use-case I can't wrap my head around.

tedsanders5• 2 years ago

We didn't cover this in the announcement post, but there are a few reasons:

- The first request with each JSON schema will be slow, as we need to preprocess the JSON schema into a context-free grammar. If you don't want that latency hit (e.g., you're prototyping, or have a use case that uses variable one-off schemas), then you might prefer "strict": false

- You might have a schema that isn't covered by our subset of JSON schema. (To keep performance fast, we don't support some more complex/long-tail features.)

- In JSON mode and Structured Outputs, failures are rarer but more catastrophic. If the model gets too confused, it can get stuck in loops where it just prints technically valid output forever without ever closing the object. In these cases, you can end up waiting a minute for the request to hit the max_token limit, and you also have to pay for all those useless tokens. So if you have a really tricky schema, and you'd rather get frequent failures back quickly instead of infrequent failures back slowly, you might also want "strict": false

But in 99% of cases, you'll want "strict": true.

solaxun• 2 years ago

On the latency of the first request - How is the CFG cached?

Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?

_vaporwave_• 2 years ago

Anyone else catch this reference in one of the examples?

9.11 and 9.9 -- which is bigger

https://community.openai.com/t/why-9-11-is-larger-than-9-9-i...

jodacola• 2 years ago

Amusingly, I immediately thought 9.11 - but in the context of a newer version of software. Ever have those moments where you're so deep in context of some ecosystem that you skip right past the basics, like 9.9 being a larger number than 9.11?

elpocko• 2 years ago

Doesn't the BNF grammar approach in llama.cpp solve this issue in a generic way that should work with any model? Why wouldn't they use that?

ejones3• 2 years ago

Similar approach to llama.cpp under the hood - they convert the schema to a grammar. Llama.cpp's implementation was specific to the ggml stack, but what they've built sounds similar to Outlines, which they acknowledged.

HanClinto2• 2 years ago

llama.cpp's GBNF grammar is generic, and indeed works with any model.

I can't speak for other approaches, but -- while llama.cpp's implementation is nice in that it always generates valid grammars token-by-token (and doesn't require any backtracking), it is tough in that -- in the case of ambiguous grammars (where we're not always sure where we're at in the grammar until it finishes generating), then it keeps all valid parsing option stacks in memory at the same time. This is good for the no-backtracking case, but it adds a (sometimes significant) cost in terms of being rather "explosive" in the memory usage (especially if one uses a particularly large or poorly-formed grammar). Creating a grammar that is openly hostile and crashes the inference server is not difficult.

People have done a lot of work to try and address some of the more egregious cases, but the memory load can be significant.

One example of memory optimization: https://github.com/ggerganov/llama.cpp/pull/6616

I'm not entirely sure what other options there are for approaches to take, but I'd be curious to learn how other libraries (Outlines, jsonformer) handle syntax validation.

behnamoh2• 2 years ago

Well, there goes one of the big advantages of open-source models...

For a long time, I was relying on such guaranteed structured outputs as a "secret sauce" that only works using llama.cpp's GBNF grammars. Now OpenAI literally introduced the same concept but a bit more accessible (since you create a JSON and they convert it to a grammar).

Those of you who have used GBNF, do you think it still has any advantage over what OpenAI just announced?

ejones3• 2 years ago

FWIW, llama.cpp has always had a JSON schema -> GBNF converter, although it launched as a companion script. Now I think it's more integrated in the CLI and server.

But yeah I mean, GBNF or other structured output solutions would of course allow you to supply formats other than JSON schema. It sounds conceivable though that OpenAI could expose the grammars directly in the future, though.

behnamoh2• 2 years ago

I think for certain tasks it's still easier to write the grammar directly. Does converting from JSON to a CFG limit the capabilities of the grammar? i.e., are there things JSON can't represent that a context free grammar can?

J_Shelby_J• 2 years ago

JSON is a sub-set of what GBNF can do, so there are still advantages to that approach. But even GBNF doesn’t go far enough. Ever try to restrict a model to a single sentence?

root ::= \" \" item{{{min_count},{max_count}}}

item ::= [A-Z] [^\\r\\n\\x0b\\x0c\\x85\\u2028\\u2029.?!]+ [a-z] (\". \" | \"? \" | \"! \")

This kinda works if you don't mind no abbreviations, but you can't do something like this with JSON grammars afaik.

enobrev• 2 years ago

In a startup I was working on last year, I had a surprisingly good experience with using a json-schema in my prompt. I had to tweak the json response a bit because it was always invalid, but the issue was generally a missing colon or misplaced bracket. Data-wise it stuck to the schema very well, and cleaning up the json was simple enough that we got to zero parsing errors. I believe this was with 3.5.

Sadly, that project was a final (relatively successful) attempt at getting traction before the startup was sold and is no longer live.

Edit: Ouch, are the down-votes disbelief? Annoyance? Not sure what the problem is.

nichochar• 2 years ago

I'm a little confused why you have to specify "strict: true" to get this behavior. It is obviously always desired, I would be surprised for people to ever specify "strict: false". That API design leaves to be desired.

I also learned about constrainted decoding[1], that they give a brief explanation about. This is a really clever technique! It will increase reliability as well as reduce latency (less tokens to pick from) once the initial artifacts are loaded.

[1] https://www.aidancooper.co.uk/constrained-decoding/

dgellow• 2 years ago

Could you develop a bit re: the API? What do you dislike other than the “strict: true”?

athyuttamre• 2 years ago

Hi, I work on the OpenAI API — structured outputs schemas have limitations (e.g. all fields must be required, no additional properties allowed): https://platform.openai.com/docs/guides/structured-outputs/s....

If your schema is not supported, but you still want to use the model to generate output, you would use `strict: false`. Unfortunately we cannot make `strict: true` the default because it would break existing users. We hope to make it the default in a future API version.

Der_Einzige4• 2 years ago

You should also mention that before you had done custom alignment accounting for this feature, that it was an excellent alignment breaker (therefor a big no-no to release too early)

For example, if I ask an LLM to generate social security numbers, it will give the whole "I'm sorry Hal, I can't do that". If I ban all tokens except numbers and hyphens, prior to your "refusal = True" approach, it was guaranteed that even "aligned" models would generate what appeared to be social security numbers.

thrance2• 2 years ago

Hi, if you're allowed to answer, are there any future plans to support custom CFGs through the API ? Like llama.cpp does with it's GBNF format.

srcreigh2• 2 years ago

The tokens that are valid at the beginning of the output include things like {, {“, {\n, etc. However, once the model has already sampled {“val, then { is no longer a valid token

Oops, this is incorrect. {“val{“:2} is valid json.

(modulo iOS quotes lol)

jhgg• 2 years ago

Valid JSON, sure, but that key does not conform to the schema provided in the example. The LLM must generate valid JSON that also conforms to the provided schema.

jeffhuys• 2 years ago

FYI you can just hold down the “ button and get a ".

simonw4• 2 years ago

The price decrease is particularly notable because it represents a 50% cut in the price to handle image inputs, across any OpenAI model.

Previously image inputs on GPT-4o-mini were priced the SAME as GPT-4o, so using mini wouldn't actually save you any money on image analysis.

This new gpt-4o-2024-08-06 model is 50% cheaper than both GPT-4o AND GPT-4o-mini for image inputs, as far as I can tell.

UPDATE: I may be wrong about this. The pricing calculator for image inputs on https://openai.com/api/pricing/ doesn't indicate any change in price for the new model.

minimaxir2• 2 years ago

The calculator doesn't account for the fact that there are now two different prices in a given price matrix.

jeffharris• 2 years ago

yep image input on the new model is also 50% cheaper

and apologies for the outdated pricing calculator ... we'll be updating it later today

brianjking• 2 years ago

So we can send an image + text to the new structured output model and use the chain of thought schema?

I'm getting an error.

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid content type. image_url is only supported by certain models.', 'type': 'invalid_request_error', 'param': 'messages.[1].content.[1].type', 'code': None}}

cvhc• 2 years ago

I wonder why the top level has to be an object instead of an array... I have some pretty normal use cases where I expect the model to extract a list of objects from the text.

``` openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid schema for response_format \'PolicyStatements\': schema must be a JSON Schema of \'type: "object"\', got \'type: "array"\'.', 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}} ```

I know I can always put the array into a single-key object but it's just so annoying I also have to modify the prompts accordingly to accomodate this.

manquer• 2 years ago

I can't say for OpenAI, but in general I have seen and used this design pattern to keep consistency of root object output and remove a lot of unnecessary validations and branching flows

Otherwise you will to handle the scenarios in code everywhere if you don't know if the root is object or array. If the root has a key that confirms to a known schema then validation becomes easier to write for that scenario,

Similar reasons to why so many APIs wrap with a key like 'data', 'value' or 'error' all responses or in RESTful HTTP endpoints collection say GET /v1/my-object endpoints do no mix with resource URIs GET /v1/my-object/1 the former is always an array the latter is always an object.

tomComb• 2 years ago

Well, this wouldn’t be a very satisfying explanation, but these JSON objects are often represented as Python dictionaries and those can’t have top level arrays.

Too• 2 years ago

Try json.loads("[1,2,3]"), you'll get a list back.

The reasons others already posted about extensibility are more correct.

heliophobicdude• 2 years ago

Back in the old days, top level arrays were a security risk because the array constructor in JS could be redefined and do bad-guy stuff. I cannot think of any json parsing clients that are vulnerable to this.

moritzwarhier• 2 years ago

It's a relatively common convention for JSON APIs.

Possible reasons:

- Extensibility without breaking changes

- Forcing an object simplifies parsing of API responses, ideally the key should describe the contents, like additional metadata. It also simplifies validation, if considered separate from parsing

- Forcing the root of the API response to be an object makes sure that there is a single entry point into consuming it. There is no way to place non-descript heterogenous data items next to each other

- Imagine that you want to declare types (often generated from JSON schemas) for your API responses. That means you should refrain from placing different types, or a single too broad type in an array. Arrays should be used in a similar way to stricter languages, and not contain unexpected types. A top-level array invites dumping unspecified data to the client that is expensive and hard to process

- The blurry line between arrays and objects in JS does not cleanly map to other languages, not even very dynamic ones like PHP or Python. I'm aware that JSON and JS object literals are not the same. But even the JSON subset of JS (apart from number types, where it's not a subset AFAIK) already creates interesting edge cases for serialization and deserialization

simonw4• 2 years ago

I've regretted designing APIs that return an array rather than an object in the past.

It's all about the extensibility. If you return an object you can add extra keys, for things like "an error occurred, here are the details", or "this is truncated, here's how to paginate it", or a logs key for extra debug messages, or information about the currently authenticated user.

None of those are possible if the root is an array.

__jl__2• 2 years ago

There is another big change in gpt-4o-2024-08-06: It supports 16k output tokens compared to 4k before. I think it was only available in beta before. So gpt-4o-2024-08-06 actually brings three changes. Pretty significant for API users

1. Reliable structured outputs 2. Reduced costs by 50% for input, 33% for output 3. Up to 16k output tokens compared to 4k

https://platform.openai.com/docs/models/gpt-4o

Culonavirus• 2 years ago

That's actually pretty impressive... if they didn't dumb it down that is, which only time will tell.

santiagobasulto• 2 years ago

I’ve noticed that lately GPT has gotten more and more verbose. I’m wondering if it’s a subtle way to “raise prices”, as the average response is going to incur I more tokens, which makes any API conversation to keep growing in tokens of course (each IN message concatenates the previous OUT messages).

sashank_15092• 2 years ago

they also spend more to generate more tokens. The more obvious reason is it seems like people rate responses better the longer they are. Lmsys demonstrated that GPT tops the leaderboard because it tends to give much longer and more detailed answers, and it seems like OpenAI is optimizing or trying to maximize lmsys.

throwaway48540• 2 years ago

It's a subtle way to make it smarter. Making it write out the "thinking process" and decisions has always helped with reliability and quality.

tedsanders5• 2 years ago

GPT has indeed been getting more verbose, but revenue has zero bearing on that decision. There's always a tradeoff here, and we do our imperfect best to pick a default that makes the most people happy.

I suspect the reason why most big LLMs have ended up in a pretty verbose spot is that it's easier for users to scroll & skim than to ask follow-up questions (which requires formulation + typing + waiting for a response).

With regard to this new gpt-4o model: you'll find it actually bucks the recent trend and is less verbose than its predecessor.

sophiabits• 2 years ago

I’ve especially noticed this with gpt-4o-mini [1], and it’s a big problem. My particular use case involves keeping a running summary of a conversation between a user and the LLM, and 4o-mini has a really bad tendency of inventing details in order to hit the desired summary word limit. I didn’t see this with 4o or earlier models

Fwiw my subjective experience has been that non-technical stakeholders tend to be more impressed with / agreeable to longer AI outputs, regardless of underlying quality. I have lost count of the number of times I’ve been asked to make outputs longer. Maybe this is just OpenAI responding to what users want?

[1] https://sophiabits.com/blog/new-llms-arent-always-better#exa...

bilater• 2 years ago

I have not been able to get it to output anywhere close to the max though (even setting max tokens high). Are there any hacks to use to coax the model to produce longer outputs?

surfingdino• 2 years ago

Is it still NTSAT (Never The Same Answer Twice)?

H8crilA4• 2 years ago

Yes, this happens by design.

oblio4• 2 years ago

Interesting, why? Is there no theoretical way to have stable models? Or some kind of executive decision?

H8crilA4• 2 years ago

How is this different from function calling?

binarymax• 2 years ago

Function calling uses JSON mode. While it has been mostly correct, I do get an incorrectly formatted response sometimes (maybe 1 in 10k requests?). So it sounds like this fixes that bug.

tedsanders5• 2 years ago

Under the hood, it's quite similar to function calling. A few differences:

- Structured Outputs is a bit more straightforward. e.g., you don't have to pretend you're writing a function where the second arg could be a two-page report to the user, and then pretend the "function" was called successfully by returning {"success": true}

- Having two interfaces lets us teach the model different default behaviors and styles, depending on which you use

- Another difference is that our current implementation of function calling can return both a text reply plus a function call (e.g., "Let me look up that flight for you"), whereas Structured Outputs will only return the JSON

(I worked on this feature at OpenAI.)

technics256• 2 years ago

How can we enable the text reply with a function call? Usually the message returned is a tool call only when it calls a tool?

nsonha2• 2 years ago

I guess practically this will be the equivalent to function calling with a single tool. The AI splits out some payload that you presumably feed into whatever you do next. There are only 2 possible outcomes: can/cannot produce payload, which is equivalent to call/not call the next functionality.

With function/tool calling, you give it a list of tools with payload and the AI will select between them, or refuse, so there are more than 2 possible outcomes.

(I've never used any of these APIs, get that from just reading the docs)

paradite3• 2 years ago

Really important update that was not mentioned:

gpt-4o-2024-08-06 has 16,384 tokens output limit instead of 4,096 tokens.

https://platform.openai.com/docs/models/gpt-4o

We don't need the GPT-4o Long Output anymore.

OutOfHere4• 2 years ago

But is this also the default or just the max? Is the default 4k or 16k?

Also, the question of the default value applies both at the server level and at the SDK level.

paradite3• 2 years ago

Unlike Anthropic, OpenAI models don't have a `max_tokens` setting for API calls, so I assume the max token output limit is automatically applied to API calls.

Otherwise the max token output limit stated on the models page would be meaningless.

floam• 2 years ago

Long Output is 64K though.

gdiamos3• 2 years ago

We've had this for over 1 year in Lamini - https://lamini-ai.github.io/inference/json_output/

Works with any open LLM, including Llama 3.1

radarsat1• 2 years ago

Looks useful!

AStrangeMorrow2• 2 years ago

Also the outlines library: https://github.com/outlines-dev/outlines

gdiamos3• 2 years ago

Yeah! - outlines, guidance, jsonformer were inspiring for this line of work

HanClinto2• 2 years ago

Also note llama.cpp with grammar support:

https://github.com/ggerganov/llama.cpp/tree/master/grammars

Supports an EBNF-like syntax, as well as JSON-Schema.

msoad• 2 years ago

Why not JSON Schema?

gdiamos3• 2 years ago

We did some user studies and found that people found it less intuitive.

zoogeny2• 2 years ago

Totally tangential, totally not related to the post (unless you squint your eyes and really blur things) ...

I was thinking about the old canard of the sufficiently smart compiler. It made me think about LLM output and how in some way the output of a LLM could be bytecode as much as it could be the English language. You have a tokenized input and the translated output. You have a massive and easily generatable training set. I wonder if, one day, our compilers will be LLMs?

jcims• 2 years ago

You definitely could, not far removed from text to image or text to audio generators.

pjc50• 2 years ago

Why would you tolerate a nonreliable compiler with no assured relationship between its inputs and its outputs? Have people just got too comfortable with the C++ model of "UB means I can insert a security bug for you"?

bigyikes• 2 years ago

In a hypothetical future where the reliability of LLMs improves, I can imagine the model being able to craft optimizations that a traditional compiler cannot.

Like there are already cases where hand-rolling assembly can eke out performance gains, but few do that because it’s so arduous. If the LLM could do it reliably it’d be a huge win.

It’s a big if, but not outside the realm of possibility.

killthebuddha• 2 years ago

A function that implements natural language -> bytecode is IMO way more likely to be under the hood an LLM operating a compiler (or maybe a compiler operating LLMs) rather than a "bare" LLM. From an end user's perspective maybe it won't matter but I think it's an important technical point. IMO there's no evidence that an LLM will ever be the best way to execute general purpose computations.

thih9• 2 years ago

I guess an actual compiler would be cheaper and more reliable.

In theory we could do the same with mathematical computations, 2+2=4 and the like; but computing the result seems easier.

shepherdjerred• 2 years ago

Compilers require strict semantics and deterministic output. It’s the exact opposite of AI.

I could see AI being used (in a deterministic way) to make decisions about what optimizations to apply, to improve error messages, or make languages easier to use/reason about, but not for the frontend/backend/optimizations themselves.

adagradschool• 2 years ago

While text and image generation are getting cheaper at a significant rate, audio still seems to be just as expensive with ElevenLabs. I wonder why it is so.

say_it_as_it_is• 2 years ago

This puts like a dozen popular python libraries out of business

AStrangeMorrow2• 2 years ago

At least depends on the approach and use: stuff like outlines (https://github.com/outlines-dev/outlines) that actually changes the sampling to adhere to a grammar and can be used with local/custom models shouldn't be too impacted. Those are not really used on top of openAI models

fastball• 2 years ago

Once it actually works, sure.

I just tried to use structured outputs with the latest release (openai-python 1.40) and it doesn't think Structured Outputs is a thing.

EDIT: turns out my JSON schema is too large (800 lines + recursive) and seems to be breaking OpenAI's "convert to CFG" step. Whoops.

sansseriff• 2 years ago

Preprocessing new schema takes 'under 10 seconds'. That's... a huge range? Unless the preprocessing time is a small fraction of the inference time, I don't see the point.

I'm working on an app that dynamically generates schema based on user input (a union of arbitrary types pulled from a library). The resulting schema is often in the 800 token range. Curious how long that would take to preprocess

pton_xd• 2 years ago

Isn't "we hardcoded JSON into the latest model" kind of the opposite direction, strategically, from "we're on the way to AGI and I need 7 trillion to get there?"

isoprophlex• 2 years ago

You are witnessing the final stages in the evolution of OpenAI from a messianic hype machine to Yet Another Product Company.

Hence all the people leaving, too.

gardenhedge• 2 years ago

I am ootl, employees are leaving openai?

KaiMagnus• 2 years ago

Yeah, definitely a way to end up with a Siri like mess if you do this long enough. The use case is there and it’s going to be very useful, but the magic is wearing off.

chamomeal• 2 years ago

I mean it’s not groundbreaking, but it makes it much easier to make simple AI tools that aren’t chat-based. It definitely has me interested.

GPT-4 has been so mind-blowingly cool, but most of the interesting applications I can think of involve 10 steps of “ok now make sure GPT has actually formatted the question as a list of strings… ok now make sure GPT hasn’t responded with a refusal to answer the question…”

Idk what the deal is with their weird hype persona thing, but I’m stoked about this release

nsonha2• 2 years ago

AGI is useless if you can't figure out how to employ it as part of a system, instead of just chit chat

irgolic• 2 years ago

Wasn’t this already fully supported in the tool calling API?

ramoz• 2 years ago

Can someone explain how this is different/better than the current state of function calling (which I’ve been using to get a consistent json schema response without issue)?

mrshu• 2 years ago

This post (from an OpenAI researcher) contains a bit more background: https://news.ycombinator.com/item?id=41174213

jacobsimon• 2 years ago

For starters, the naming is much less confusing. But the behavior also appears to be enforced/validated at some layer (hopefully?), which function calling did not seem to be. I was experimenting with it a couple weeks ago and it would work like 75% of the time but would often give me invalid results for schemas with relatively simple nested objects.

zbyforgotp• 2 years ago

This is guaranteed, function calling without it is not. The old way can work for you, but my experience is different, especially with complex schemas.

Der_Einzige4• 2 years ago

Now the question is when they will support soft constraints like this: https://huggingface.co/blog/constrained-beam-search

agtech_andy• 2 years ago

I have had a lot of success using BoundaryML (https://www.boundaryml.com/) for this. They have also been super responsive for any of my questions.

aaronvg• 2 years ago

thanks for the shoutout, we benchmarked our approach against other function-calling techniques and we've been able to beat all other approaches every time (even by 8%!) just by getting better at parsing the data and representing schemas with less tokens using type definitions instead of json schema.

You can take a look at our BFCL results on that site or the github: https://github.com/BoundaryML/baml

We'll be publishing our comparison against OpenAI structured outputs in the next 2 days, and a deeper dive into our results, but we aim to include this kind of constrained generation as a capability in the BAML DSL anyway longterm!

tarofchaos• 2 years ago

Two years too late. I think we are going through a bozo period at OpenAI where small things are being highlighted as achievements.

damsta• 2 years ago

Can we get something like that in Gemini 1.5 Flash?

mugivarra69• 2 years ago

cohere had this like a while ago

MattDaEskimo• 2 years ago

"We have ripped code from a bunch of open-source variations and slapped it behind our brutally abstracted API.

Interoperable with other external models like the open source versions? What, are you mad?"

jumploops9• 2 years ago

By using JSON mode, GPT-4{o} has been able to do this reliably for months (100k+ calls).

We use GPT-4o to build dynamic UI+code[0], and almost all of our calls are using JSON mode. Previously it mostly worked, but we had to do some massaging on our end (backtick removal, etc.).

With that said, this will be great for GPT-4o-mini, as it often struggles/forgets to format things as we ask.

Note: we haven't had the same success rate with function calling compared to pure JSON mode, as the function calling seems to add a level of indirection that can reduce the quality of the LLMs output YMMV.

Anyhow, excited for this!

[0]https://magicloops.dev

qwertox• 2 years ago

What a cool product! I was about to recommend you to submit it as a "Show HN", but it turns out that it already got submitted one year ago.

Would you mind sharing a bit on how things have evolved?

jumploops9• 2 years ago

Thanks and great question :)

When we first launched, the tool was very manual; you had to generate each step via the UI. We then added a "Loop Creator agent" that now builds Loops for you without intervention. Over the past few months we've mostly been fixing feature gaps and improving the Loop Creator.

Based on recent user feedback, we've put a few things in motion:

- Form generator (for manual loops)

- Chrome extension (for local automations)

- In-house Google Sheets integration

- Custom outputs (charts, tables, etc.)

- Custom Blocks (shareable with other users)

With these improvements, you'll be able to create "single page apps" like this one I made for my wife's annual mango tasting party[0].

In addition to those features, we're also launching a new section for Loop templates + educational content/how-tos, in an effort to help people get started.

To be super candid, the Loop Creator has been a pain. We started at an 8% success rate and we're only just now at 25%. Theoretically we should be able to hit 80%+ based on existing loop requests, but we're running into limits with the current state of LLMs.

[0]https://mangota.ngo

tomcam2• 2 years ago

Very interesting. Did you build magicloops using this tech?

jumploops9• 2 years ago

We first built Magic Loops with GPT-4, about a year ago, well before JSON mode was a thing.

We had to a do a bunch of extra prompting to make it work, as GPT would often include backticks or broken JSON (most commonly extra commas). At the time, YAML was a much better approach.

Thankfully we've been able to remove most of these hacks, but we still use a best effort JSON parser[0] to help stream partial UI back to the client.

[0]https://www.npmjs.com/package/best-effort-json-parser

diego_sandoval• 2 years ago

Can I use Magic Loops to generate Magic Loops for me?

jumploops9• 2 years ago

Technically yes, but it would require reverse-engineering some of our APIs.

Practically speaking, we have quite a few use-cases where users call Loops from other Loops, so we're investigating a first-class API to generate Loops in one go.

Similar to regular software engineering, what you put in is what you get out, so we've been hesitant to launch this with the current state of LLMs/the Loop Creator as it will fail more often than not.

turnsout• 2 years ago

Had the same experience with function calling—we get much better results simply asking for JSON. With simple schemas (basically dictionaries), gpt-4 and 4o are basically bulletproof.

geepytee• 2 years ago

This model appears to be full of surprises.

The 50% drop in price for inputs and 33% for outputs vs. the previous 4o model is huge.

It also appears to be topping various benchmarks, ZeroEval's Leaderboard on hugging face [0] actually shows that it beats even Claude 3.5 Sonnet on CRUX [1] which is a code reasoning benchmark.

Shameless plug, I'm the co-founder of Double.bot (YC W23). After seeing the leaderboard above we actually added it to our copilot for anyone to try for free [2]. We try to add all new models the same day they are released

[0]https://huggingface.co/spaces/allenai/ZeroEval

[1]https://crux-eval.github.io/

[2]https://double.bot/

usaar333• 2 years ago

ZeroEval's Leaderboard on hugging face [0] actually shows that it beats even Claude 3.5 Sonnet on CRUX [1] which is a code reasoning benchmark.

The previous version of 4o also beat 3.5 Sonnet on Crux.

__jl__2• 2 years ago

Thanks for sharing your experience.

We also get pretty reliable JSON output (on a smaller scale though) even without JSON mode. We usually don't use JSON mode because we often include a chain of thought part in <brainstorming> and then ask for JSON in <json> tags. With some prompt engineering, we get over 98% valid JSON in complex prompts (with long context and modestly complex JSON format). We catch the rest with json5.loads, which is only used as a fallback if json.loads fails.

4o-mini has been less reliable for us particularly with large context. The new structured output might make it possible to use mini in more situations.

borsch• 2 years ago

my recommendation? use chain of thought, then feed that into a second prompt asking for json

JimDabell• 2 years ago

The linked article includes a section on this, under “Separating a final answer from supporting reasoning or additional commentary”. They suggest defining a JSON schema with a reasoning field and an answer field.

msp26• 2 years ago

Is the JSON actually being fed into the LLM's context or is it still being converted into typescript?

The previous setup didn't allow for custom types, only objects/string/num/bool.

Are the enums put into context or purely used for constrained sampling?

OutOfHere4• 2 years ago

Using this feature will obviously "lock you in" to OpenAI, specifically to this model too, at least until other companies catch on. While text prompts can more easily be moved to other LLMs, this feature cannot currently be ported as such. I would use it only if a text prompt is insufficient despite retries.

BoorishBears2• 2 years ago

It's already supported by multiple other providers. Fireworks, Together, probably more.

dtquad• 2 years ago

OpenAI-style JSON mode and function calling rapidly became the industry standard way of doing it. It will probably also happen for this feature.

toomuchtodo• 2 years ago

“S3 compatible”

PufPufPuf• 2 years ago

This feature has existed for quite some time in several inference libraries, like Outlines, under the names "constrained decoding" or "guided decoding". Some even include it in their OpenAI-compatible API in a very similar form (allowing to pass in a JSON Schema). All this required doing your own inference, though -- so the announcement really just brings this popular feature "to the masses".

moralestapia• 2 years ago

OTOH, not using it could "lock you out" of building a cool product for your users, so ...

OutOfHere4• 2 years ago

It depends. Quite often the intended schema is simple enough that parsing of a plaintext lines using a regex is sufficient. I do however have to use a verbose prompt to get it to comply with the expected format. Combine this with retries in case of rare failures, and the job is done.

I think this new feature is more relevant for intricate schemas and dynamic schemas when a text prompt cannot do the job.

faizshah• 2 years ago

The converse API in AWS bedrock lets you use function calling across a number of different providers (doesn’t support OpenAI): https://docs.aws.amazon.com/bedrock/latest/userguide/convers...

I have been using it so that my agents aren’t specific to a particular model or api.

Like others have said many other providers already have function calling and json schema for structure outputs.

jappgar2• 2 years ago

It's nice that they're not making me pay for broken json anymore but touting this as a "feature" is laughable.

It's a bug fix. They should never have been charging for malformed responses in the first place!

LAC-Tech• 2 years ago

Good to see JSON Schema being more widely adopted. I remember doing a project a few years ago in XML just because XML schemas were everywhere and JSON ones were still not really used.

Brosper• 2 years ago

I think they would like to have something like artifacts in Claude

sub7• 2 years ago

To "look up all my orders in May that were fulfilled and not delivered" is a 1 line query in every ORM I know of.

Why in the world would I literally pass around a schema file with each request that is (according to the example) 91 lines of code and pay someone for the privilege of maybe doing it correctly?

roseway4• 2 years ago

We extensively use vLLM's support for Outlines Structured Output with small language models (llama3 8B, for example) in Zep[0][1]. OpenAI's Structured Output is a great improvement on JSON mode, but it is rather primitive compared to vLLM and Outlines.

# Very Limited Field Typing

OpenAI offers a very limited set of types[2] (String, Number, Boolean, Object, Array, Enum, anyOf) without the ability to define patterns and max/min lengths. Outlines supports defining arbitrary RegEx patterns, making extracting currencies, phone numbers, zip codes, comma-separated lists, and more a trivial exercise.

# High Schema Setup Cost / Latency

vLLM and Outlines offer near-zero cost schema setup: RegEx finite state machine construction is extremely cheap on the first inference call. While OpenAI's context-free grammar generation has a significant latency penalty of "under ten seconds to a minute". This may not impact "warmed-up" inference but could present issues if schemas are more dynamic in nature.

Right now, this feels like a good first step, focusing on ensuring the right fields are present in schema-ed output. However, it doesn't yet offer the functionality to ensure the format of field contents beyond a primitive set of types. It will be interesting to watch where OpenAI takes this.

[0] https://help.getzep.com/structured-data-extraction

[1] https://help.getzep.com/dialog-classification

[2] https://platform.openai.com/docs/guides/structured-outputs/s...

bamboozled• 2 years ago

This is a game changer

tuan3w• 2 years ago

Just curious if structured outputs/constrained generation improve model accuracy, e.g., for information extraction problems. Does anyone have experience with this and why?

thrance2• 2 years ago

And you still can't provide a custom grammar to the API...

The company I work for desperately need the LLM to consistently generate results in a subset of HTML. I was able to craft a small grammar file that does just that, in under 5 minutes, and use it successfully with llama.cpp. Yet, there are still no API offering this basic feature that could really benefit everyone.

Instead we have a thousand garbage medium articles with "tips & tricks" on how to prompt the AI to get better results. It's as if people don't care anymore about consistency and reliability.

blackcat201• 2 years ago

Do beware on some reasoning task, our recent work[0] actually found it may cause some performance degradation as well as possible reasoning weakening in JSON. I really hope they fix this in the latest GPT-4o version.

[0] https://arxiv.org/abs/2408.02442

kiratp• 2 years ago

Thank you! This confirms my intuition!

Structured generation seems counter to every other signal we have that chain of thought etc improves performance.

myprotegeai• 2 years ago

I send dynamic jsonschemas to LLMs in manifest[1] to simulate function calls based on function signatures. These structured outputs will be enormously useful.

Other posts claim that you can generate jsonschema-conformant output reliably without this, and while I mostly agree, there is an edge case where gpt4o struggles, and that is simple data types. For example, a string, in jsonschema, has a schema of simply {"type": "string"}, and an example value of "hello world." However, gpt4o would produce something like {"value": "hello world"} at a very high probability. I had to include specific few shot examples of what not to do in order to make this simple case reliable. I suspect there are other non-obvious cases.

1. https://github.com/amoffat/manifest

franze• 2 years ago

finally

gpt.franzai.com

we use a 3 times check until now

if chatgpt does not return valid json then trim anything before and after last {}

if this is not valid json feedback him full response and ask for valid json

if this is not valid json put full response into self made json to get al least something we can work with in an "something did not work out" response

tomduncalf• 2 years ago

Would constraining the tokens that the model can choose from impact its “intelligence”/“creativity” in other (possibly negativel) ways?

Havoc• 2 years ago

Surprised to see OpenAI staff in comments. I don’t recall that being the case on previous threads. (Hello!)

armcat• 2 years ago

This is awesome, and simplifies lot of my workflows when using their APIs directly. I also want to give a shout out to Outlines team, https://github.com/outlines-dev/outlines, they've been doing structured outputs for the last 12 months, and their open source lib can be applied across all open weight and closed/API-based models. Most likely, Outlines heavily inspired the OpenAI team, maybe even using some of their codebase.

brap• 2 years ago

Structured outputs seem like an important step forward for LLMs to make them more useful (this includes intermediate "outputs" that are internal to the model, e.g. producing code and then running it without AI, producing mathematical/logical statements and then proving them without AI, etc). Basically, once you have structure, you can use non-AI tools that are far more efficient and don't hallucinate. These tools can either be external, or embedded in the model itself (think RAG, reasoning, etc.).

I wonder if JSON is just a first step, and eventually this will be generalized to any formal grammar...

mariarmestre• 2 years ago

Hmmm... this is not eligible for their zero data retention policy anymore. Not sure how this will go down.

kevlened2• 2 years ago

They added a clause that the schemas themselves aren't covered by ZDR. Their policy for the prompts appears unchanged.

https://platform.openai.com/docs/models/default-usage-polici...

blixt• 2 years ago

I was impressed by Microsoft’s AICI where the idea is a WASM program can choose the next tokens. And relatedly their Guidance[1] framework which can use CFGs and programs for local inference to even speed it up with context aware token filling. I hope this implies API-based LLMs may be moving in a similar direction.

[1] https://github.com/guidance-ai/guidance

munro• 2 years ago

Wow, this is cool-- first time I've seen generating well formed output has been made easy, the use of Pydantic is clever/easy...

Which has lead me to the prior art: using same Pydantic API to generate structured output from local LLMs

https://github.com/outlines-dev/outlines?tab=readme-ov-file#...

Terretta2• 2 years ago

Commercial use of LLMs is one of those spaces a startup probably needs more than skating ahead of the incumbents on the incumbents' “missing” features as startup value prop. Even if the startup has a year or two head start, it may find its beta “OBE” (overcome by events):

https://dottxt.co

That said, Outline has been making this concept portable for a long time, it's the langchain the community deserves:

https://outlines-dev.github.io/outlines/cookbook/

tipsytoad• 2 years ago

How is this different from instructor? github.com/jxnl/instructor

namely, why did they take so long for something that just seems like a wrapper around function calling?

hallqv• 2 years ago

They promised us super intelligence - all we got was valid json

MeetBagelPPML• 2 years ago

This is super cool!

If you're looking to get in early on a free tool that allows you to finetune your model using encrypted data sets you can sign up here: https://waitlist.bagel.net/

Structured Outputs in the API

Related Stories