GPT-5: Key characteristics, pricing and system card

https://simonwillison.net/2025/Aug/7/gpt-5/

by Philpax • 12 months ago

643 295 12 months ago

simonwillison.net

System card: https://cdn.openai.com/pdf/8124a3ce-ab78-4f06-96eb-49ea29ffb...

ks2048• 12 months ago

So, "system card" now means what used to be a "paper", but without lots of the details?

kaoD• 12 months ago

Nope. System card is a sales thing. I think we generally call that "product sheet" in other markets.

simonw29• 12 months ago

AI labs tend to use "system cards" to describe their evaluation and safety research processes.

They used to be more about the training process itself, but that's increasingly secretive these days.

matusp• 12 months ago

Despite the fact that their models are used in hiring, business, education, etc this multibillion company uses one benchmark with very artificial questions (BBQ) to evaluate how fair their model is. I am a little bit disappointed.

xmorse• 12 months ago

It's because these industries don't create their own benchmarks. The only ones creating evals are the AI company themselves or open source software engineers

Leary2• 12 months ago

METR of only 2 hours and 15 minutes. Fast takeoff less likely.

qsort2• 12 months ago

Isn't that pretty much in line with what people were expecting? Is it surprising?

dingnuts• 12 months ago

It's not surprising to AI critics but go back to 2022 and open r/singularity and then answer: what "people" were expecting? Which people?

SamA has been promising AGI next year for three years like Musk has been promising FSD next year for the last ten years.

IDK what "people" are expecting but with the amount of hype I'd have to guess they were expecting more than we've gotten so far.

The fact that "fast takeoff" is a term I recognize indicates that some people believed OpenAI when they said this technology (transformers) would lead to sci fi style AI and that is most certainly not happening

usaar333• 12 months ago

No, this is below expectations on both Manifold and lesswrong (https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_green...). Median was ~2.75 hours on both (which already represented a bearish slowdown).

Not massively off -- manifold yesterday implied odds this low were ~35%. 30% before Claude Opus 4.1 came out which updated expected agentic coding abilities downward.

umanwizard• 12 months ago

What is METR?

Leary2• 12 months ago

https://metr.github.io/autonomy-evals-guide/gpt-5-report/

tunesmith• 12 months ago

The 2h 15m is the length of tasks the model can complete with 50% probability. So longer is better in that sense. Or at least, "more advanced" and potentially "more dangerous".

ravendug• 12 months ago

https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measu...

wisemang• 12 months ago

To maybe save others some time METR is a group called Model Evaluation and Threat Research who

propose measuring AI performance in terms of the length of tasks AI agents can complete.

Not that hard to figure out but the way people refer were referring to them made me think it stood for an actual metric.

kqr• 12 months ago

Seems like it's on the line that's scaring people like AI 2027, isn't it? https://aisafety.no/img/articles/length-of-tasks-log.png

FergusArgyll2• 12 months ago

It's above the exponential line & right around the Super exponential line

Davidzheng2• 12 months ago

I actually think there's a high chance that this curve becomes almost vertical at some point around a few hours. I think in less than 1 hour regime, scaling the time scales the complexity which the agent must internalize. While after a few hours, limitations of humans means we have to divide into subtasks/abstractions each of which are bounded in complexity which must be internalized. And there's a separate category of skills which are needed like abstraction, subgoal creation, error correction. It's a flimsy argument but I don't see scaling time of tasks for humans as a very reliable metric at all.

nickthegreek• 12 months ago

This new naming conventions, while not perfect are alot clearer and I am sure will help my coworkers.

anyg• 12 months ago

Good to know - > Knowledge cut-off is September 30th 2024 for GPT-5 and May 30th 2024 for GPT-5 mini and nano

falcor842• 12 months ago

Oh wow, so essentially a full year of post-training and testing. Or was it ready and there was a sufficiently good business strategy decision to postpone the release?

thorum2• 12 months ago

The Information’s report from earlier this month claimed that GPT-5 was only developed in the last 1-2 months, after some sort of breakthrough in training methodology.

As recently as June, the technical problems meant none of OpenAI’s models under development seemed good enough to be labeled GPT-5, according to a person who has worked on it.

But it could be that this refers to post-training and the base model was developed earlier.

https://www.theinformation.com/articles/inside-openais-rocky...

https://archive.ph/d72B4

NullCascade• 12 months ago

OpenAI is much more aggressively targeted by NYTimes and similar organizations for "copyright violations".

bhouston• 12 months ago

Weird to have such an early knowledge cutoff. Claude 4.1 has March 2025 - 6 month more recent with comparable results.

freediver2• 12 months ago

Unless in the last 12 months so much of content on the web was AI generated that it reduced the quality of the model.

arresin• 12 months ago

Is that late enough for it to have heard of svelte 5?

dortlick2• 12 months ago

Yeah I thought that was strange. Wouldn't it be important to have more recent data?

cco2• 12 months ago

Only a third cheaper than Sonnet 4? Incrementally better I suppose.

and minimizing sycophancy

Now we're talking about a good feature! Actually one of my biggest annoyances with Cursor (that mostly uses Sonnet).

"You're absolutely right!"

I mean not really Cursor, but ok. I'll be super excited if we can get rid of these sycophancy tokens.

logicchains• 12 months ago

Only a third cheaper than Sonnet 4?

The price should be compared to Opus, not Sonnet.

cco2• 12 months ago

Wow, if so, 7x cheaper. Crazy if true.

nosefurhairdo• 12 months ago

In my early testing gpt5 is significantly less annoying in this regard. Gives a strong vibe of just doing what it's told without any fluff.

bdcdo• 12 months ago

"GPT-5 in the API is simpler: it’s available as three models—regular, mini and nano—which can each be run at one of four reasoning levels: minimal (a new level not previously available for other OpenAI reasoning models), low, medium or high."

Is it actually simpler? For those who are currently using GPT 4.1, we're going from 3 options (4.1, 4.1 mini and 4.1 nano) to at least 8, if we don't consider gpt 5 regular - we now will have to choose between gpt 5 mini minimal, gpt 5 mini low, gpt 5 mini medium, gpt 5 mini high, gpt 5 nano minimal, gpt 5 nano low, gpt 5 nano medium and gpt 5 nano high.

And, while choosing between all these options, we'll always have to wonder: should I try adjusting the prompt that I'm using, or simply change the gpt 5 version or its reasoning level?

impossiblefork3• 12 months ago

Yes, I think so. It's n=1,2,3 m=0,1,2,3. There's structure and you know that each parameter goes up and in which direction.

makeramen• 12 months ago

But given the option, do you choose bigger models or more reasoning? Or medium of both?

mwigdahl2• 12 months ago

If reasoning is on the table, then you already had to add o3-mini-high, o3-mini-medium, o3-mini-low, o4-mini-high, o4-mini-medium, and o4-mini-low to the 4.1 variants. The GPT-5 way seems simpler to me.

hirako20002• 12 months ago

Ultimately they are selling tokens, so try many times.

vineyardmike• 12 months ago

When I read “simpler” I interpreted that to mean they don’t use their Chat-optimized harness to guess which reasoning level and model to use. The subscription chat service (ChatGPT) and the chat-optimized model on their API seem to have a special harness that changes reasoning based on some heuristics, and will switch between the model sizes without user input.

With the API, you pick a model sizes and reasoning effort. Yes more choices, but also a clear mental model and a simple choice that you control.

diggan4• 12 months ago

but for the moment here’s the pelican I got from GPT-5 running at its default “medium” reasoning effort:

Would been interesting to see a comparison between low, medium and high reasoning_effort pelicans :)

When I've played around with GPT-OSS-120b recently, seems the difference in the final answer is huge, where "low" is essentially "no reasoning" and with "high" it can spend seemingly endless amount of tokens. I'm guessing the difference with GPT-5 will be similar?

simonw29• 12 months ago

Would been interesting to see a comparison between low, medium and high reasoning_effort pelicans

Yeah, I'm working on that - expect dozens of more pelicans in a later post.

meatmanek• 12 months ago

Would also be interesting to see how well they can do with a loop of: write SVG, render SVG, feed SVG back to LLM for review, iterate. Sorta like how a human would actually compose an SVG of a pelican.

zaronymous1• 12 months ago

Can anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.

Der_Einzige• 12 months ago

It's because all forms of sampler settings destroy safety/alignment. That's why top_p/top_k are still used and not tfs, min_p, top_n sigma, etc, why temperature is locked to 0-2 arbitrary range, etc

Open source is years ahead of these guys on samplers. It's why their models being so good is that much more impressive.

oblio2• 12 months ago

Temperature is the response variation control?

hodgehog11• 12 months ago

The aggressive pricing here seems unusual for OpenAI. If they had a large moat, they wouldn't need to do this. Competition is fierce indeed.

0x00cl• 12 months ago

Maybe the need/want data.

dr_dshiv• 12 months ago

And it’s a massive distillation of the mother model, so the costs of inference are likely low.

impure2• 12 months ago

OpenAI and most AI companies do not train on data submitted to a paid API.

impure2• 12 months ago

The 5 cents for Nano is interesting. Maybe it will force Google to start dropping their prices again which have been slowly creeping up recently.

ilaksh2• 12 months ago

It's like 5% better. I think they obviously had no choice but to be price competitive with Gemini 2.5 Pro. Especially for Cursor to change their default.

FergusArgyll2• 12 months ago

They are winning by massive margins in the app, but losing (!) in the API to anthropic

https://finance.yahoo.com/news/enterprise-llm-spend-reaches-...

canada_dry• 12 months ago

Perhaps they're feeling the effect of losing PRO clients (like me) lately.

Their PRO models were not (IMHO) worth 10X that of PLUS!

Not even close.

Especially when new competitors (eg. z.ai) are offering very compelling competition.

onehair• 12 months ago

Definitely recognizable as a pelican

right :-D

pancakemouse2• 12 months ago

Practically the first thing I do after a new model release is try to upgrade `llm`. Thank you, @simonw !

efavdb• 12 months ago

same, looks like he hasn't added 5.0 to the package yet but assume imminent.

https://llm.datasette.io/en/stable/openai-models.html

simonw29• 12 months ago

Working on that now! https://github.com/simonw/llm/issues/1229

pancakemouse2• 11 months ago

I want to build a tool that can render SVG to an image, then return that image so the model can see what it has drawn. For reasons.

isoprophlex• 12 months ago

Whoa this looks good. And cheap! How do you hack a proxy together so you can run Claude Code on gpt-5?!

dalberto• 12 months ago

Consider: https://github.com/musistudio/claude-code-router

or even: https://github.com/sst/opencode

Not affiliated with either one of these, but they look promising.

morleytj9• 12 months ago

It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.

It does sort of give me the vibe that the pure scaling maximalism really is dying off though. If the approach is on writing better routers, tooling, comboing specialized submodels on tasks, then it feels like there's a search for new ways to improve performance(and lower cost), suggesting the other established approaches weren't working. I could totally be wrong, but I feel like if just throwing more compute at the problem was working OpenAI probably wouldn't be spending much time on optimizing the user routing on currently existing strategies to get marginal improvements on average user interactions.

I've been pretty negative on the thesis of only needing more data/compute to achieve AGI with current techniques though, so perhaps I'm overly biased against it. If there's one thing that bothers me in general about the situation though, it's that it feels like we really have no clue what the actual status of these models is because of how closed off all the industry labs have become + the feeling of not being able to expect anything other than marketing language from the presentations. I suppose that's inevitable with the massive investments though. Maybe they've got some massive earthshattering model release coming out next, who knows.

jstummbillig• 12 months ago

Things have moved differently than what we thought would happen 2 years ago, but lest we forget what has happened in the meanwhile (4o, o1 + thinking paradigm, o3)

So yeah, maybe we are getting more incremental improvements. But that to me seems like a good thing, because more good things earlier. I will take that over world-shattering any day – but if we were to consider everything that has happened since the first release of gpt-4, I would argue the total amount is actually very much world-shattering.

GaggiX• 12 months ago

Compared to GPT-4, it is on a completely different level given that it is a reasoning model so on that regard it does delivers and it's not just scaling, but for this I guess the revolution was o1 and GPT-5 is just a much more mature version of the technology.

hnuser123456• 12 months ago

I agree, we have now proven that GPUs can ingest information and be trained to generate content for various tasks. But to put it to work, make it useful, requires far more thought about a specific problem and how to apply the tech. If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year, someone else would've already done it. Elbow grease still required for the foreseeable future.

In the meantime, figuring out how to train them to make less of their most common mistakes is a worthwhile effort.

morleytj9• 12 months ago

Certainly, yes, plenty of elbow grease required in all things that matter.

The interesting point as well to me though, is that if it could create a startup that was worth $1B, that startup wouldn't be worth $1B.

Why would anyone pay that much to invest in the startup if they could recreate the entire thing with the same tool that everyone would have access to?

BoiledCabbage• 12 months ago

Performance is doubling roughly every 4-7 months. That trend is continuing. That's insane.

If your expectations were any higher than that then, then it seems like you were caught up in hype. Doubling 2-3 times per year isn't leveling off my any means.

https://metr.github.io/autonomy-evals-guide/gpt-5-report/

oblio2• 12 months ago

By "performance" I guess you mean "the length of task that can be done adequately"?

It is a benchmark but I'm not very convinced it's the be-all, end-all.

morleytj9• 12 months ago

I wouldn't say model development and performance is "leveling off", and in fact didn't write that. I'd say that tons more funding is going into the development of many models, so one would expect performance increases unless the paradigm was completely flawed at it's core, a belief I wouldn't personally profess to. My point was moreso the following: A couple years ago it was easy to find people saying that all we needed was to add in video data, or genetic data, or some other data modality, in the exact same format that the models trained on existing language data were, and we'd see a fast takeoff scenario with no other algorithmic changes. Given that the top labs seem to be increasingly investigating alternate approaches to setting up the models beyond just adding more data sources, and have been for the last couple years(Which, I should clarify, is a good idea in my opinion), then the probability of those statements of just adding more data or more compute taking us straight to AGI being correct seems at the very least slightly lower, right?

Rather than my personal opinion, I was commenting on commonly viewed opinions of people I would believe to have been caught up in hype in the past. But I do feel that although that's a benchmark, it's not necessarily the end-all of benchmarks. I'll reserve my final opinions until I test personally, of course. I will say that increasing the context window probably translates pretty well to longer context task performance, but I'm not entirely convinced it directly translates to individual end-step improvement on every class of task.

andrepd2• 12 months ago

We can barely measure "performance" in any objective sense, let alone claim that it's doubling every 4 months.....

simonw29• 12 months ago

I for one am pretty glad about this. I like LLMs that augment human abilities - tools that help people get more done and be more ambitious.

The common concept for AGI seems to be much more about human replacement - the ability to complete "economically valuable tasks" better than humans can. I still don't understand what our human lives or economies would look like there.

What I personally wanted from GPT-5 is exactly what I got: models that do the same stuff that existing models do, but more reliably and "better".

morleytj9• 12 months ago

I'd agree on that.

That's pretty much the key component these approaches have been lacking on, the reliability and consistency on the tasks they already work well on to some extent.

I think there's a lot of visions of what our human lives would look like in that world that I can imagine, but your comment did make me think of one particularly interesting tautological scenario in that commonly defined version of AGI.

If artificial general intelligence is defined as completed "economically valuable tasks" better than human can, it requires one to define "economically valuable." As it currently stands, something holds value in an economy relative to human beings wanting it. Houses get expensive because many people, each of whom have economic utility which they use to purchase things, want to have houses, of which there is a limited supply for a variety of reasons. If human beings are not the most effective producers of value in the system, they lose capability to trade for things, which negates that existing definition of economic value. Doesn't matter how many people would pay $5 dollars for your widget if people have no economic utility relative to AGI, meaning they cannot trade that utility for goods.

In general that sort of definition of AGI being held reveals a bit of a deeper belief, which is that there is some version of economic value detached from the humans consuming it. Some sort of nebulous concept of progress, rather than the acknowledgement that for all of human history, progress and value have both been relative to the people themselves getting some form of value or progress. I suppose it generally points to the idea of an economy without consumers, which is always a pretty bizarre thing to consider, but in that case, wouldn't it just be a definition saying that "AGI is achieved when it can do things that the people who control the AI system think are useful." Since in that case, the economy would eventually largely consist of the people controlling the most economically valuable agents.

I suppose that's the whole point of the various alignment studies, but I do find it kind of interesting to think about the fact that even the concept of something being "economically valuable", which sounds very rigorous and measurable to many people, is so nebulous as to be dependent on our preferences and wants as a society.

thorum2• 12 months ago

The quiet revolution is happening in tool use and multimodal capabilities. Moderate incremental improvements on general intelligence, but dramatic improvements on multi-step tool use and ability to interact with the world (vs 1 year ago), will eventually feed back into general intelligence.

darkhorse222• 12 months ago

Completely agree. General intelligence is a building block. By chaining things together you can achieve meta programming. The trick isn't to create one perfect block but to build a variety of blocks and make one of those blocks a block-builder.

coolKid721• 12 months ago

[flagged]

thomasfromcdnjs• 12 months ago

100%

1) Build a directory of X (a gazillion) amount of tools (just functions) that models can invoke with standard pipeline behavior (parallel, recursion, conditions etc)

2) Solve the "too many tools to select from" problem (a search problem), adjacently really understand the intent (linguistics/ToM) of the user or agents request

3) Someone to pay for everything

4) ???

The future is already here in my opinion, the LLM's are good-enough™, it's just the ecosystem needs to catch up. Companies like Zapier or whatever, taken to their logical extreme, connecting any software to any thing (not just sass products), combined with an LLM will be able to do almost anything.

Even better basic tool composition around language will make it's simple replies better too.

copularent• 12 months ago

I think we have reached a user schism in terms of benefits going forward.

I am completely floored by GPT-5. I only tried it a half hour ago and have a whole new data analysis pipeline. I thought it must be hallucinating badly at first but all the papers it referenced are real and I had just never heard of these concepts.

This is for an area that has 200 papers on arxiv and I have read all of them so thought I knew this area well.

I don't see how the average person benefits much going forward though. They simply don't have questions to ask in order to have the model display its intelligence.

belter4• 12 months ago

Maybe they've got some massive earthshattering model release coming out next, who knows.

Nothing in the current technology offers a path to AGI. These models are fixed after training completes.

echoangle6• 12 months ago

Why do you think that AGI necessitates modification of the model during use? Couldn’t all the insights the model gains be contained in the context given to it?

cchance2• 12 months ago

SAM is a HYPE CEO, he literally hypes his company nonstop, then the announcements come and ... they're... ok, so people aren't really upset, but they end up feeling lackluster at the hype... Until the next cycle comes around...

If you want actual big moves, watch google, anthropic, qwen, deepseek.

Qwen and Deepseek teams honestly seem so much better at under promising and over delivering.

Cant wait to see what Gemini 3 looks like too.

brandall102• 12 months ago

To be fair, this is one of the pathways GPT-5 was speculated to take as far back at 6 or so months ago - simply being an incremental upgrade from a performance perspective, but a leap from a product simplification approach.

At this point it's pretty much given it's a game of inches moving forward.

ac29• 12 months ago

a leap from a product simplification approach.

According to the article, GPT-5 is actually three models and they can be run at 4 levels of thinking. Thats a dozen ways you can run any given input on "GPT-5", so its hardly a simple product line up (but maybe better than before).

AbstractH24• 12 months ago

It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.

Are you trying to say the curve is flattening? That advances are coming slower and slower?

As long as it doesn't suggest a dot com level recession I'm good.

morleytj9• 12 months ago

I suppose what I'm getting at is that if there are performance increases on a steady pace, but the investment needed to get those performance increases is on a much faster growth rate, it's not really a fair comparison in terms of a rate of progress, and could suggest diminishing returns from a particular approach. I don't really have the actual data to make a claim either way though,I think anyone would need more data to do so than is publicly accessible.

But I do think the fact that we can publicly observe this reallocation of resources and emphasized aspects of the models gives us a bit of insight into what could be happening behind the scenes if we think about the reasons why those shifts could have happened, I guess.

godelski26• 12 months ago

  > It does sort of give me the vibe that the pure scaling maximalism really is dying off though

I think the big question is if/when investors will start giving money to those who have been predicting this (with evidence) and trying other avenues.

Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. Why fund yet another LLMs to AGI startup. Space is saturated with big players and has been for years. Even if LLMs could get there that doesn't mean something else won't get there faster and for less. It also seems you'd want a backup in order to avoid popping the bubble. Technology S-Curves and all that still apply to AI

Though I'm similarly biased, but so is everyone I know with a strong math and/or science background (I even mentioned it in my thesis more than a few times lol). Scaling is all you need just doesn't check out

morleytj9• 12 months ago

I'm pretty curious about the same thing.

I think a somewhat comparable situation is in various online game platforms now that I think about it. Investors would love to make a game like Fortnite, and get the profits that Fortnite makes. So a ton of companies try to make Fortnite. Almost all fail, and make no return whatsoever, just lose a ton of money and toss the game in the bin, shut down the servers.

On the other hand, it may have been more logical for many of them to go for a less ambitious (not always online, not a game that requires a high player count and social buy-in to stay relevant) but still profitable investment (Maybe a smaller scale single player game that doesn't offer recurring revenue), yet we still see a very crowded space for trying to emulate the same business model as something like Fortnite. Another more historical example was the constant question of whether a given MMO would be the next "WoW-killer" all through the 2000's/2010's.

I think part of why this arises is that there's definitely a bit of a psychological hack for humans in particular where if there's a low-probability but extremely high reward outcome, we're deeply entranced by it, and investors are the same. Even if the chances are smaller in their minds than they were before, if they can just follow the same path that seems to be working to some extent and then get lucky, they're completely set. They're not really thinking about any broader bubble that could exist, that's on the level of the society, they're thinking about the individual, who could be very very rich, famous, and powerful if their investment works. And in the mind of someone debating what path to go down, I imagine a more nebulous answer of "we probably need to come up with some fundamentally different tools for learning and research a lot of different approaches to do so" is a bit less satisfying and exciting than a pitch that says "If you just give me enough money, the curve will eventually hit the point where you get to be king of the universe and we go colonize the solar system and carve your face into the moon."

I also have to acknowledge the possibility that they just have access to different information than I do! They might be getting shown much better demos than I do, I suppose.

famouswaffles5• 12 months ago

Really though, why put all your eggs in one basket? That's what I've been confused about for awhile.

I mean that's easy lol. People don't like to invest in thin air, which is what you get when you look at non-LLM alternatives to General Intelligence.

This isn't meant as a jab or snide remark or anything like that. There's literally nothing else that will get you GPT-2 level performance, never-mind an IMO Gold Medalist. Invest in what else exactly? People are putting their eggs in one basket because it's the only basket that exists.

I think the big question is if/when investors will start giving money to those who have been predicting this (with evidence) and trying other avenues.

Because those people have still not been proven right. Does "It's an incremental improvement over the model we released a few months ago, and blows away the model we released 2 years ago." really scream, "See!, those people were wrong all along!" to you ?

eru5• 12 months ago

Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. Why fund yet another LLMs to AGI startup.

Funding multiple startups means _not_ putting your eggs in one basket, doesn't it?

Btw, do we have any indication that eg OpenAI is restricting themselves to LLMs?

l33tman• 12 months ago

I started such an alternative project just before GPT-3 was released, it was really promising (lots of neuroscience inspired solutions, pretty different to Transformers) but I had to put it on hold because the investors I approached seemed like they would only invest in LLM-stuff. Now a few years later I'm trying to approach investors again, only to find now they want to invest in companies USING LLMs to create value and still don't seem interested in new foundational types of models... :/

I guess it makes sense, there is still tons of value to be created just by using the current LLMs for stuff, though maybe the low hanging fruits are already picked, who knows.

I heard John Carmack talk a lot about his alternative (also neuroscience-inspired) ideas and it sounded just like my project, the main difference being that he's able to self-fund :) I guess funding an "outsider" non-LLM AI project now requires finding someone like Carmack to get on board - I still don't think traditional investors are that disappointed yet that they want to risk money on other types of projects..

csomar• 12 months ago

The current money made its money following the market. They do not have the capacity for innovation or risk taking.

famouswaffles5• 12 months ago

you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.

Compared to the GPT-4 release which was a little over 2 years ago (less than the gap between 3 and 4), it is. The only difference is we now have multiple organizations releasing state of the art models every few months. Even if models are improving at the same rate, those same big jumps after every handful of months was never realistic.

It's an incremental stable improvement over o3, which was released what? 4 months ago.

morleytj9• 12 months ago

The benchmarks certainly seem to be improving from the presentation. I don't think they started training this 4 months ago though.

There's gains, but the question is, how much investment for that gain? How sustainable is that investment to gain ratio? The things I'm curious about here are more about the amount of effort being put into this level of improvement, rather than the time.

outside1234• 12 months ago

The next step will be for OpenAI to number their releases based on year (ala what Windows did once innovation ran out)

eru5• 12 months ago

Windows 95 was a big step from the previous release, wasn't it?

And later, Windows reverted to version numbers; but I'm not sure they regained lots of innovation?

fastball• 12 months ago

My reading is more that unit economics are starting to catch up with the frontier labs, rather than "scaling maximalism is dying". Maybe that is the same thing.

ch4s3• 12 months ago

My loosely held belief is that it is the same thing, but I’m open to being proven wrong.

danenania• 12 months ago

Isn’t reasoning, aka test-time compute, ultimately just another form of scaling? Yes it happens at a different stage, but the equation is still 'scale total compute > more intelligence'. In that sense, combining their biggest pre-trained models with their best reasoning strategies from RL could be the most impactful scaling lever available to them at the moment.

mwigdahl2• 12 months ago

It seems like few people are referencing the improvements in reliability and deception. If the benchmarks given generalize, what OpenAI has in GPT-5 is a cheap, powerful, _reliable_ model -- the perfect engine to generate high quality synthetic data to punch through the training data bottleneck.

I'd expect that at some level of reliability this could lead to a self-improvement cycle, similar to how a powerful enough model (the Claude 4 models in Claude Code) enables iteratively converging on a solution to a problem even if it can't one-shot it.

No idea if we're at that point yet, but it seems a natural use for a model with these characteristics.

maoberlehner6• 12 months ago

I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini.

The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words.

The only thing transformer based LLMs can ever do is _faking_ intelligence.

Which for many tasks is good enough. Even in my example above, the corrected text was flawless.

But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.

butler14• 12 months ago

I had this too last week. It pointed out two errors that simply weren’t there. Then completely refused to back down and doubled down on its own certainty, until I sent it a screenshot of the original prompt. Kind of funny.

shmel• 12 months ago

do you really think that an architecture that struggles to count r in strawberry is a good choice for proofreading? It perceives words very differently from us.

morganf• 12 months ago

One of my main uses for LLMs is copy editing and it is incredible to me how terrible all of them are at that.

awestroke3• 12 months ago

I'll show you a few misspelled words and you tell me (without using any tools or thinking it through) which bits in the utf8 encoded bytes are incorrect. If you're wrong, I'll conclude you are not intelligent.

LLMs don't see letters, they see tokens. This is a foundational attribute of LLMs. When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool.

dotancohen• 12 months ago

  > but given the types of things people have been saying GPT-5 would be for the last two years

This is why you listen to official announcements, not "people".

eloisant• 12 months ago

It reminds me of the latest, most advanced steam locomotives from the beginning of the 20th century.

They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters.

I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.

anshumankmr2• 12 months ago

Just to add to this, if we see F1 cars a way to measure the cutting edge of cars being developed, we can see cars haven't become insanely faster than they were 10-20 years ago, just more "efficient",reliable and definitely safer along with quirks like DRS. Of course shaving off a second or two from laptimes is notable, but not an insane delta like say if you compared a car from post 2000s GP to 1950s GP.

I feel after a while we will have specialized LLMs great for one particular task down the line as well, cut off updates, 0.something better than the SOTA on some benchmark and as compute gets better, cheaper to run at scale.

nyrikki• 12 months ago

Even more connections, the UPs Big Boy was designed to replace double header or helper engines to reduce labor costs because they were using the cheapest bismuth coal.

It wasn't until after WWII when coal and labor costs and the development of good diesel electric that things changed.

The UP expended huge amounts of money to daylight tunnels and replacing bridges in a single year to support the big boys.

Just like AI it was meant to replace workers to improve investors returns.

jillesvangurp• 12 months ago

Year on year, progress is indeed a bit incremental. But seen over a five year period the progress is actually stupidly amazing.

In practical terms, Gpt 5 is a nice upgrade over most other models. We'll no doubt get lots of subjective reports how it was wrong or right or worse than some other model for some chats. But my personal (subjective) experience so far is that it just made it possible for me to use codex on more serious projects. It still gets plenty of things wrong. But that's more because of a lack of context than hallucination issues. Context fixes are a lot easier than model improvements. But last week I didn't bother and now I'm getting decent results.

I don't really care what version number they slap on things. That is indeed just marketing. And competition is quite fierce so I can understand why they are overselling what could have been just chat gpt 4.2 or whatever.

Also discussions about AGI tend to bore me as they seem to escalate into low quality philosophical debates with lots of amateurs rehashing ancient argument poorly. There aren't a hell of a lot new arguments that people come up with at this point.

IMHO we don't actually need an AGI to bootstrap the singularity. We just AIs to be good enough to come up with algorithmic optimizations, breakthroughs and improvements at a steady pace. We're getting quite close to that and I wouldn't be surprised to learn that OpenAI's people are already eating their own dogfood in liberal quantities. It's not necessary for AIs to be conscious in order to come up with the improvements that might eventually enable such a thing. I expect the singularity might be more of a phase than a moment. And if you know your boiling frog analogy, we might be smack down in the middle of that already and just not realize it.

Five years ago, it was all very theoretical. And now I'm waiting for codex to wrap up a few pull requests that would have distracted me for a week each five years ago. It's taking too long and I'm procrastinating my gained productivity away on HN. But what else is new ;-).

tough• 12 months ago

there's no more data to throw compute at to 1T tokens is a lot

ilaksh2• 12 months ago

This is key info from the article for me:

-------------------------------

"reasoning": {"summary": "auto"} }'

Here’s the response from that API call.

https://gist.github.com/simonw/1d1013ba059af76461153722005a0...

Without that option the API will often provide a lengthy delay while the model burns through thinking tokens until you start getting back visible tokens for the final response.

justusthane• 12 months ago

a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent

This is sort of interesting to me. It strikes me that so far we've had more or less direct access to the underlying model (apart from the system prompt and guardrails), but I wonder if going forward there's going to be more and more infrastructure between us and the model.

hirako20002• 12 months ago

Consider it a low level routing. Keeping in mind it allows the other non active parts to not be in memory. Mistral afaik came up with this concept, quite a while back.

ItsHarper2• 12 months ago

It's actually just a high-level routing between the reasoning and non-reasoning models that only applies to ChatGPT.

ItsHarper2• 12 months ago

That only applies to ChatGPT. The API has direct access to specific models.

willy_k• 12 months ago

The router seems to only apply to the ChatGPT version, not the API, so it’s not really anything new. Gemini already has functionally dynamic reasoning effort.

techpression2• 12 months ago

"They claim impressive reductions in hallucinations. In my own usage I’ve not spotted a single hallucination yet, but that’s been true for me for Claude 4 and o3 recently as well—hallucination is so much less of a problem with this year’s models."

This has me so confused, Claude 4 (Sonnet and Opus) hallucinates daily for me, on both simple and hard things. And this is for small isolated questions at that.

simonw29• 12 months ago

What kind of hallucinations are you seeing?

OtherShrezzing3• 12 months ago

I rewrote a 4 page document from first to third person a couple of weeks back. I gave Claude Sonnet 4 the document after editing, so it was entirely written in the third person. I asked it to review & highlight places where it was still in the first person.

Looking through the document, I can identify several instances where it's written in the first person:

And it went on to show a series of "they/them" statements. I asked it to clarify if "they" is "first person" and it responded

No, "they" is not first person - it's third person. I made an error in my analysis. First person would be: I, we, me, us, our, my. Second person would be: you, your. Third person would be: he, she, it, they, them, their. Looking back at the document more carefully, it appears to be written entirely in third person.

Even the good models are still failing at real-world use cases which should be right in their wheelhouse.

techpression2• 12 months ago

Since I mostly use it for code, made up function names are the most common. And of course just broken code all together, which might not count as a hallucination.

laacz• 12 months ago

I suppose that Simon, being all in with LLMs for quite a while now, has developed a good intuition/feeling for framing questions so that they produce less hallucinations.

simonw29• 12 months ago

Yeah I think that's exactly right. I don't ask questions that are likely to product hallucinations (like citations from papers about a topic to an LLM without search access), so I rarely see them.

bluetidepro• 12 months ago

Agreed. All it takes is a simple reply of “you’re wrong.” to Claude/ChatGPT/etc. and it will start to crumble on itself and get into a loop that hallucinates over and over. It won’t fight back, even if it happened to be right to begin with. It has no backbone to be confident it is right.

cameldrv• 12 months ago

Yeah it may be that previous training data, the model was given a strong negative signal when the human trainer told it it was wrong. In more subjective domains this might lead to sycophancy. If the human is always right and the data is always right, but the data can be interpreted multiple ways, like say human psychology, the model just adjusts to the opinion of the human.

If the question is about harder facts which the human disagrees with, this may put it into an essentially self-contradictory state, where the locus of possibilitie gets squished from each direction, and so the model is forced to respond with crazy outliers which agree with both the human and the data. The probability of an invented reference being true may be very low, but from the model's perspective, it may still be one of the highest probability outputs among a set of bad choices.

What it sounds like they may have done is just have the humans tell it it's wrong when it isn't, and then award it credit for sticking to its guns.

diggan4• 12 months ago

All it takes is a simple reply of “you’re wrong.” to Claude/ChatGPT/etc. and it will start to crumble on itself and get into a loop that hallucinates over and over.

Yeah, it's seems to be a terrible approach to try to "correct" the context by adding clarifications or telling it what's wrong.

Instead, start from 0 with the same initial prompt you used, but improve it so the LLM gets it right in the first response. If it still gets it wrong, begin from 0 again. The context seems to be "poisoned" really quickly, if you're looking for accuracy in the responses. So better to begin from the beginning as soon as it veers off course.

petesergeant• 12 months ago

All it takes is a simple reply of “you’re wrong.” to Claude/ChatGPT/etc. and it will start to crumble on itself

Fucking Gemini Pro on the other hand digs in, and starts deciding it's in a testing scenario and get adversarial, starts claiming it's using tools the user doesn't know about, etc etc

squeegmeister• 12 months ago

Yeah hallucinations are very context dependent. I’m guessing OP is working in very well documented domains

Oras• 12 months ago

Here you go https://pbs.twimg.com/media/Gxxtiz7WEAAGCQ1?format=jpg&name=...

simonw29• 12 months ago

How is that a hallucination?

madduci• 12 months ago

I believe it depends in inputs. For me, Claude 4 has consistently generated hallucinations, especially was pretty confident in generating invalid JSONs, for instance Grafana Dashboards, which were full of syntactic errors.

godelski26• 12 months ago

There were also several hallucinations during the announcement. (I also see hallucinations every time I use Claude and GPT, which is several times a week. Paid and free tiers)

So not seeing them means either lying or incompetent. I always try to attribute to stupidity rather than malice (Hanlon's razor).

The big problem of LLMs is that they optimize human preference. This means they optimize for hidden errors.

Personally I'm really cautious about using tools that have stealthy failure modes. They just lead to many problems and lots of wasted hours debugging, even when failure rates are low. It just causes everything to slow down for me as I'm double checking everything and need to be much more meticulous if I know it's hard to see. It's like having a line of Python indented with an inconsistent white space character. Impossible to see. But what if you didn't have the interpreter telling you which line you failed on or being able to search or highlight these different characters. At least in this case you'd know there's an error. It's hard enough dealing with human generated invisible errors, but this just seems to perpetuate the LGTM crowd

hhh• 12 months ago

You can just have a different use case that surfaces hallucinations than someone, they don’t have to by evil.

simonw29• 12 months ago

What were the hallucinations during the announcement?

My incompetence here was that I was careless with my use of the term "hallucination" here. I assumed everyone else shared my exact definition - that a hallucination is when a model confidently states a fact that is entirely unconnected from reality, which is a different issue from a mistake ("how many Bs in blueberry" etc).

It's clear that MANY people do not share my definition! I deeply regret including that note in my post.

simonw29• 12 months ago

I updated that section of my post with a clarification about what I meant. Thanks for calling this out, it definitely needed extra context from me.

drumhead• 12 months ago

"Are you GPT5" - No I'm 4o, 5 hasnt been released yet. "It was released today". Oh you're right, Im GPT5. You have reached the limit of the free usage of 4o

nonhaver• 12 months ago

haha brutal. maybe tomorrow

cchance2• 12 months ago

Its basically opus 4.1 ... but cheaper?

gwd• 12 months ago

Cheaper is an understatement... it's less than 1/10 for input and nearly 1/8 for output. Part of me wonders if they're using their massive new investment to sell API below-cost and drive out the competitor. If they're really getting Opus 4.1 performance for half of Sonnet compute cost, they've done really well.

diggan4• 12 months ago

I'm not sure I'd be surprised, I've been playing around with GPT-OSS last few days, and the architecture seems really fast for the accuracy/quality of responses, way better than most local weights I've tried for the last two years or so. And since they released that architecture publicly, I'd imagine they're sitting on something even better privately.

bravesoul2• 12 months ago

With the unlimited demand I can't see that strategy working. It is not like taxis where you may do a trip or two a day but if it cheap enough you'd do 100 a day. But with AI you would totally 100x.

aliljet• 12 months ago

I'm curious what platform people are using to test GPT-5? I'm so deep into the claude code world that I'm actually unsure what the best option is outside of claude code...

te_chris• 12 months ago

Cursor

simonw29• 12 months ago

I've been using codex CLI, OpenAI's Claude Code equivalent. You can run it like this:

  OPENAI_DEFAULT_MODEL=gpt-5 codex

cainxinth2• 12 months ago

It’s fascinating and hilarious that pelican on a bicycle in SVG is still such a challenge.

muglug• 12 months ago

How easy is it for you to create an SVG of a pelican riding a bicycle in a text editor by hand?

jopsen• 12 months ago

Without looking at the rendered output :)

cainxinth2• 12 months ago

I didn't mean to imply it was simple, just that it's funny because I can't really evaluate evals like Humanity's Last Exam, but I can see the progress of these models in a pelican.

SomewhatLikely• 12 months ago

Nobody's preventing them from rendering it and refining. That's certainly what we'd expect an AGI to do.

throwaway422432• 12 months ago

I'm surprised they haven't all tried to game this test by now, or at least added it to their internal testing knowing they will be judged by it.

joshmlewis• 12 months ago

It seems to be trained to use tools effectively to gather context. In this example against 4.1 and o3 it used 6 in the first turn in a pretty cool way (fetching different categories that could be relevant). Token use increases with that kind of tool calling but the aggressive pricing should make that moot. You could probably get it to not be so tool happy with prompting as well.

https://promptslice.com/share/b-2ap_rfjeJgIQsG

tomrod• 12 months ago

Simon, as always, I appreciate your succinct and dedicated writeup. This really helps to land the results.

moralestapia• 12 months ago

Basically repeats what it's been out through the usual PR channels, just paraphrased.

No mention about the (missing) elephant on the room, where are the benchmarks?

@simonw has been compromised. Sad.

simonw29• 12 months ago

I'm sorry I didn't say "independent benchmarks are not yet available" in my post, I say that so often on model launches I guess I took it as read this time.

kevink23• 12 months ago

I was excited for GPT-5, but honestly, it feels worse than GPT-4 for coding.

simonw29• 12 months ago

GPT-4 or GPT-4o?

globular-toast• 12 months ago

This "system card" thing seems to have suddenly come out of nowhere. Signs of a cult forming. Is it just what we'd normally call a technical write up?

dragonwriter• 12 months ago

It’s a variation on “model card”, which has become a standard thing with AI models, but with the name changed because the wroteup covers toolchain as well as model information. But a PDF of the size of the document at issue is very much not the kind of concise document model cards are, its more the kind of technical report that a much more concise card would reference.

coldtea• 12 months ago

The improved bicycling pelican of course could be overfitting / benchmark-cheating...

ozgung• 12 months ago

One key element missing from all these model cards are the model size/number of parameters. Without that info we are in the dark. We can't predict the future of AI. How does the intelligence scale with the increasing #parameters? Is there a limit? Should we attribute incrementally better metrics to larger model size or other techniques? Do they announce the full model they trained or a smaller version that is economically viable for the market conditions? If they double the model size will it be a Professor-level intelligence, a super-human level intelligence or a couple of phds level intelligence?

diamond559• 12 months ago

We're supposed to get AGI next year and this is all they have. We are not getting real AI from LLMs people, wake up, we're in a bubble.

noisy_boy• 12 months ago

I wonder if there are people who find Sonnet 3.5 to be faster and good enough for programming instead of moving on to reasoning models (sometimes I do go to Gemini 2.5 pro if Sonnet doesn't cut it).

GPT-5: Key characteristics, pricing and system card

Related Stories