tedsanders

I work at OpenAI and I can assure you that if we say we don't train on your data, we don't.

I acknowledge that if you don't trust OpenAI, then you may not trust me either. But lying about this would be bad for legal liability, customer retention, and employee retention. Even if you model us as evil (and we really aren't), it's still not obvious to me that it would be a good decision to lie. As soon as a whistleblower revealed the scam, it would tank revenue and employee morale.

Arena can definitely be benchmaxxed a bit, if you try. The distribution of prompts there is very different than usage by regular coders. E.g., lots of requests for one-shot games from scratch. So if you fine-tuned your model to be great at making fun one-shot games from underspecified prompts, your coding model might look better than it is (on general tasks, at least).

I work at OpenAI, and am happy to say we don't try to juice our scores here, as doing so would be counterproductive and make Arena a worse signal for everyone.

Not entirely fixed yet, but should be rarer with 5.6. Don’t have a quantification, unfortunately.

Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.

We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.

https://agents-last-exam.org/leaderboard

Would have loved to publish a full curve for Fable if anyone makes the data available.

Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.

Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.

(I work at OpenAI.)

As usual, even though GPT-5.6 is releasing today, the rollout in ChatGPT and Codex will be gradual over many hours so that we can make sure service remains stable for everyone (same as our previous launches). We usually start with Pro/Enterprise accounts and then work our way down to Plus. We know it's slightly annoying to have to wait a random amount of time, but we do it this way to keep service maximally stable.

The timescale is typically hours not minutes, so if you don't see it now, I'd try again later today.

We mention it will be a gradual rollout over the next 24 hours in the Availability section at the bottom of the blog but I admit it's pretty buried.

(I work at OpenAI.)

Pointing out problems (e.g., hidden tests that assume narrow implementation details) is much easier than fixing them (e.g., creating tests that work for any possible choice of implementation).

I believe what the post meant to communicate is:

- alpha testers will start getting access now

- everyone will get access Thursday (barring banned countries / individuals)

Historically, some companies and individuals have gotten alpha access before public launches, to give feedback and adapt their products to the new models. With GPT-5.6, some folks had early alpha access, but this was paused while the model was being evaluated and approved. Now, alpha access will be enabled for partners in the next two days before our wider launch.

(I work at OpenAI.)

I work at OpenAI and can confirm that's correct: reasoning tokens are discarded after each new user turn (though not after each message or tool call).

Our docs show a diagram here:

https://developers.openai.com/api/docs/guides/reasoning

Input and output tokens from each step are carried over, while reasoning tokens are discarded.

Keeping reasoning tokens around is better for caching and for remembering past insights, so you might reasonably wonder why we designed it this way. The main benefit of dropping reasoning tokens is that you can fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step. This was a larger consideration with our earlier reasoning models that had shorter context windows (~200k), longer thinking times (up to ~100k per message), and poor compaction. However, now that we've shipped longer context windows, we've trained our models think much more efficiently, and we've made compaction way better than it used to be, the balance of factors is changing. Tune in Thursday!

Unfortunately we're not in a position where we can promise an exact date, but we expect it to take weeks (not days or months). It's the best coding model we've ever trained and we're bummed we can't release it to everyone yet. When we do launch, we'll share a lot more evals and testimonials and demos that help show what it's good/bad at. Personally hoping that both GPT-5.6 Sol and Fable 5 get broadly released soon so that everyone (myself included) can try them head to head.

(I work at OpenAI.)

Yeah, we'll share a lot more details and evals when we can release GPT-5.6 widely. We focused on cyber (and bio) here to help explain why it's being held back for now. We would have loved to launch it to everyone - it's the best coding model I've ever used - and we plan to do so as soon as we can ('coming weeks').

(I work at OpenAI.)

Makes sense, thanks. I suppose error bars are tricky if trying to handle problem-to-problem variance, rubric-to-rubric variance, and run-to-run variance all at once.

The nonprofit (OpenAI Foundation) owns ~26% of the for-profit, plus some extra warrants.

The for-profit (OpenAI Group PBC) is what's filing the S-1 Draft.

The OpenAI Foundation also exclusively appoints the board of the OpenAI Group PBC and can replace directors at any time.

https://openai.com/our-structure/

(I work at OpenAI, but I am not a lawyer and am not speaking on behalf of OpenAI - just sharing my personal understanding.)

Very cool! So glad to see people building and sharing evals that are better than SWE bench.

I'm curious - any particular reason you didn't put error bars on the graphs? Seems like it could be helpful when there are only 50 unique problems in the diamond set.

What do you mean by this? We don’t train on evals, and if we did I’d quit on the spot.

(The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve.)

Thanks - let me clarify that we don’t switch to lightly quantized models by time of day or when under heavy load either.

(I used the adjective heavily because that’s what the original post said. I have no intention of making misleading but technically true statements.)

For what it's worth, I work at OpenAI and I can guarantee you that we don't switch to heavily quantized models or otherwise nerf them when we're under high load. It's true that the product experience can change over time - we're frequently tweaking ChatGPT & Codex with the intention of making them better - but we don't pull any nefarious time-of-day shenanigans or similar. You should get what you pay for.

FYI, Elo isn't an acronym - it's a person's name. No need to capitalize it as ELO.

Very cool! The demos felt fairly contrived - e.g., count things while I talk. I wonder what more useful or commercial applications look like.

To clarify the title, Daybreak is not a new AI model or a new product. It's a rebranding of OpenAI for Cyber, which is an umbrella over multiple things that OpenAI is doing with companies.

Dec 2025, actually: https://developers.openai.com/api/docs/models/gpt-5.5

(though knowledge cutoffs in practice can be bit fuzzy)

Whether a problem is "good" or "bad" is not always objective or simple.

For example, you can have problems that are underspecified, with hardcoded tests for a particular solution (out of multiple possible solutions). If your solution works fine but used a different function name than the one hardcoded in the tests, you can unfairly score 0.

When an eval has underspecified problems like these, you can still score 100% if you remember the original solution from your training data or if you just have taste similar to the original human authors. And both of these qualities - good memory and good taste - are great, but they'll be rewarded unfairly relative to a model that still did exactly what it was asked but in a different way than the hardcoded tests expected.

We don't want hallucinations either, I promise you.

A few biased defenses:

- I'll note that this eval doesn't have web search enabled, but we train our models to use web search in ChatGPT, Codex, and our API. I'd be curious to see hallucination rates with web search on.

- This eval only measures binary attempted vs did not attempt, but doesn't really reward any sort of continuous hedging like "I think it's X, but to be honest I'm not sure."

- On the flip side, GPT-5.5 has the highest accuracy score.

- With any rate over 1% (whether 30% or 70%), you should be verifying anything important anyway.

- On our internal eval made from de-identified ChatGPT prompts that previously elicited hallucinations, we've actually been improving substantially from 5.2 to 5.4 to 5.5. So as always, progress depends on how you measure it.

- Models that ask more clarifying questions will do better on this eval, even if they are just as likely to hallucinate after the clarifying question.

Still, Anthropic has done a great job here and I hope we catch up to them on this eval in the future.

Honest answer is that it isn't done running yet. It takes some human bandwidth and time to run, so results weren't ready by this morning. We don't know what the score will be, but will probably go up on the leaderboard sometime soon. I personally don't put a lot of stock in the ARC-AGI evals, as it's not relevant to most work that people do, but should still be interesting to see as a measure of reasoning ability.

(I work at OpenAI.)

Agreed. Would be great if everyone starts reporting cost per task alongside eval scores, especially in a world where you can spend arbitrary test-time compute. This is one thing I like about the Artificial Analysis website - they include cost to run alongside their eval scores: https://artificialanalysis.ai/

Yep, it's more expensive per token.

However, I do want to emphasize that this is per token, not per task.

If we look at Opus 4.7, it uses smaller tokens (1-1.35x more than Opus 4.6) and it was also trained to think longer. https://www.anthropic.com/news/claude-opus-4-7

On the Artificial Analysis Intelligence Index eval for example, in order to hit a score of 57%, Opus 4.7 takes ~5x as many output tokens as GPT-5.5, which dwarfs the difference in per-token pricing.

The token differential varies a lot by task, so it's hard to give a reliable rule of thumb (I'm guessing it's usually going to be well below ~5x), but hope this shows that price per task is not a linear function of price per token, as different models use different token vocabularies and different amounts of tokens.

We have raised per-token prices for our last couple models, but we've also made them a lot more efficient for the same capability level.

(I work at OpenAI.)

Not this time, no.

Just as a heads up, even though GPT-5.5 is releasing today, the rollout in ChatGPT and Codex will be gradual over many hours so that we can make sure service remains stable for everyone (same as our previous launches). You may not see it right away, and if you don't, try again later in the day. We usually start with Pro/Enterprise accounts and then work our way down to Plus. We know it's slightly annoying to have to wait a random amount of time, but we do it this way to keep service maximally stable.

(I work at OpenAI.)

Meanwhile, everyone can read the news about layoffs attributed to AI and can see that hiring (especially of junior engineers) has slowed to a trickle.

According to FRED/Indeed[1], software job openings have been roughly flat for 2-3 years, and they've actually been slightly increasing again. What data source are you looking at?

[1] https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE

Following up - I was wrong about 10x/40x. Here's how it actually works:

$20 = 1x

$100 = 5X (but temporarily 10x for just Codex til May 31st)

$200 = 20x

We'll send out new tweets and clarify our pricing page.

I remember the gpt-5 benchmarks and how wildly inaccurate they were data-wise. Linking one[0] that I found so that other people can remember what I am talking about. I remember some data being completely misleading or some reaching more than 100% (iirc)

Yeah, I found that slide very embarrassing. It wasn't intentionally inaccurate or misleading - just a design error made right before we went live. All the numbers on that slide were correct, and there was no problem in terms of research accuracy or data handling or reward hacking. A single bar height had the wrong value, set to its neighbor. Back then, we in the research team would generate data and graphs, and then hand them off to a separate design team, who remade the graphs in our brand style. After the GPT-5 launch with multiple embarrassingly bad graphs, I wrote an internal library so that researchers could generate graphs in our brand style directly, without the handoff. Since then our graphs have been much better.

I don't think it's unfair to assume our sloppiness in graphs translates to sloppiness in eval results. But they are different groups of people working on different timelines, so I hope it's at least plausible that our numbers are pretty honest, even if our design process occasionally results in sloppy graphs.

Regarding the DoW deal, I don't want to comment too publicly. I also can't say anything with confidence, as I wasn't part of the deal in any way shape or form. My perception from what I have read and heard is that both Anthropic and OpenAI have good intentions, both have loosened their prior policies over time to allow usage by the US military, and both have red lines to prohibit abuse by the US military. One place they differ is in the mechanisms employed to enforce those red lines (e.g. usage policies vs refusals vs human oversight). Each company asserts their methods are stronger than the other's, so I think we have to make our own judgments there. Accounts from the parties involved in the negotiations also conflict, so I don't think anyone's account can be trusted 100%. With that caveat, I thought this article on the DoW's POV was interesting (seems to support the notion that the breakdown wasn't over differing red lines, especially since they almost managed to salvage the deal): https://www.piratewires.com/p/inside-pentagon-anthropic-deal...

Lastly, I hope it's obvious to everyone that Anthropic is not at all a supply chain risk and the threats there were incredibly disappointing. I support them 100% and I'm glad to see them unhurt by the empty threats.