OpenAI reduces Codex Model Context Size from 372k to 272k

https://github.com/openai/codex/pull/33972/files

371 152 4 days ago

dannyw• 4 days ago

This was tweeted about when it happened, with some explanation from Tibo here: https://x.com/thsottiaux/status/2076543065045795309

mkl• 3 days ago

To see replies: https://xcancel.com/thsottiaux/status/2076543065045795309

The linked tweet is an unofficial reply to Tibo's official info and Tibo makes a correction in a reply.

imgyuri• 3 days ago

How can the overall trajectory length be the same across reasoning efforts? I don't see how this is possible even if reasoning is not included in the trajectory length calculation.

chaos_emergent2• 3 days ago

I think Tibo was just keeping all else fixed and it’s an illustrative example rather than a perfect real-world trajectory.

causal• 3 days ago

Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?

ph4rsikal• 3 days ago

The blue line (200K context) is lower than the red line (300K context). Indicating that it's cheaper to run blue rather than red over longer cycles.

chaos_emergent2• 3 days ago

The chart makes sense and is describing the cumulative cost of a trajectory. Cache tokens are created when a trajectory’s prefix is used more than once. A larger pre-compaction context window means that a greater number of cache tokens are used per turn, and a larger number of turns are completed before compaction runs. So you get a cumulative cost that grows quadratically until the compaction event.

crazylogger• 3 days ago

The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.

trilogic3• 4 days ago

HugstonOne increased coding context size, from 1 to 4 Million ctx

cyb3ralbert• 3 days ago

Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, and most sessions probably don't need anywhere near 372k tokens anyway. Curious if this affects people who were actually relying on the larger window for big codebases.

mikhailt2• 3 days ago

We'll have to wait because Tibo (OpenAI) said this was a temporary measure and it'd go back to 372k soon.

The issue was more specific to higher token burn rates, not latency.

bel8• 3 days ago

That's quite small for my workloads.

I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact.

Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?

SwellJoe• 3 days ago

Nobody does caching as well as DeepSeek, so I guess it's a big enough difference in the implementation to make it difficult.

If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.

cyanydeez• 3 days ago

using open models and local AI via llamacpp, I'm compacting (via agent directed) between 55k-85k; going up to 120k is rare unless the agent is actually following a non trivial log tracing or some other thing that absolutely requires keeping large context.

I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.

It mostly works to allow iteration into multiple sub components.

d4rkp4ttern• 3 days ago

The other day I was trying to find out exactly how much context size 5.6 has in codex CLI (via subscription) and I didn’t find that documented in any of the obvious places. Sort of ridiculous that we have to find this out in X or Reddit, e.g:

https://www.reddit.com/r/codex/s/hVv29obfFD

tekacs8• 3 days ago

You can run ’codex debug models' into jq!

I forget whether it's documented or not, and it is kind of annoying that that's how you find it, but it does tell you. Maybe useful for the future if nothing else.

spongebobstoes• 3 days ago

it's in the GitHub repo

davidkuennen• 3 days ago

I never felt that context size was any issue at all in codex. I don't know how their compaction works, but it just keeps on going as if it has no context limit at all. At least in my experience.

embedding-shape2• 3 days ago

I never felt that context size was any issue at all in codex.

I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago.

Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface what goes into the "concise summary" afterwards, make it really hard to know if it actually got everything important or not.

In general, codex seems to be moving into the way of hiding as much from the user as possible, and it wouldn't surprise me if they eventually start encrypting the entire session logs just like they did with agent>sub-agent prompts recently. Sad though as it's easily the best harness+model combo available today out of the ones I've tried.

cmrdporcupine2• 3 days ago

Claude also had issues with this kind of thing for months early on, too, where it simply couldn't compact its own emissions.

But definitely earlier GPT models suffered greatly as context got large, and the compaction itself in Codex was really crappy. That changed around January.

mikhailt2• 3 days ago

That has nothing to do with context window or the models at all; it's the compute capacity.

Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.

Claude had the same issues when they released their newer models.

throwuxiytayq• 3 days ago

For me, Codex regularly forgets to complete its last task when compaction occurs, especially when the last message I sent was right before compaction.

dbbk• 3 days ago

It seems odd to me that Codex doesn't carry the plan file through context compaction? Claude does this, it re-reads it in full from disk

efromvt• 3 days ago

I’ve noticed it having weird message dropping and replay in general, but the compaction boundary has been pretty solid.

formerly_proven• 3 days ago

Most problems should be divide-and-conquerable into chunks that 300 vs 400k context is basically never an issue. A coding agent isn't an infinite chat.

simonw2• 3 days ago

Another interesting change in that commit is the addition of this section to the system prompt:

  Before taking a destructive action:
  
  - Make sure the action is clearly within
    the user's request.
  - Resolve the exact targets with
    read-only checks when necessary.
  - Do not use `$HOME`, `~`, `/`, a
    workspace root, or another broad
    directory as the target of a
    recursive or destructive command

Looks like the fix for this bug where Codex would occasionally accidentally delete your entire home directory: https://twitter.com/thsottiaux/status/2077630111499882637

AmazingTurtle• 3 days ago

This happened to me one time. On windows though, deleting C:/ lmao. The direction I'm heading now is a better harness, i.e. isolating codex at a container level with dedicated workspaces / mounts etc. I'm building something at the moment that serves my needs.

Razengan• 3 days ago

That's ok, if AI deletes my home directory I can just ask AI to regenerate my home directory.

tekacs8• 3 days ago

https://github.com/tekacs/fast-rm

I've overridden my rm with this, which I threw together for fast-deletes of things like Rust target/ directories, and after seeing the GPT horror story, I taught it to flatly reject deletions directly under `/` and under home directories, with a message printing the path that it's trying to delete.

Not exactly a perfect mitigation, but given that the stated risk was the model mistakenly using the wrong $HOME, it seems like a reasonable safety. I should probably make it use an even scarier rejection notice, though.

I also... have backups.

skerit• 3 days ago

No matter how good compaction is, on some big projects it needs to read a lot of files. In my experience the first 200.000 tokens go FAST, but after that it slows down. Most of my Fable sessions don't go over 500.000 tokens, I don't need to compact once. But when I use Codex a single session has to compact over and over again.

HDThoreaun2• 3 days ago

In my experience the only reason it needs to read a lot of files is if your agents.md is deficient. It should only need to read the file it's actually working on, maybe a few others. Everything else should be explained in documentation

ashu1461• 3 days ago

I am wondering why the codex repo is open source ? Can they afford it to be open source in the longer run ?

simonw2• 3 days ago

Not much point making it closed source at this point, coding agent harnesses are trivial to reverse-engineer.

Being open source was also a useful community lever they could pull when they were trying to catch up with Claude Code.

CamperBob22• 3 days ago

Not much point making anything closed source at this point, if the client has access to the binary.

Topfi2• 3 days ago

What do you mean “afford”?

Topfi2• 3 days ago

Good move given some experienced issues and compaction across the 5.6 range is closer to 5.4 than 5.5, i.e solid and reliable.

Will say that 5.6-Sol is a minor bump in my benchmarks in most areas vs 5.5 but a severe regression in a few specific task focused on rearranging trees, addressing merge conflicts, etc. where the model to accomplish the task does not properly adhere to prompts in a way GPT-5 originally managed, not retaining parts of history in the way prompted despite specific instructions not to as that made the final completion easier…

I am of the conservative and cautious opinion that no model should be able to run destructive tasks at all, I have seen every model do things that make me concerned enough to maintain that opinion and know my evals can’t catch everything. But for 5.6-Sol specifically, I’d caution everyone to reevaluate how you run the model, maybe take a few more precautions you tend to forgo.

It is extremely capable as a reviewer and for extensive tasks, though for the later, the safety net I feel is required to be comfortable limits the utility. The code 5.6-Sol provides also still is a bit harder to parse in reviews.

Release strategy wise, feel it’s have been smarter to release only Luna and Sol now, then Terra a few weeks of posttraining later, I simply cannot see a purpose for it in the current form given how well both Luna and Sol scale up and down respectively with reasoning. Two models from a lab at a time is also the limit I feel one can properly assess at a time.

damsta• 3 days ago

Not a fan of their context compaction and I feel like 1M-token context should minimum today. Each day I see how GPT 5.5 and 5.6 struggle a bit after each compaction before they get to the full speed, sometimes focusing too much on some older steering message that made it into the compacted context.

iugtmkbdfil834• 3 days ago

Full agreement. I sincerely hope that reducing token is mostly a cost saving measure and not some roundabout way to get more token use. It still does not help. In my corp, bean counters rendered initially useful internal llm largely pointless as they severely limited context.

I swear there is a executive convention out there sharing worst practices.

Sammi2• 3 days ago

Just use md files for working memory. No need for large context. LLMs get dumber as you put more in context, as it stretches their attention. Keeping context small is better for quality.

jubilanti2• 3 days ago

The fact there is no way to disable auto-compaction like every other coding harness makes me scream. It fires randomly at 10-20%, so really you now only have guaranteed context of 272k * 80%. So many times I've had it work on a large codebase, it is almost done with a task, it needs maybe a 2000 token response, but it drops below 20%, it churns and churns and churns and then I see the dreaded ``Context compacted`` and scream. You can't go back in the history to the state of the conversation before the compaction. So then it has to investigate the codebase again gets to 20% and then auto-compacts. So it then has to read more, and then auto-compacts again, and then I'm out of tokens.

n4te• 3 days ago

Use pi. Can't trust OAI or Anthropic to make good tools.

bredren• 3 days ago

Pardon the plug, but I have built a tool that ingests the state of conversations to a local DB in realtime. It has both macOS and Linux clients.

When context compaction introduces a gap, I use the /total-recall skill to pull prior turns back into context and off it goes.

The tool is free for personal use and has a source available local cloud option to sync convo histories across multiple machines.

http://contextify.sh/docs

rbranson2• 3 days ago

Note that GPT-5.6-Sol is ~2x as token efficient as Opus/Fable, so a 258K max is equivalent to ~516K on Claude models. You also may just be fooling yourself, as context rot is still a problem[1][2] and there's evidence compaction can be equivalent or better than longer context windows for agentic tasks[3]. Obviously it would be best if the models could reason over 1M context like they do over 256K context, but they can't yet.

[1] https://arxiv.org/abs/2605.12366 [2] see F1 of GraphWalks 256K vs 1M in Opus 4.8 System Card: https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80f... [3] https://context-folding.github.io/

bigmadshoe• 3 days ago

The improved token efficiency would only account for the output and reasoning tokens occupying context, right? What about the user messages and other context like file contents? I’m not sure how these measure up in a long conversation

amelius• 3 days ago

"272k ought to be enough for anybody"

trilogic3• 3 days ago

Thats not even enough to read a simple codebase, how is that enough?

sokoloff• 3 days ago

Almost certainly an homage to “640kB ought to be enough for anybody.”

https://quoteinvestigator.com/2011/09/08/640k-enough/

tekacs8• 3 days ago

I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately.

Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine?

The lack of long context is the main reason that I still end up using Anthropic.

The worst is when you need it to hold for example a number of papers in its head, or large and complex materials that it needs full resolution on and your context window ends up being perennially at 16%. You have about five minutes of conversation and it compacts and then you have to wait for it to read that again, get to 16%... and repeat.

372 was not perfect, but it was so much better and a godsend. It turned that 12 to 20% into more like 40%.

planckscnst3• 3 days ago

I agree. Compaction sucks, so I made tools that let the LLM selectively delete (and recall if needed) chunks of its context. You might want to try context bonsai if you're routinely hitting the auto-compaction wall.

https://github.com/Vibecodelicious/context-bonsai-agents

tekacs8• 3 days ago

I've implemented a similar approach – although I'm surprised not to see mention of cache prefix busting in there!

znnajdla• 3 days ago

I have a hunch that OpenAIs proprietary compaction endpoint actually does something similar on the backend.

adyavanapalli• 3 days ago

This is actually brilliant. I've seen other approaches where the context is selectively pruned, but never in a recoverable way.

P.S. Love the "bonsai" name btw!

thih92• 3 days ago

What plan are you on?

With Anthropic I run out of quota very fast on a Pro plan with long contexts.

Is long context primarily for Max tier or for API usage? Or are there special ways of working with long context on a Pro plan?

tekacs8• 3 days ago

Yep, Max. I would definitely struggle to code with Anthropic's plans on Pro.

skybrian2• 3 days ago

That seems quite different from my design process. I write a plan.md that goes through multiple revisions. The plan is the memory. Restarting and reading the plan again to do another review is a good way to get a different perspective.

tekacs8• 3 days ago

I have these plan files as well, but it depends on the scope and scale of the things you're executing on, I think. However much detail gets put into the plan, it still doesn't help if part of what the model needs to understand is the fine-grained / perfect detail of a large surface area.

Sammi2• 3 days ago

So far I have three different folders in my project for three different types of md documents that I keep creating and revising together with the agent: reports, plans, and reviews

Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.

This is my super simple worse-is-better development process. I make up what is needed on the spot.

jmalicki5• 3 days ago

I like to actually have hierarchical plans.

Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.

That document gets saved, and has the big picture context.

Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.

The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.

Then have a DAG of stacked pull requests designed for subagents to work through.

All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.

BoorishBears• 2 days ago

I've found if you pay close attention, models in Codex get briefly lost on where it is in the plan post-compaction. If it was in the middle of a test it often tries to resume the test from the start, or will seem "surprised" that past steps are completed already.

It's hard to believe that sort of confusion doesn't hurt performance a bit, the question is if that degradation is worse than the performance fall off from long-context (which is highly task specific)

dnautics• 1 day ago

I write a plan.md that goes through multiple revisions

instrumentally, isn't this the same as just doing plan mode?

Plan mode is just a .md file in a 'well-known' location that is privileged to a certain spot in the kv cache (after the system prompt).

eagerpace2• 3 days ago

The best way around this for me is planning a project well ahead of time and using subagents to execute individual tasks while the primary agent acts as a PM.

whearyou• 3 days ago

I haven’t found I need to plan way too far ahead of time as long as I’ve defined the interface between the tasks

tekacs8• 3 days ago

That isn't really a way around it for the kinds of things I'm talking about. The whole point is that getting all of the relevant context to even orchestrate it takes up most of the model's 272K.

troupo• 3 days ago

The lack of long context is the main reason that I still end up using Anthropic.

Anthropic's "long context" is mostly marketing bullshit. It semi-reliably holds about the same amount of info, and then starts suffering the same issues.

nojito• 3 days ago

Why arent you using /goal?

jmalicki5• 3 days ago

Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.

thejazzman• 3 days ago

This used to be really effective and cheap, now it lights my quota on fire, even after I fix the config and restore smaller subagents instead of like-agents

Multiple (slow) conversations seems to be the efficient path for me the past week or so

ljm• 3 days ago

How does that work out in terms of usage? I imagine that, unless you're on the highest tier subscription or have a no expense spared approach to agentic coding, then you're going to hit a wall pretty fast.

azuanrb• 3 days ago

Curious, what kinds of tasks do you do that require such a large context window? Anything specific?

serf2• 3 days ago

for me it's often codebase decomposition.

it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.

so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.

yearolinuxdsktp2• 3 days ago

If you’re using the pi.dev harness, I found this tool to be quite good as a compaction alternative: https://pi.dev/packages/pi-blackhole — it keeps a memory so every prior conversation piece is recallable, not lost.

scotty79• 3 days ago

OpenAI compaction is very different than Anthropic compaction, especially in 5.6

I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.

serf2• 3 days ago

I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.

that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.

but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.

large context limits is one of the things that make the open model competitors very attractive to me.

jubilanti2• 3 days ago

The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me on any codebase > 5kloc. Even worse it seems to fire randomly at 10-20% of context left. So really you have 80% of 272k as real usable context. Compaction kills my sessions, it hallucinates and is worse than starting fresh. I've had enough times screaming at my computer when it burns tokens on a large codebase, gets to 15%, auto-compacts, and hallucinates so bad it has to read the entire codebase agin, gets to 15%, auto-compacts....

volker48• 3 days ago

Have you tried using another harness like Pi? You can easily turn off auto compaction and if you leave it on you can use /tree to go back before the compaction if you like.

mycall• 3 days ago

So the practical workaround is to set model_auto_compact_token_limit very near, but still below, the model’s context window. This postpones compaction rather than disabling it.

config.toml

  model_context_window = 27000
  model_auto_compact_token_limit = 17000
  model_auto_compact_token_limit_scope = "total"

behnamoh• 3 days ago

The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me

Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.

ethin• 3 days ago

I suffered this same problem with Anthropic, ironically. Mainly with reverse engineering things. That would usually end up requiring a ton of context it had to remember, and then compaction would kick in and it would just forget half of it (that half being really important too), and even switching to Claude Code and having it document everything wouldn't really fix the problem. It was honestly quite sad because I've usually gotten better results with it, but it falls a part the moment you try doing something like that that's really complicated and requires a ton of different attempts to get it right.

QuantumGood• 3 days ago

Compaction is a feature like swapping out a car for a tricycle is a feature for greater economy.

reinitctxoffset• 3 days ago

You can mad customize it if you're willing to set up like, oh my pi or whatever.

I didn't quite get it tuned up enough to be a daily driver but I'm basically sure it can be hotrodded however you want. It comes out of the box with like four different vendor hostile compact strategies, including compacting into images at the smallest size Claude can read, which the "coding is basically solved" geniuses conveniently leak the resolution heuristic out of their website along with all the other side channels they print for the MSS.

MattRix• 3 days ago

If you want stuff to be referenced regularly, why are you doing it in the conversation instead of just putting the documents in the directory and referring to them in AGENTS.md or whatever?

rcarmo• 3 days ago

Depends on what you are using. I have been experimenting with various compaction techniques in piclaw (https://github.com/rcarmo/piclaw/blob/main/docs/pipelined-co...) and Codex has native "opaque" server-side compaction at a dedicated OpenAI endpoint.

voxgen• 3 days ago

Pay attention to what it forgets, and start telling it to proactively note down those things into nominated files while it works, e.g. indexing topics covered by each chapter/section/page of each paper/material. GPT-5.6 is really good at finding context again, if given the smallest hint where to start looking.

You can also ask it to read your past sessions, find places where it was wrong after compaction, and figure out a strategy to persist the most important summary information.

I have several Hermes threads that have each had >10 compactions with a 200k context limit, and with the right instructions on note-taking, they require at most a "double-check that against our past decision records" to be put back on track after a slip-up.

cortesoft• 3 days ago

The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.

CamperBob22• 3 days ago

Exactly. Just about every prompt I issue these days generates or edits an .md file.

HDThoreaun2• 3 days ago

Yea by the time you get to 300k context its IQ is a solid standard deviation lower anyway, why would you really want to beyond that?

keeganpoppen• 3 days ago

sometimes compaction is a death knell and other times it's totally fine. it really just depends on whether the thing you were doing at the very end has high correlation with stuff from early on in the session, or if there was an obvious ~break s.t. you were just on some specific subtask. the situation has clearly gotten a lot better than even 6 months ago-- i think openai handles compaction better in my own experience. but it's exactly the kind of thing that yegge was trying to get at with gastown-- i used to run every session s.t. i could re-animate it every time the compaction boundary came... now, when the compaction boundary comes i am so pot committed that i just hold my breath.

sidewndr46• 3 days ago

I think I've seen the opposite end of this as well: I get the most value from any LLM when I ask it one very targeted question with a followup. I don't fall outside of any sort of context window sizing when I am doing that. That isn't to say that I don't have a few long engineering discussions that are helpful, but most are very to the point. It helps that I have decades of experience at this point and effectively know what to ask to narrow the scope quickly.

matheusmoreira• 3 days ago

What. Anthropic's models have million token context windows! I'm switching to OpenAI next month, I can't believe they're still stuck on ~300k... Better start adapting to the new reality I guess.

wwind123• 3 days ago

A typical suggestion is to make the agent create or update a .md file once in a while, so the agent would remember the new important things that have shown up since the last update. The problem is, how would the agent know which things are really important. If it knows that well, then /compact should work well too...

PeterStuer• 3 days ago

All depends on context coherence. E.g. in Claude Code with the "1M context" models I am reluctant to push past 30%. So if that Codex 272K is 100% coherent than it is less of a difference. Still, even then the "hard fail" boundary has moved. Longer running agentic processes will surely run into this more often.

cmrdporcupine2• 3 days ago

Codex compaction is really quite good. Smaller context doesn't really harm me.

I had a /goal running last night for 9.5 hours straight while I slept. When I woke up in the morning it was fully on task and focused.

Write up a detailed design doc. Build a decent AGENTS.md, and write up a good prompt or /goal.

Long context can be more of a curse than a benefit sometimes anyways.

maherbeg2• 2 days ago

Yeah, agreed. It makes everything else look pretty bad. I still like to manage my work in a more structured form, but Codex can just rip on a goal end to end in the same thread.

weeksie• 3 days ago

You just need good compaction and a memory system with decent project layout and documentation. If you don't have that 1M tokens won't really save you either.

Topology1• 3 days ago

Looks like we will be waiting quite a while to see a 1M context window from OpenAI

miranaproarrow• 3 days ago

claudes/codex performance suffers after 250k anyway and my workflow reset after every 200 so this is a non-issue

fandorin2• 3 days ago

Does the context size really matter that much if a “dumb zone” starts around 120k-150k anyway?

conradkay2• 3 days ago

Things change fast! For Fable 5 it definitely feels past at least 272k

fandorin2• 3 days ago

hmm, what about other models?

wonnage2• 3 days ago

Not sure if this is still up to date (2023), but https://arxiv.org/abs/2307.03172 shows that performance degrades mostly in the middle of the context.

Anecdotally I've been stuck in that situation of being at 400-500k tokens and "just one more prompt bro" will get the task done, and I appreciate not having to wait through a compaction. If anything, keeping the bloated context helps with accuracy at the expense of speed in these cases.

aurbxyajwur• 3 days ago

5.6 is a mess

gurjeet• 3 days ago

Please elaborate.

Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.

Sincerely, HN Guidelines Police :-)

https://news.ycombinator.com/newsguidelines.html

hakanderyal2• 3 days ago

I have rule files that guides the agent towards my coding standards, code style, house rules etc. They alone cost 60-80k tokens, and they are the backbone of my system that prevents slop. Pre 1M context, I had to build complicated tooling to re-include the relevant docs to the context upon compaction, which relied on unstable transcription file format, which was a pain to maintain. With 1M context I deleted all of those. Nowadays most of my sessions uses 300-450k context.

Another thing that's preventing me from trying Codex. (the other is @ referencing files not auto including them to the context)

1M should be table stakes for frontier models at this point for programming.

yearolinuxdsktp2• 3 days ago

Try running your rule files through an LLM for optimization. 60k-80k tokens is massive.

Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs.

I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better.

I had one critical rule I was maintaining about searching the codebase using a structural index/graph and not grep. Every time the agent missed it, I asked it how to improve the rules. Eventually, I asked the AI to review that rule file and it rewrote it to be 30% smaller, but, crucially, structured to be more understandable by the LLM.

Another helpful thing was to ask AI to review my rules for things it can load on-demand when it works in that area.

hakanderyal2• 3 days ago

It’s optimized to death. Every word, every comma. I have a rule file for writing rule files.

Problem stems from using my own abstractions instead of common frameworks.

For a full stack session, backend + front end + docs about the part we are working on adds to that amount.

I’ve accepted it because it results in code exactly the style I would’ve written. It’s a good tradeoff to cut off the slop.

wonnage2• 3 days ago

AI cannot tell you why it did something. If you ask it why it didn't do X even though your prompt said so, it can come up with some plausible sounding explanation, which might even be correct, but there's a fundamental impedance mismatch between giving you the most plausible sounding answer and actually grounding that in knowledge of how it generated that answer.

TBF having a 60k personal style guide is also a bad idea, people get borderline AI-psychosed about how their special prompts are really steering the AI when in reality there is no way to evaluate this stuff. You invariably end up trying to define your style in terms of what it is and it isn't, it's like trying to define "red". "follow the style of my 5 most recent PRs" works pretty well.

brandoncarl• 3 days ago

With purely quadratic attention, the cost of the token at 372K is 87% more than the token at 272K.

There are attention mechanisms that help to mitigate this, but you can clearly see in the chart that Tibo posted that the attention mechanisms they are using are still scaling quadratically.

This becomes an optimization problem for token t:

min f[cost, quality loss] = compaction cost[t] + compaction quality loss[t] + token cost[t]

conradkay2• 3 days ago

It's not quadratic attention, you get that curve from the input tokens going up linearly, since the graph is measuring cumulative cost at each token count. Basically for y=5 it's 5+4+3+2+1, or f(x) = x(x+1)/2

https://pbs.twimg.com/media/HNFc4Dma8AA76FW.jpg?name=orig

salamo• 3 days ago

On the one hand, compacting at 300k saves OpenAI 40%. That's great.

On the other hand, $10 for 1M tokens still seems really high? It's not too hard to blow through that in an hour or two.

monster_truck• 3 days ago

If ChatGPT 5.6 is so good why do they continue to have the same motherfucking cache and usage issues that they were struggling with when 5.2 is new? Could not pay me to use this shit. Enjoy your reset bingo

quotemstr2• 3 days ago

Codex is open source. You can build it yourself. You don't have to use upstream's parameter choices, compaction strategies, subagent heuristics, or whatever. Strange how people act like tweaks to these parameters are mandates when they're really just suggestions relayed to you via git.

For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you go to the /model menu. I didn't like this change, so I undid it locally. Easy peasy.

noname1202• 3 days ago

Not true. Try to increase the context window and you will be greeted with an error when you reach the real limit enforced server-side.

quotemstr2• 3 days ago

Perhaps not this for this parameter, but for others? Client side.

searealist• 3 days ago

The models themselves support up to 1M. You are just charged more for all context over 400k. The 272k limit is just a client limitation to ensure you can never go over 400k since the maximum output size of the model is 128k.

KronisLV• 3 days ago

On long sessions even with workflows and sub-agents I typically hit around 500-750k tokens in context (say 4-16 hours of work in a session, across multiple 5-hour limits) with Claude Code.

I was going to explore paying for Codex since OpenAI seems to be a bit more generous with rate limits, but I'm now not sure - for the stuff I do limited context size would be a dealbreaker - basically lots and lots of documentation and guideline references, code review loops, documentation references and web searches, multi-repo exploration, lots of tool calls and so on.

I might just upgrade to Anthropic's more expensive Max subscription or something to get more subsidized tokens. Even with plan files and the plan mode, it's like a slot machine after compacting the context, sometimes steps or other details just evaporate in thin air, the less of that I need to do the better. It's not that it can't work, it just doesn't work reliably enough not to be annoying.

Luckily DeepSeek V4 Pro, GLM 5.2 and Kimi K3 don't seem to have those limits either - though DS is around Sonnet, GLM 5.2 feels a bit above Sonnet and only K3 only really is in the proper Opus ballpark that's good enough for me to work undisturbed, even if it seems to be slower.

throwatdem12311• 3 days ago

I primarily use Opus for my day to day and I /clear often. 1M context sound great but it quickly degrades once you start hitting the 50% mark. I usually just /clear once I hit between 30 and 40 percent context and get much better results from the model. Compaction doesn’t really help much in my experience, starting fresh and having the model front load its context from scratch works much better IME. I have multiple skills with a pile of markdown docs organized by feature to assist in that first load by telling it where to find relevant information for the task. Works great.

onetrickwolf• 3 days ago

I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway.

I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.

It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.

enraged_camel• 3 days ago

> I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is.

It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.

My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.

andix2• 3 days ago

Exactly my experience. I would even draw the border at 100-150k. Long contexts would be nice, but they just don't work well, even if the models support them.

Havoc• 3 days ago

Yeah I also try to compact/restart at 250k. I noticed though that it tracks with how big the project is so I guess the people needing more are just working on bigger projects

smcleod• 3 days ago

I haven't used Codex in some time, but in Claude Code I force my context size down to 280k on purpose as in my experience your best work is done within the 30-250k~ range regardless of a models maximum context size.

coderenegade• 3 days ago

This is a shame, and hopefully it reverts soon. I switched from Claude to Codex just before the Fable release because Claude doesn't follow instructions and goes off the ranch too frequently. I really don't want to have to switch back to what I consider to be a fundamentally worse model.

kccqzy• 3 days ago

Even Fable doesn't fully follow instructions.

andix2• 3 days ago

I didn't try long coding sessions for a while. I had really noticeable performance degradation with all models above 100-150k and always try to stay below those numbers.

hendersoon• 3 days ago

This was done because they were charging the 100%/50% premium for long-context over 272k to Codex users, even though they set 372k as the default. It cost people a LOT of money.

Fr4nZ82• 3 days ago

Big context windows made everyone lazy about deciding what actually deserves to be in there... compaction makes it worse, it's lossy compression applied to everything at once so the details you actually needed probably get lost. I always thought that retransmitting the full chat every turn is a foundamental problem... in my experience with conversational agents I clear the context every turn and reinject all the relevant info with a memory/context plugin I built myself, so small context stopped being a problem cos the model reads a few k of curated state instead of 200k of scrollback. Coding agents are harder and I haven't cracked that part, but in my opinion the real fix to the context problem is in this direction: a policy that keeps what's needed to finish the job or to do the next job and drops the rest. This can be achieved with a separate LLM specialized for exactly that.

paytonjjones• 3 days ago

I always thought that retransmitting the full chat every turn is a foundamental problem

So did many NLP experts who worked on LSTM, GRU, etc. But transformers empirically won out and here we are.

It will be interesting to see if future model architectures need to take this into account again.

If humans are any indication, one thing we're still missing is efficient movement of information from short term to long term memory. Fine tuning in principle does something akin to this, but not efficiently.

yalogin• 3 days ago

I have over the last two months switched to OpenAI from Claude since it’s been vastly better for my cases. Wonder if I see any difference in the quality of the output with this change

majestic5762• 2 days ago

Just use GPT-5.6 Model Router from the Codex plugins

OpenAI reduces Codex Model Context Size from 372k to 272k

Related Stories