criemen

Partially, 2.2 Submission workflow W2 deals with this:

Stage W2 The five project-active models, see Table 2, attempted the question. Their answers were compared to the original answer by an LLM judge. If at most three models answered correctly, the contributor could proceed.

So "trivially contained in the training data" is excluded, as then all models could/should easily come up with the solution.

Ah, reminds me of good old "There are only 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."

You quip, but LLM KV caching (from the harness side) is quite easy: You get a cache hit on stable prompt prefixes, period. That means you want to keep the prefix stable, and only append at the end of the conversation. Made up example: Don't put the git branch name into the system prompt part (that comes first), as whenever the branch name changes, that'd trigger a cache invalidation of the entire prompt.

Getting this right requires some care to not by accident modify the prefix, basically, and some design on communicating the things that can change (user configuration, working dir, git information, ...).

There's also a runaway effect of model improvement from the discovery, triage and fix data. This is likely already the most potent corpus of curated offensive data ever assembled and will only get better.

But that corpus of data is accessible to all competitors, American or not. I don't believe that this can't be replicated. I'd posit that there's enough annotated data out there (CVE+patch), only increasing thanks to Mythos, that if you specifically RL for this scenario, you can improve your models performance on finding vulnerabilities without access to Mythos.

Well is that a statement about the quality of Opus 4.7 or about compose 2.5? :P

Is there an equivalent ultra-high-end LLM you can have if you’re willing to pay? Or does it not exist because it would cost too much to train?

I guess at the time that was GPT-4.5. I don't think people used it a lot because it was crazy expensive, and not that much better than the rest of the crop.

it's them trying to push the models to burn less compute

I'm curious, how does using more tokens save compute?

Pretty cool if true!

What people don't realize is that cache is free

I'm incredibly salty about this - they're essentially monetizing intensely something that allows them to sell their inference at premium prices to more users - without any caching, they'd have much less capacity available.

I tried ChatGPT over the holidays (paid) vs. claude.ai (paid). After trying some prompts that worked well on Claude in ChatGPT, I understand why people are so annoyed about AI slop. The speech patterns in text output for ChatGPT are both obvious and annoying, and impossible to unsee when people use them in written communication.

Claude isn't without problems ("You're absolutely right"), but I feel that some of the perception there is around the limited set of phrases the coding agent uses regularly, and comes less from the multi-paragraph responses from the chatbot.

Out of curiosity, what are some good use cases for a MBP now with the MBAs being so powerful?

Local software development (node/TS). When opus-4.6-fast launched, it felt like some of the limiting factor in turnaround time moved from inference to the validation steps, i.e. execute tests, run linter, etc. Granted, that's with endpoint management slowing down I/O, and hopefully tsgo and some eslint replacement will speed things up significantly over there.

Even if was copying sqlite code over, wouldn't the ability to automatically rewrite sqlite in Rust be a valuable asset?

I had assumed that reasoning models should easily be able to answer this correctly.

I thought so too, yet Opus 4.6 with extended thinking (on claude.ai) gives me > Walk. At 50 meters you'd spend more time parking and maneuvering at the car wash than the walk itself takes. Drive the car over only if the wash requires the car to be there (like a drive-through wash), then walk home and back to pick it up.

which is still pretty bad.

This is amazing.

At $workplace, we have a script that extracts a toolchain from a GitHub actions windows runner, packages it up, stuffs it into git LFS, which is then pulled by bazel as C++ toolchain.

This is the more scalable way, and I assume it could still somewhat easily be integrated into a bazel build.

One other thing I'd assume Anthropic is doing is routing all fast requests to the latest-gen hardware. They most certainly have a diverse fleet of inference hardware (TPUs, GPUs of different generations), and fast will be only served by whatever is fastest, whereas the general inference workload will be more spread out.

aka: For those not living in 2026, we have uncovered a new clue to the mystery of where all the low-power DRAM chips have suddenly vanished to!

I love the writing style!

What's the power hookup to just boot one rack? I'd imagine that's more than you get anywhere in residential areas for a single house.

I’m currently using a mix of Zed, Sublime, and VS Code.

Can you elaborate on when you use which editor? I'd have imagined that there's value in learning and using one editor in-depth, instead of switching around based on use-case, so I'd love to learn more about your approach.

This takes a fairly large mindset shift, but the hard work is the conceptual thinking, not the typing.

But the hard work always was the conceptual thinking? At least at and beyond the Senior level, for me it was always the thinking that's the hard work, not converting the thoughts into code.

The large open-weights models aren't really usable for local running (even with current hardware), but multiple providers compete on running inference for you, so it's reasonable to assume that there is and will be a functioning marketplace.

It used to be that the answer was logprobs, but it seems that is no longer available.

Of course there is undefined behavior that isn't security critical.

But undefined behavior is literally introduced as "the compiler is allowed to do anything, including deleting all your files". Of course that's security critical by definition?

I've been thinking a lot about that lately, and I agree. I used to be hard in the "You can't solve social problems with technical solutions", but that's not the whole truth. If people aren't using your thing, sure, you can brand that as social problem (lack of buy-in on the process, people not being heard during rollout, ...). However one way of getting people to use your thing/process is to make it easier to use. Integrate it well into the workflow they're already familiar with, bring the tooling close, reduce friction, provide some extra value to your users with features etc. That's technical solutions, but if you choose them based on knowledge of the "social problem" they can be quite effective.

I've been thinking lately a lot about this. What is it I do when I want to convince someone of something (i.e. "creating alignment" in corporate speak)? I listen to them, am empathic, ask meaningful questions etc. Afterwards, that opens a space for me to make a proposal that is well-received.

Anthropic released the Opus 4.1 (basically, a new Opus 4 checkpoint) right around the big GPT-5 release date too, if I remember correctly. At this point, anything goes to stay relevant.

Assuming you earn enough, throw money at your problems. Many problems can be solved, or made significantly easier by money.

Don't do the research yourself, pay a specialist instead. Missing a winter wardrobe? Find a personal stylist (okay, requires some research to find, but much less than finding fitting wardrobe yourself). They will guide the shopping process, the outcome is most likely better than what you can come up with even with weeks of research. Costs a pretty penny, might feel funny, but it's effective.

Not happy with your art? Get a personal tutor for drawing. Learning a language, but struggling with it? Tutor it is. Learning is so much faster with personalized feedback and accountability, a good tutor provides both.

Why would someone in China not select Claude?

Because Anthropic doesn't provide services in China? See https://www.anthropic.com/supported-countries

Okay I see, that's a nice secure-by-default point, whereas TS is arguably not secure-by-default.

I don't know Rust, and I'm genuinely curious: How does it improve over that problem?

When you call a REST API (or SQL query for that matter), how does it ensure that the data coming back matches the types?

TS allows you to do parse the JSON, cast it into your target type, done (hiding correctness bugs, unless using runtime verification of the object shape, see sibling comment). Does Rust enforce this?

I'd expect at least API pricing for all major players to be margin-positive. Positive ROI if you include model training cost? Maybe not, but in a hypothetical scenario where all capital for new model training goes away, the latest frontier model weights still continue to exist.

Subscription prices for claude code et. al? No idea.

But why? Training hardware maybe will appear, but everything that's running inference has real, paying customers on it, with current capabilities level. Even if the bubble bursts, why would that demand evaporate? People still would want to use chatgpt, claude code etc., even if they stop getting any better tomorrow.