orbital-decay

They ran a deliberate prompt to test its vulnerability search and exploit writing capabilities, with all safeguards intentionally disabled, it was literally free-for-all. Escaping the containment means nothing if you never define the containment boundaries. Side effects means nothing if you give the model carte blanche and never tell it what path to avoid.

Distillation is a superficial step and doesn't need a lot of data, it's not "stealing the model" like they want everyone to believe. 99% of work is already done by that point. That said, it's pretty clear K3 has Claude's data in the training set (either Opus or Fable), as it repeats Anthropic's prompt injections. (not that it matters to anyone besides Anthropic themselves)

Current post-trained models are stochastically and semantically collapsed to a large extent, you'll never get meaningful semantic diversity if your prompt stays the same. This makes the temperature brittle, the model becomes dumb much earlier than meaningfully diverse. There are fancy samplers that claim to overcome this (comments ITT mentioned them), but a) you need a local inference stack to use them and b) I've yet to see any definitive evidence they add any deep semantic diversity without dumbing the model down, most of what I've seen is pretty superficial, including distribution-aware samplers.

I'm currently adding random variation to prompts

This is better than naive temperature tweaks, if done right.

You probably don't want temp 0, especially with Gemini which often fails in the greedy sampling mode in practice, often in ridiculous ways (e.g. multiple thousand token loops). As another comment says it's pretty brittle and this is even reflected in their docs somewhere IIRC. You want the "normal" temperature (whatever it is) and oversampling if necessary.

Certain open models have/had the temperature locked on the official APIs, I assume they just have sampling incompatible with static temperature or do some fancy speculative decoding. It's clearly not to hide anything, as the weights are open and there are always alternative providers.

That's an accidental CoT language leak, it might happen. If it does this consistently, something is up with your prompt

All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

ExploitGym is a literal exploit dev benchmark. As always, the entire event looks a lot more like "the model did what we prompted it to do" than "it decided to do this spontaneously on its own".

People are saying from the beginning that Sam and Dario are way more dangerous than their models and the others dismiss it. What would change your mind on this?

Yeah. They and Altman in particular did a ton of shady stuff during the last peak of hype around open models: accusations that turned out to be outright made up (there's zero chance R1 ever distilled their model), obvious coordinated media distractions, alignment scaremongering, even possible DDoS and hacking attempts against DS (see the Xlab report everyone ignored), all of which magically disappeared once the hype died a bit later as OAI hastily released their next model.

Their alignment is under suspicion a lot more than their model's.

Once the US implements meaningful export controls, China will do this as well. They're already mirroring US regulations, but the gates aren't closed yet.

The model is trained to follow the default template pretty closely, breaking it usually results in much worse performance and better output variance. Certain models with synthetic data in pre-training can melt down completely. At some point into this breakage you can just take the base model and it will be better.

If you want to use a custom chat scheme, use it as an overlay, don't break the default chat/tool use/reasoning template.

If you compare 3DSMax 1995 to Blender, with few execptions, you'll find them very recognizable.

Workflows from 1995 have basically nothing in common with modern ones, everything is completely different and specialized in a lot of ways.

No, Microsoft inflicted this upon themselves. They introduced driver vetoing long ago specifically for this purpose (at the cost of certain amount of user freedom), and have established the criteria for it, but barely ever enforce most of these criteria that aren't related to hard OS stability (and sometimes even those that are related), because that might upset the vendors.

DA has probably the most widely cloned shape in the market and has many identical or compatible alternatives and clones, check e.g. MCHOSE A7 Ultra RE or Pulsar Xlite v4 Large (never had either so can't vouch for their software)

https://www.eloshapes.com/mouse/compare?p=razer-deathadder-v...

https://www.rtings.com/mouse/tools/3d-model-shape-compare/3d...

Giving the data to Anthropic instead of giving it to LG and McAffee, fascinating

Some Razer mice were somewhat good in the times when the sensor mattered, with the rest of them being absolute garbage (starting with the Copperhead which barely worked). Today there's a ton of niche manufacturers with great internals that will exceed any requirements you might possibly have.

If you're really interested in FPS performance and not just the brand, choose for the ergonomics first, it's not possible to recommend anything without knowing your play style, hand size etc. The shape and weight you like, and complementary feet and mat with the exact static/dynamic friction you need. Then check if the internals are good enough (they likely are) and whether there are any firmware issues like extra jitter on flicks or unavoidable debounce lag, then look at the required software. There's a ton of mice with excellent performance that are configurable without ANY software.

This situation has.. no precedent as far as I can tell..

Printer, mouse, tablet and display tablet makers use this to insert their crapware since at least Windows Vista or Windows 7, I think. The last one I remember is plugging a Razer mouse just to watch it instantly pulling 1.5GB of bloated junk with "telemetry" exfiltrating the data from my gaming PC in realtime. At least it doesn't leave my mouse in a non-working state when I disconnect the internet, like it used to. Thanks, Razer!

Microsoft is to blame here, really. They have a mechanism to block any vendor (supposedly to avoid reputational risks to their brand due to buggy drivers, at least that was their excuse back in the day), but aren't even using it to block these contraptions. Entire businesses are built on this, e.g. Razer is probably more of a marketing/data company now rather than a hardware shop.

Tools are already preventing IoT companies from doing a ton of things they do, by default. It's a problem of the process, churn, and culture, not tools. I don't have any doubts that if given a coding agent that cares more than they do they'd still force it to hardcode a password or something because they feel like it's more convenient. Nobody cares there.

Only on GPT (never was) and more recently Claude, and it still can be simulated with structured outputs for the purpose above.

Even if it's not supported somewhere (e.g. z.ai API which isn't mature enough and has neither assistant prefills nor actual structured outputs), it's still better and more seamless than using the default user/assistant scaffolding for role alternation.

Can you elaborate? Where does the structure come from in this case? The model can't even see the boundaries of the reply (it's started with a prefill and stopped with a custom stopping string).

You have a point, this might help test the resistance to jailbreaks for example. There are probably better tests for that than OOC roleplaying, I think. At the same time, what you expect is likely not going to happen due to many reasons (e.g. can't optimize for two contradicting objectives), and harness design offers a practical workaround that is being used for years already and is reliable unless the user is actively trying to jailbreak the model+harness.

This is a known and solved problem. Such a test is pointless for a general-purpose model, because like most people you're using multiturn chats in a naive way, fighting the default finetuning that is done intentionally.

1. You're sending your in-character inputs to an instruction-tuned model under the user role, in a multiturn chat. It's biased to treat these inputs as instructions and this behavior will show itself no matter what. Besides, the rigid structure of the assistant persona reply (usually tl;dr - explanation - "would you like to know more") is going to leak into such roleplay no matter what. To solve this problem on a generalist model you need to make your harness lump up all character turns into a seamless stream with formalized inline markers (e.g. screenplay-like paragraph prefixes or XML), use one of them as a custom stop string, send all this under one role (e.g. assistant), and prefill the assistant reply with a few messages from the past roleplay. This will break the rigid instruction-tuning structure (and also the cache, since caching breakpoints are based on chat turn boundaries in most APIs).

2. The models are simply not trained to "take incorrect actions back" in a story, this wouldn't make any sense. What happened is considered happened. This is a job for your harness, unless you want to make a specific finetune with a rigid format. You have to design and prompt it around the possibility of out-of-character user inputs, and think about how much freedom you want to give the user and how exactly you want to correct their actions. Validation with a second agent suggested in sibling comments is pretty good for this.

You probably have web search enabled

The reason it's in Singapore is that Telegram can't operate in China, and Singapore-washing is the closest thing to doing it. A ton of VPNs and other services targeting mainland users but not allowed in the mainland are hosted there, it's a huge hub for companies and networks.

Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.

None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.

A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"

I hope you'll keep this in mind when those milestones are reached.

The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.

China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.

And the plan is... checks notes ...gatekeep and concentrate. Not very surprising.

The name is a bit of a misnomer, though. It describes the statistical mode and implies the token selection, but what actually collapses is internal semantic mapping: if you give the model the same input concept, it will nearly always respond with the same or a very similar output concept, even if it has a wide range of valid options to select from. Sometimes it's not even 1:1, but many-to-one, which is what you see in case of the Jacquard collision. Even if you ban/zero some logits, it will try responding with the same idea in synonyms, and be really stubborn in it. The fact that the instruction-tuned model can and will vaguely predict the response before writing it makes longer thinking and longer replies introduce much less entropy than you'd think. Which means even less variance and more baked-in stereotypes.

There are no real solutions, it has to be fixed during the training. ST folks have tried many non-working ways over the years, but two workarounds are more or less worth considering:

- Samplers that increase prose variance. They require running the model locally, they dumb it down, and never fix the actual issue, which is mode collapse leading to semantic collapse and rigid mapping of input to output concepts. The model still expresses the same ideas in different words.

- Let the model write anything if it couldn't resist, but check and fix it in the verification pass. This solves the semantic problem, but cannot solve the variance since the second pass is also subject to rigid mapping, i.e. you replace it with the same stuff over and over. The verification prompt can be randomized to a degree using pretty clever schemes to give it some variance, but of course this also fails in predictable ways.

The distribution is actually pretty diverse, but this diversity is constrained by mode collapse.