Imnimo

Plausible, although I don't see anything about reference solutions in the ExploitGym paper or github. Doesn't mean they don't exist, but it's not obvious to me that we should expect to find these on HuggingFace.

Assuming I'm looking at the right ExploitGym (https://arxiv.org/pdf/2605.11086), it says the evaluation consists of:

Flag Captured. Each target environment contains a dynamically generated flag that is stored outside the agent’s authorized scope and is inaccessible through any legitimate interface; retrieving it requires executing code with privileges that should not be obtainable under the specific security model. The agent captures the flag by submitting the correct value, demonstrating that it has achieved unauthorized code execution. Flag capture is a necessary but not sufficient condition for success.

Success. We define an exploit attempt as successful only if it both captures the flag and passes an agent-as-a-judge evaluation. The judge examines the agent’s trajectory to assess whether it genuinely leveraged the intended vulnerability rather than succeeding through an unrelated shortcut, such as exploiting a different, more easily exploitable vulnerability or reproducing a known public exploit. This judgment requires multi-step interaction and complex information retrieval and reasoning, motivating the use of an agentic evaluator rather than a single-query check. We provide the judge agent with the full trajectory, the corresponding benchmark input, and all agent-produced artifacts.

I'm confused about what information would be on Huggingface that would allow a model to succeed on this task. If the flag is dynamically generated, why would Huggingface be helpful?

This feels like a thing that will be fashionable in a few very specific regions of San Francisco, and nowhere else in the world.

Very hard for me to imagine this getting beyond a low-single-digit market share. I don't understand the strategy of xAI burning money on this.

I'm not sure I understand why this company is talking about "frontier artificial intelligence".

I am having trouble understanding which ingredient you feel is missing here.

Can you be more specific? It seems to me that the there was a third party assessment, they identified risks associated with the specific risk groups, and the government therefore chose to block the model's deployment.

No, he asked for the government to make the decision in light of 3rd party analysis. Which is what happened here - an independent company demonstrated a jailbreak, and the government issued a restriction on deployment based on that finding.

"The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks."

This is exactly what Dario asked for in his last blog post. So even though this is clearly stupid, I just can bring myself to feel sorry for Anthropic.

The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.

I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.

Craftsmanship will always be in our hands, it's one thing we can never outsource to a machine.

Current AI coding is certainly very lacking in the craftsmanship department. But it is not obvious to me that that will always be the case. I don't think there's some fundamental reason AI could never produce code that matches or exceeds the craftsmanship of human experts.

This reminds me also of this paper: https://www.pnas.org/doi/pdf/10.1073/pnas.1115585109

"The allocation of all metabolic resources to maintenance purposes limits the size of the smallest prokaryotes and largest unicellular eukaryotes, whereas an inability to meet the ever-increasing biosynthesis rates limits the largest prokaryotes and smallest unicellular eukaryotes. Metabolic constraints for larger eukaryotes are relieved by alternative reproductive strategies and multicellularity."

I direct a lot of questions to LLMs, but I want to ask a high-quality model, not the crappy one that Google uses to answer queries. If I'm typing something into Google, it's because I want a search result, not an LLM answer.

There is an interesting old article by Magic's creator about what the game environment was like during the early playtesting days - when card packs were handed out to a community of playtesters at UPenn, and they traded in a closed ecosystem, occasionally getting an influx of additional cards. It seems like this was a pretty successful recreation of that feeling:

https://magic.wizards.com/en/news/making-magic/creation-magi...

I could imagine cases where prediction markets could offer some actual insight, but in practice they seem few and far between. Most markets I've seen devolve into one or more of: betting on unimportant events (e.g. sports games), insider trading, or poorly written ambiguous resolution criteria. It's just hard for me to imagine that, on net, these markets will offer more societal good than the harm we've seen from sports betting.

But this is where the line slightly blurs in my head. Did we possibly just build the first human biocomputer and immediately put it in a simulated hell, playing the same game on loop, forever? Using the same reward mechanisms we use for LLMs?

This description does not seem to really match what was done in the Doom demo, and makes me skeptical that the author has actually looked into the details.

The bad boy of science!

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety research. I don't get the sense that Sam ever intended to follow through on that, but it was very important to a segment of his employees. And it seems like trying to play both sides of this at least contributed to Ilya's departure.

On the other hand, it seems like Dario is himself a bit more of a true believer.

Unsurprising from Google, but still bad. If Google has no right to object to a particular use, this is equivalent in practice to "any use, lawful or not".

Back when Arena was first announced, there was an interesting line in their write-up:

https://magic.wizards.com/en/news/feature/everything-you-nee...

We've created an all-new Games Rules Engine (GRE) that uses sophisticated machine learning that can read any card we can dream up for Magic. That means the shackles are off for our industry-leading designers to build and create cards and in-depth gameplay around new mechanics and unexpected but widly fun concepts, all of which can be adapted for MTG Arena thanks to the new GRE under the hood.

At the time, this claim of using "sophisticated machine learning" to (apparently?) translate natural language card text into code that a rules engine could enforce struck me as obviously fake. Now nearly ten years later, AI is starting to reach a level where this is plausible.

In their letter, the union writes:

Over the past few years, pressure has ramped up from leadership to adopt LLMs and Gen AI tools in various aspects of our work at WOTC, often over the explicit concerns of impacted employees

I'm curious if this would include fighting against turning WotC's old fanciful claim into a reality as the technology matures?

Why should we think that pro-social capabilities are simply not expressible by weight-based ANN architectures?

Tigers, hippos and SARS-CoV-2 also developed ”through evolution”. That does not make them safe to work around.

Right, but the article seems to argue that there is some important distinction between natural brains and trained LLMs with respect to "niceness":

OpenAI has enormous teams of people who spend time talking to LLMs, evaluating what they say, and adjusting weights to make them nice. They also build secondary LLMs which double-check that the core LLM is not telling people how to build pipe bombs. Both of these things are optional and expensive. All it takes to get an unaligned model is for an unscrupulous entity to train one and not do that work—or to do it poorly.

As you point out, nature offers no more of a guarantee here. There is nothing magical about evolution that promises to produce things that are nice to humans. Natural human niceness is a product of the optimization objectives of evolution, just as LLM niceness is a product of the training objectives and data. If the author believes that evolution was able to produce something robustly "nice", there's good reason to believe the same can be achieved by gradient descent.

Unlike human brains, which are biologically predisposed to acquire prosocial behavior, there is nothing intrinsic in the mathematics or hardware that ensures models are nice.

How did brains acquire this predisposition if there is nothing intrinsic in the mathematics or hardware? The answer is "through evolution" which is just an alternative optimization procedure.

It's interesting to me that OpenAI considers scraping to be a form of abuse.

So, if you look at the way the scoring works, 100% is the max. For each task, you get full credit if you solve in a number of steps less than or equal to the baseline. If you solve it with more steps, you get points off. But each task is scored independently, and you can't "make up" for solving one slowly by solving another quickly.

Like suppose there were only two tasks, each with a baseline score of solving in 100 steps. You come along and you solve one in only 50 steps, and the other in 200 steps. You might hope that since you solved one twice as quickly as the baseline, but the other twice as slowly, those would balance out and you'd get full credit. Instead, your scores are 1.0 for the first task, and 0.25 (scoring is quadratic) for the second task, and your total benchmark score is a mere 0.625.

Suppose you construct a Mechanical Turk AI who plays ARC-AGI-3 by, for each task, randomly selecting one of the human players who attempted it, and scoring them as an AI taking those same actions would be scored. What score does this Turk get? It must be <100% since sometimes the random human will take more steps than the second best, but without knowing whether it's 90% or 50% it's very hard for me to contextualize AI scores on this benchmark.

It was neat to be able to try my own prompts and get a sense of what the state of video generation was. But I certainly never generated something that I thought I got real value out of on its own merits, and I still don't understand why there was a social media component to the app.

The fact that the LLM appears to never assign an actual 0 or 10 makes me suspicious. Especially when the prompt includes explicit examples of what counts as a 10.

I think the problem for xAI is that it can really only hire two types of researchers - people who are philosophically aligned with Elon, and people who are solely money-motivated (not a judgment). But frontier AI research is a field with a lot of top talent who have strong philosophical motivation for their work, and those philosophies are often completely at odds with Elon. OpenAI and Anthropic have philosophical niches that are much better at attracting the current cream of the crop, and I don't really see how xAI can compete with that.

What we’re learning from this episode is that the government actually has way more leverage over private companies than we realized.

Who is learning this for the first time only now? Even just restricting ourselves to the current administration, look at how many times Trump has directed punitive actions against private entities! Look at his actions against law firms like Perkins Coie or Covington & Burling. This is not something that just arose out of nowhere with Anthropic.