Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

https://schema-harness.github.io/

by jasondavies • 7 days ago

129 83 7 days ago

schema-harness.github.io

westurner• 7 days ago

Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol.

Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?

---

When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.

BECs, SVT, Superfluid Quantum Gravity

Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.

(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...

could not detect the medium light was supposed to wave in,

Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.

Alifatisk• 7 days ago

What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?

modeless2• 7 days ago

It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.

levocardia• 7 days ago

(1) What does it score on the private test set? (2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)

stared2• 7 days ago

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/

A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

sva_• 7 days ago

I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.

pixl975• 7 days ago

Heh, probably something like this.

Works fast - tells people how to overthrow the government.

Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.

Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.

Training models seems more akin to raising a kid than a computer application.

striking• 7 days ago

https://www.bloomberg.com/news/articles/2026-07-16/google-ge...

Tadpole91814• 7 days ago

FWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.

stared2• 7 days ago

It was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench....

In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.

modeless2• 7 days ago

It turns out knowledge and ability are not the same thing. We should test both.

pants2• 7 days ago

Forget Baba Is You, I want to see an LLM beat Elden Ring

dezgeg• 6 days ago

I've been trying this too: https://www.youtube.com/watch?v=brkP58pZ23w&list=PL8C_UWcLmv...

Cool to see others trying Baba Is You too.

scotty795• 6 days ago

With this harness or without?

causal• 7 days ago

We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.

daytonellwanger• 7 days ago

Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?

Tadpole91814• 7 days ago

This is not actually running the Arc-AGI-3 anymore. To summarize TFA:

1. The AI plays the game and records outputs.

2. The AI does TDD using those outputs to create its own copy of the game.

3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.

4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.

To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.

pixl975• 7 days ago

model's ability to intuit game rules naturally, like a human

This line of thinking terrifies me for the future of humanity...

If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).

Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.

Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.

YeGoblynQueenne4• 6 days ago

> 2. The AI does TDD using those outputs to create its own copy of the game.-
> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.

In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.

Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?

> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.

That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.

It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.

Have the ARC people (Chollet and friends) claimed that this result is cheating?

gandalfgeek• 7 days ago

Big jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.

teravor3• 7 days ago

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it.

it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.

what this harness does is get the model to write a simulator first, it's measuring something entirely different.

UltraSane• 7 days ago

The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3

teravor3• 7 days ago

    > simulator the model builds is comparable to the mental model of the game humans create

then they should try to use that for a more complicated game than Arc AGI. Arc games are simple by design, if you have the model simulate them they become trivial.

vessenes6• 7 days ago

This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.

teravor3• 7 days ago

Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.

the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.

with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.

Tadpole91814• 7 days ago

The point of Arc-AGI-3 is to measure model performance. We already know that models can one-shot and iterate on very rudimentary game implementations. And, naturally, once it effectively has a copy of the source code, it can use that to play the game better.

This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.

And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.

HarHarVeryFunny3• 7 days ago

State grounding turns raw observations into objects, variables, and relations that can be tracked. Mechanism discovery finds how that state changes under an action and writes the rule as an executable program

The way I'm reading this isn't that they are writing a game simulator, but rather that they have two things they are evolving - a perceptual model of the game mapping from pixels to objects, and a behavioral model of how each action acts upon these perceptual objects. The behavioral model is written as a program that can be backtested by the game states and actions they have already taken to see if they are correctly predicting the resulting next game state.

The ARC AGI 3 games are non-trivial, and I think it's very impressive to see them doing well using this approach.

I'd agree with their conclusion:

We read a saturated ARC‑3 as the new beginning: mechanism discovery as a general capability — grounding the causal structure of a world through the agentic loop of action and perception, in environments far richer than a 64×64 grid. This is where we are heading to.

This is the way that an animal learns about it's environment - by observation (and innate biases) to recognize the objects in the environment, and predict their behavior, both autonomous (which AGI ARC 3 doesn't test - the objects in the environment are passive), and in reaction to the animal's behavior. The animal predicts and observes, updating its predictions when it is wrong.

A system that could do this in a messy, dynamic, real-world environment would seem a like genuine step in the direction of animal intelligence, especially if it could ditch the symbolic representations.

try-working• 6 days ago

it's recursion

snewman• 7 days ago

Writing a simulator requires understanding the game well enough to spec the simulator. Acquiring said understanding - within the action limits of the benchmark – seems like the heart of the challenge, so this doesn't strike me as "cheating" at all.

doctorpangloss• 7 days ago

okay, what if this benchmark were just called Arc, and all you knew about it was what questions it asked? it just looks like a bunch of arcade games, which should tell you that it doesn't test AGI at all.

on the flip side, the idea that most tests are bad, even standardized tests, the tests that you scored well on that gave you all your opportunities in life: it cuts to the emotional, grounded core, the absolute foundation, of too many people. in the crowd of hacker news commenters; people who buy anthropic shares at retail; the people who work at tech companies; and their kids, families, etc., who are a bunch of nobodies, there are a lot more incentives to believe "stupid fucking arcade games test AGI" than not.

lubujackson• 7 days ago

That is a fair assessment, but the "something else" is independently valuable, maybe even more so than model improvements - constructing an architecture for efficient rule determination and execution. In other words, I think the goal here isn't so much to beat Arc-AGI but to develop a generic improvement beyond "Ralph loop", which could dramatically extend frontier capabilities for all kinds of uses.

ClassAndBurn2• 7 days ago

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.

Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

vessenes6• 7 days ago

Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.

ClassAndBurn2• 7 days ago

It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?

We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.

couscouspie• 7 days ago

Letting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.

deepfriedbits2• 7 days ago

Indeed. And you can make this case about any tooling at all that is model adjacent.

jvanderbot• 7 days ago

Including by extension all programs, operating systems, or eventually hardware, I suppose.

cyanydeez• 7 days ago

I seriously doubt this, especially in a world in which there's not just one model.

This makes sense if the models some how become unified.

pixl975• 7 days ago

I guess this is kind of an information theory thing. As a model converges AGI would a generalized harness ability appear?

re-thc• 7 days ago

Sol Ultra style is the path forward

That’s called brute force so not really.

satvikpendem• 7 days ago

Codex and Claude Code already have the workflow feature which basically does this on demand, so it's getting there.

Ancalagon• 7 days ago

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend.

Does "don't re-invent the wheel" not apply to agents for some reason?

idiotsecant• 7 days ago

Bespoke tools can be simpler than general purpose ones

dpe82• 7 days ago

Humans (exhibiting "general intelligence") are tool builders; virtually all our capabilities stem from our ability to create and use tools - often extremely specialized to a task. Why would an artificial general intelligence be any different?

Jensson• 6 days ago

Does "don't re-invent the wheel" not apply to agents for some reason?

Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.

And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.

Zababa5• 6 days ago

"don't reinvent the wheel" isn't a law of physics and I think is mostly said by people that never designed anything with wheels.

notjackiechan• 5 days ago

I think there is a chasm to cross. the model's training to be aware of the harness it is in, at the same time building probes and observation tool can help it to cross that chasm.

Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.

vessenes6• 7 days ago

To be clear, we’ll want to see how this performs against the hold-out set. If it holds up, though, it’s a big deal, and kind of in line with the vibes this year, which I’d typify as ‘harness matters’. Maybe we’d upgrade to ‘harness matters immensely’ if this can 100% ARC-AGI-3 on existing models (more in the 13% range without this harness).

I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.

andriy_koval• 7 days ago

To be clear, we’ll want to see how this performs against the hold-out set.

they could take open weight model, and check what will be impact from that harness on hold-out

Zababa5• 6 days ago

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

YeGoblynQueenne4• 6 days ago

ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome.

ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests.

It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form.

Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.

ubermon• 7 days ago

Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained.

hmm, this is like pass@n until you get the high watermark? How would this mean anything?

pixl975• 7 days ago

Do you throw every problem at your most expensive programmer first? Or do you toss most things to your fleet of juniors and let them knock out the easy work and pass the harder problems to your senior developers. Because that's what real world problems look like and work like in business.

Token costs are about the number one thing discussed by businesses these days.

scotty795• 6 days ago

How do humans get their results at these tasks? Don't they try until they succeed and claim their best run as their result?

nkmnz• 7 days ago

Where's the code?

scotty795• 7 days ago

I'd bet gpt-5.6 could reimplement it in few days from this article.

scotty795• 7 days ago

I pretty much predicted this. If such smart models capable of doing math research fail so hard on such simple games the interface is the problem, not the model. Right harness provides a good interface between the problem and the intelligence.

netinstructions• 7 days ago

I can't tell based on this article if the authors intend to submit this to Arc-AGI for the private / held-out set of games for a verified score. The final section sorta seems like they won't bother because Arc-AGI-3 is "now saturated"

nathan_compton• 7 days ago

God, who the fuck are they even writing this slop for? Other machines?

Neat. Maybe even deeply interesting. Absolutely garbage write up.

markisus• 6 days ago

It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.

When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

Related Stories