HN user
Did the model really need to hack huggingface to get access to ExploitGym data? I'd imagine that once it had full internet access it could have just used the HF API or website (but the heavy prompting/nudging towards hacking made it do things the hard way).
What would evidence of "intelligence" or "creativity" look like for you?
One "solution" to these problems is to have the probes land on planets instead of asteroids, and build the necessary infrastructure there.
The difference is that artistic sensibility is largely subjective. This means that:
1. It's hard to measure (and people can disagree about it)
2. It can't really be improved using RL without a human in the loop (which is how math is being trained)
I feel like the data should have been generated by a much less predictable policy.
It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what would happen next during training without paying attention to the inputs)
This post misses a very important point: humans aren't 100% correct either. This means that the bar for being useful (at tasks that humans usually do) isn't perfection, it's human-level correctness. If we can create AI models that make fewer mistakes than humans (which is almost certainly possible, even if not easily or soon), then we will actually need to spend less time double-checking for correctness than we do now.
We see it with self-driving cars. It's really impressive what is possible. But they aren't true self-driving. A person needs to be sitting at the wheel, keeping their attention focussed on traffic as if they would be driving themselves, so they can intervene when the AI makes an inevitable mistake.
There are self-driving cars all over San Fransisco transporting people on public roads with no human at the wheel. This proves my point: those cars are not perfect, but they are human-level (or close), and that's all that's needed.
Why can't a next token predictor do math? Humans aren't calculators either, but we can do math.
If you want proof just look at the benchmarks. Modern frontier models can get basically perfect accuracy on American Invitational Mathematics Examination tests: https://matharena.ai/?comp=aime--aime_2026
If you want an explanation of how they do math, we've found geometric calculators inside their neural networks: https://www.goodfire.ai/research/a-geometric-calculator#
Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier.
If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions it gets wrong are the questions where verification or hallucination detection are the most difficult
Neither of these strike me as particularly groundbreaking.
The first idea (as I understand it as retrieving token ids rather than hidden states) is going to really struggle to do useful compositional reasoning and contextual recall.
The second idea has been been done a million times, with Linear Attention being maybe the first modern example. Hyena, state-space models, DeltaNet, and LaCT also lie in different regions of the performance-parallelizability spectrum of fixed-size models.
LLMs cannot do math
This is plainly not true anymore
These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are.
Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more).
This is important because, from my own experience, simplifications and alternatives to standard attention can look fine in the under-trained regime but lag after over-training. This happens because attention has very little out-of-the-gate inductive bias, so it takes a lot of training for the expressiveness to really shine through.
I can't fault the authors since longer training runs cost money, but it warrants pointing out.
I'm also disappointed that they didn't report reasoning benchmark results for the Q=K-V case, since that is by far the most theoretically interesting case (in my eyes).
The problem is "public schools". The idea itself is wrong, and it can't be made to work.
Do you have an alternative idea in mind?
Care to elaborate?
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.
I don't know what memory would look like other than an encode-retrieve loop.
Relevant: Transformers are Multi-State RNNs - https://arxiv.org/abs/2401.06104
It's not changed by the experience
The entire file is not changed, but the KV cache is.
It doesn't remember anything
The model definitely remembers previous exchanges within the same conversation.
Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms.
gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org/abs/2309.10668
We know how the models are built and trained, but we have a very limited understanding of how the final products work.
That is to say, we don't know why they give the outputs that they do.
If we did know how they worked, AI interpretability would not be an open and growing field.
But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers
A bold claim given that the current top post on HN is "An OpenAI model has disproved a central conjecture in discrete geometry": https://news.ycombinator.com/item?id=48212493
Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter
The reference I always go back to is the GPT-3 paper. The cross-entropy loss (an upper bound for entropy) got down to 1.75 nats (2.5 bits). I took 2.1 because 2.5 is an upper bound and I wanted the estimate to end up as a round number.
If KV cache embedding are storing more than just simple tokens but entire concepts with context and nuance, that'll bump the entropy up quite quickly.
Here's the thing: the concepts that the model stores in the KV cache are a deterministic function of the input tokens. Similar to the data processing inequality, this implies that no entropy is actually added.
Looking at it mechanically, a sufficiently powerful model only needs to encode the tokens and can recompute concepts later as needed.
I think you underestimate just how much information 100M words-ish of information is. It's like a 300,000 page novel. That's a 50 foot (~15 meter) thick book.
Surely with (much less than) 300K pages you could describe every meaningful detail of a video series' plot. You don't need to remember the exact pixel values.
You can also scale the numbers up. I specifically chose a relatively small model and short context length as a reference, so 100x bigger is not out of question. At that point, with a 10B token capacity, you are looking at all of English Wikipedia in a single state.
While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high.
A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a reasonable estimate), then the state can encode 100M tokens worth of information.
Real models obviously aren't powerful enough to operate at the limit, but you can see why this is a promising research direction.
They basically just added DeltaNet hypernetworks to existing LLMs.
Nothing super novel or groundbreaking, but a moderately interesting read.
Keep in mind that all LLMs are trained first on text and then fine-tuned on code.
No, they are trained on a mixture of text and code from the start.
What about things like AlphaZero and Atari gameplay, where the model has zero prior knowledge and learns superhuman ability purely using RL?
With sufficient RL sampling/training, there's no reason an LLM couldn't similarly develop entirely new skills, especially in verifiable domains like math and code.
It simply alters the probabilities.
Yes? What else would a learning system do besides alter its behavior? (and you can just sample with argmax or pseudo-randomly of you think probabilities are a problem)
What does RL do then if not discover strategies and solutions that weren't in its training data?
This is true after pretraining, but reinforcement learning allows the model to discover strategies and ideas that weren't in its training corpus.
I wonder how different their method actually is from other sub-quadratic sparse attention methods like Reformer [1] and Routing Transformer [2].
Using the logarithmic mean of your range of about 3 kg of CO2 per day, and the fact that the average car emits about 0.2 kg of CO2 per km, this means that a typical day of Gemini coding produces about the same amount of CO2 as a 15 km (~9 mile) round-trip commute by car.
The ARC-AGI benchmark is basically this already
People are trying to solve it with software too, even if you don't hear about it.
The most high-profile example is the latest set of Qwen models, which replace most of the attention mechanisms with Gated DeltaNet (which uses constant memory with respect to sequenc length).
Test-time training architectures are also getting a lot of attention, and have shown great performance in the acedemic setting. It's only a matter of time before we start getting open TTT models.