reasonableklout

What was weird during the first Internet bubble?

We did hear about this incident from a third party this time, from HuggingFace. What claim are you doubting?

Interesting that HuggingFace's disclosure was 5 days ago, it seems neither they nor OpenAI figured out it was an OpenAI model in evals until now

OP, you will get a much better reception if you clarify that "hand built" just means you're not using existing libraries. It's a cool project, but it's disappointing to come in expecting to see a rare gem of a 100% human project in 2026 but it's really something else.

I'm confused, the authors of the website are arguing that the development of AI should be slowed down so that society can adapt. What do you think they are trying to convince others of?

AI is not yet a top 5 concern for the American public, but polls show that it is for DC staffers: https://x.com/TimSchnabel/status/2072725548405518574

How is commoditisation of models incompatible with AGI?

It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.

The author of the posts you linked also wrote https://www.tobyord.com/writing/inference-scaling-reshapes-a... which posits:

AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.

...

So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.

So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).

Hm, Anthropic reported it was close to operating profit last quarter: https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...

And a third party estimates it will exceed $1B profit in Q3: https://newsletter.semianalysis.com/p/anthropic-3q26-profit-...

Hm, China is also beginning to invoke export controls to restrict homegrown models: https://www.reuters.com/world/beijing-is-looking-curbing-ove...

Do autonomous systems need to solve humanoid robotics to exert power over the physical world? Seems like a lot can be done with drones.

One of the authors talks about this here: https://www.astralcodexten.com/p/introducing-plan-a

It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.

...

Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.

...

If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.

I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".

The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096

Hm, they are still the most transparent lab when it comes to publishing system cards and safety research. For example the system card for Fable 5 runs 319 pages.

The stuff with Fable falling back to Opus was a bad business move but seems consistent with their position on safety and was published in the system card. Is Ben Bernanke joining the board a dishonest move?

What predictions about the technology are the authors making that you do not believe?

There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.

Mythos doesn't appear to be on the verified leaderboard for ARC-AGI 3

It's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked

Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"

The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.

They also need compute. The US has more compute than anywhere else, so Anthropic won't be moving until the EU can offer something equivalent.

A law firm already has: https://www.reuters.com/legal/litigation/legal-tech-firm-sue...

There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence.

The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code. The great R&D difficulty here is to scale pretraining so that it can proceed smoothly in vast distributed datacenters in a fault-tolerant manner.

The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning.

In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining.

The reinforcement learning stage used to be a small part of model training, but ever since AI-assisted coding took off, it has become larger and larger chunk of training. In recent models, the compute spend on RL has allegedly come to rival or even exceed that of pretraining [1], which is a bit scary because RL is classically what lead to sci-fi like AIs which are extremely good at accomplishing goals to the detriment of everything else.

The way that RL works is that you put an instance of your model in some environment (such as a VM containing a git repository) and give it a task (such as fix the linked github issue). The model will then generate a bunch of attempts to solve the task which we call "trajectories", in most cases there is either an objective measure of the task success (such as passing the tests), or a fuzzy measure (such as having another LLM look at the results and provide a score). This is called the reward, and the model will learn slowly by producing trajectories that receive reward. It can actually be quite hard to prevent "reward hacking" from the model here and the rewards must be shaped very carefully, much R&D labor goes into here, as well as similar challenges to distributed pretraining.

A significant challenge is that coding/knowledge work tasks these days are getting extremely difficult, we are far beyond 2024 days where models could barely solve the easiest problems in SWE-bench. Tasks at the frontier now look more like mini projects that would take humans multiple hours or even days to finish (or in some cases, research-style tasks that would be beyond reach for even top human experts, such as the Erdős unit distance problem which was posed in 1946 but wasn't solved until recently, by GPT-5.5). Huge amounts of trajectories must be produced, and huge amounts of them produce zero reward and therefore are useless for learning. Getting a cold start requires running tens of thousands of instances of your model in VMs in parallel for multiple days to produce trajectories, to say nothing of the GPU costs.

So what do you do when you only have a model which is capable of basic conversations but cannot even begin to tackle basic coding tasks, use tools, etc? The approach that companies behind the frontier have decided on is to bootstrap their learning process by having an already extremely intelligent model such as Claude produce hundreds of thousands of seed trajectories for them. Then they can use this data to get a warm start and begin learning immediately. And if you use Claude for your reward model too, you get to skip the nastiness of reward shaping.

Therefore, even if in number of raw tokens the data are much smaller than internet-scale pretraining data, the value that each token provides is far far greater.

[1] For example, Grok 4 compute spend on RL was ~100% of that of pretraining: https://www.interconnects.ai/p/grok-4-an-o3-look-alike-in-se...

Anthropic did pay $1.5B to authors. But yes, it would be much better if they paid everyone on the internet dividends from every Claude chat. Or released Claude as an open model.

In practice, the former isn't very realistic, while the latter is politically dead as this is becoming a national security issue.

Yes, there are lots of obvious LLM tells that don't add value, like "the math has to be empirical, not aspirational", use of colorful technical language like "knobs" and "wiring", etc. It distracts from the content.

This particular article is reported as 100% AI-generated by pangram: https://www.pangram.com/history/52f51cff-9166-4b8e-ab2e-39fd...

I assume the author found the data sources and did some editing, and they have a few other articles that appear more human-written.

Interesting bit blaming Jimmy Ba for much of the cultural issues:

Interestingly, the lawsuit doesn’t implicate Musk himself as a reason for a lack of safety. Rather, Kim’s lawyers describe Musk as having directed xAI to follow the law and implement appropriate safety and testing processes. Instead the claim targets Kim’s supervisor, xAI co-founder Jimmy Ba — who left the company earlier this year — saying that Ba ignored Musk’s directives and retaliated against Kim for pushing for safeguards, in an effort to “silence his repeated complaints about AI safety and biases.”

Very bad news for Gemini - the brief comeback with 2.5 Pro last year looked to be driven by Noam

It can both be true that models at Fable capability level are national security concerns while Anthropic is being hypocritical.

If the Trump admin is also willing to apply the same scrutiny to GPT-5.6 and other Fable-level models, I think it is a good thing. But given the admin's history with Anthropic (such as declaring it a supply chain risk while ignoring Chinese labs), there is some smell of targeting.

See also https://www.owenmcgrann.com/p/the-dead-economy-theory which was thoroughly discussed 17 days ago at https://news.ycombinator.com/item?id=48324712 (1426 comments).

But you just hit the nail on the head: "Solo software engineer can't create on of these now".

The current boom in AI and the cloud/social media boom in the recent decade have required ungodly amounts of capital for their resident companies to get off the ground. It's no longer a creative endeavour that basement hackers can participate in. In many ways it is toxic to the original nerd/hacker ethos by shutting out newcomers to the field and increasing wealth inequality, hence the hostility you now see on HN.

Great, so let's have the government apply the standard across the industry and see whether Anthropic sticks to their stated beliefs.

How was this release more rushed than previous ones?

I have the opposite impression using Fable, behavior-wise it is much more polished than Opus 4.7 was at launch.