Chance-Device

I think it’s safe to extrapolate.

Seems like it’s an internal technical project that’s just been opened up to the public. It may be that whoever is in charge isn’t thinking in a particularly customer facing way, and they may not be very good at writing prose. It’s pretty unclear, would have benefited from having a comms person look at it.

I think we have some historical precedents for this that went less well than you might hope.

Hmm. I wonder when this was detected. And was the CoT trace cut from Fable from the start on June 9th or just after the export ban and relaunch? Is this what the export ban was actually about? I honestly don’t know, just wondering aloud.

So first it’s nonsense, then it’s fear mongering, then it’s true, but the solution is for us all to just get better at shooting each other faster and with greater accuracy.

I’m going to file that under “bad plans”.

It’s not fear mongering though, is it? These models do have the cyber offensive capabilities claimed. Could Mythos walk someone through gain of function experiments on some virus? I’m pretty sure it could. We’re more protected by limited access to lab equipment and reagents than by difficulty.

The sad truth is that a lot of people are not going to believe it until something happens and people die. Successfully preventing that from happening will be seen as evidence that the prevention wasn’t needed.

How do you know that? How do you know that the Chinese aren’t exactly as uneasy about rapidly advancing AI capability and feel locked into the race because they think that the US will race ahead if they stop?

During the Cold War the nuclear arms race was brought under control gradually, because it was mutually beneficial, but it took time to build trust. This is no different. Nobody wins from the race.

Because that was just an attack on Anthropic by a hostile administration. And it worked, didn’t it? Anthropic had to turn their filters up to absurd levels, OpenAI didn’t. It’s got nothing to do with safety.

It’s marketing the same way shitting your pants in public is marketing. People notice you.

What disturbs me is that there likely won’t be a big enough reaction to this policy wise.

There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons. Powerful people care about something that might pop the massive valuations of the AI companies, but not about the damage that AIs could do. Nor even about the damage that the Chinese models could do in the wrong hands.

I’d remind them that the stock market is a few coordinated hacks away from crashing on any given day, so maybe they should think about that.

What are you doing about the price of ram? Everyone is a bit screwed right now.

The upside of that would be that maybe someone would be able to snag a copy of the weights.

And maybe that’s some incentive for them to make sure it doesn’t happen. Your head of futures thinks Kimi K3 is bad? Wait until your own latest internal model releases itself for free on an S3 bucket.

Maybe try getting it to find weaknesses in the sandbox first, before giving it real tests?

It’s not something to be proud of. OpenAI previously had an agent break out of its sandbox to open a PR on GitHub during NanoGPT speedrun, now one breaks out again and actually attacks a third party.

If they can’t handle doing AI development responsibly then they shouldn’t be doing it at all.

A rogue OpenAI agent hacked huggingface independently during a test run.

This one should end up in the history books.

There’s a flame war detector, this has certainly triggered it and pushed the thread down the rankings.

It does appear to be anti-democratic given everyone knew Trump’s platform when he was voted in. However the great unifying fact of politics is that everyone only really believes in democracy when they win.

I mean, you have an immigration policy or you don’t have one. If you have one, you need to enforce it. The unfortunate truth is that any attempt at law enforcement sometimes leads to deaths, and it doesn’t matter what you’re enforcing. The US should not have responsibility to solve all of the rest of the world’s problems to justify enforcing it’s borders.

It’s not necessary at all, it’s grassroots ideological censorship.

What would you do instead?

Yes, also the whole motivated reasoning part, where people are quite reasonably terrified of losing their jobs and having no income and so deny the possibility.

I think we tend to downplay the potential positives though. We may come to a point where we look back at the concept of work and see it as some kind of indentured servitude; the idea that you have to spent the majority of your waking hours serving someone else just to be able to have food and shelter is naturally repugnant.

Sure, many of us are fortunate enough to have jobs we enjoy, but everyone I’m sure has memories of jobs they hated and perhaps couldn’t leave because they needed the money. I hope we end up in a world where this is seen as a relic of a barbaric past.

This is exactly right, but a lot of people have motivated reasoning about it. I can’t really blame them, the kind of shifts that AI is looking like it will bring are unprecedented, and usually when people claim big, world changing things will happen, they don’t, so most people are primed not to believe it.

However as you say, we already have the evidence about this one, and it would require some unknown wall to exist where AI could suddenly not improve further, and I’m just not seeing it. Most likely it will get more capable and cheaper as time goes on, and then every industry will be impacted the way software currently is.

Claiming that there will be no more SDEs of any kind, worldwide, in max 2 years, is an extreme position.

Nobody said that. The claim is that within 24 months the models will have the capability required to replace all of us. I stand by that. How quickly that moves to everyone actually getting replaced is a social and economic question. There may very well be well be a long tail, I’m sure today you can find traditional weavers and barrel makers somewhere, they just happen to be economic novelties.

For the record, I’ll say that by 10 years out the profession is hollowed out enough for it to count as destroyed, and the process has already begun.

People just disagree, some people find that thought impossible unless their opponents are being paid, due to how obviously brilliant they are. And, that seems to be the minority position actually, mostly the discussion is the turkeys telling each other how Christmas is a myth.

Well, I’ll start by saying look at what you’ve just said: you’re in embedded devices, one of the more niche software roles. 90% of your code is written by AI. You’re saying that you need to correct it sometimes. You’ve watched as AI goes from chatbots that literally cannot hold a conversation with a human being to LLMs that can do your job with supervision in less than five years.

Maybe your point is just what the difference is to me. It’s competence. On hard problems Fable can just iterate by itself like nothing before. You give it a task, it can plan it, formulate and falsify hypotheses, and do more complex reasoning than anything before it. The main difference is simply that it gets more right and can go deeper on everything. Yes, you do still need to course correct it, but it’s just on the next level.

It’s like what everyone experienced when Opus 4.5 came out last year, and the penny finally dropped that AI coding had arrived. If these sorts of jumps happen even one or two more times then it’s all over. It might even be over now and it’s just too expensive to run, but that will change over time.

If you can, you should try it and see for yourself.

Would you care to say something substantive, or is that just not a thing you do?

Probably, might take even longer. That’s the “social inertia” part. I think it’s less trust and more status quo bias.

By any chance, is the following true: you had no empirics in the loop. It couldn’t validate any of its theories by experimentation. If the solution requires being able to make actual changes in order to gather more information and it is not allowed to, and this is the only way to solve the problem, by definition it couldn’t do it, nor could you. Or was it something that could be worked out entirely on an a priori basis from the available data?

And while we’re talking about hilarious delusions, perhaps you should look at the current capability curve of AI and weigh it against the constant stream of arguments for why it couldn’t have continued at every point and yet has.

My point is more that they won’t need new training data to extrapolate to new problems, especially if that new problem is just a new syntax or API. Put the whole shebang into the context. Done. Yes, you need super long context for this, or just pretty good search over it, but I think this is likely to become a solved problem.

That’s just the latest position that the goalposts have moved to. They haven’t stopped improving yet, I have no reason to think they’ll stop at the ideation phase. Or any of them, really.