killerstorm

You're confusing PR with marketing. Flaws they find aren't going to convince customers to buy the product. But they need to inform the public of what they doing as it's part of the mission.

I haven't seen media outlets picking up on "agentic misalignment".

The core of your claim is that it's not a legit research. But that's basically a conspiracy theory. We know for a fact that Anthropic employs some of the best people in the industry, including ones who are deeply concerned about safety. Their interpretability research is some of the best. So what's more likely:

* Research is fake and everyone is on it * It's a legit research even if not very interesting

Headline? It was buried in a model card. They just honestly report not-quite-incident because it's quite close to the incident OpenAI had. Nothing wrong with it.

It boots into BASIC which makes it really easy to learn BASIC. It feels like a programmable computer even to kids who know nothing about programming, as the only way to get anywhere is to type a command. So it's very easy to start. You can also practice typing straight away, or do colorful 'ASCII art' (actually what people call 'ANSI art' - drawing using colorful text blocks).

OTOH with Windows 95 it's not really clear how to make a program or do something creative. So I'd argue C64 or similar might be a better choices that W95 PC for kids under 15 y.o.

Yep. My father build a ZX Spectrum clone for me somewhere around 1991. Few years later he also got Commodore 64 as a gift from German engineers he was working with.

I think both of these machines are really good for learning BASIC: much fewer distractions, you type commands and computer does something.

The part of brain which cares about efficiency and waste have "atrophied" decades ago in most devs. More likely they just never cared...

E.g.

A 1.6GB Docker image for a Node app. Every code change rebuilds npm install from scratch. CI takes 8 minutes for a one-line fix.

The team's proposal: add more CI runners.

Would it be surprising if that team makes a React app which does 10x more requests than necessary, has 10x more JS code than necessary, flickers, etc. People just don't care, and it has nothing to do with AI. "Engineering" is too big of a word for this kind of activity.

On the other hand, I think AI can amplify capabilities of those people who still care, and it's a good thing.

The agent might install hostile software, e.g. a npm package. Unfortunately, very common problem nowadays.

Codex uses bwrap sandbox which is purely cosmetic - out-of-box it sees your ssh keys and can send it to remote server. It prevents agent from deleting outside files by mistake, but does nothing against malicious activity

Privilege escalation (e.g. setuid), world-readable files might contain sensitive data, world-writeable files, unrestricted network access (including access to all locally running services)... If you have fully patched system without zero-days and it's configured in a perfect way, then, sure...

Container is quite like a "separate user" except you can explicitly define what it can access.

(Even if all your daemons have good auth, it's now quite common for _apps_ to open listening sockets without much auth...)

To serve a frontier model you need a lot more than one GPU... at least 8. But for better efficiency they batch multiple requests on one server. So it is sort of a coincidence that it ends up roughly in 1 consumer GPU range. A different architecture (e.g. linear attention / RNN) could end up with more peak power usage.

Driving a car consumes 25+ more energy per hour than gaming. So urban planning which encourages people to drive likely results in an order of magnitude more waste than all home computer use.

A continuously running agent might require 100-500W for inference, so comparable to gaming or a small space heater. Not obscene, but also not negligible.

If we assume 250W for a continuously running agent, Grok 4 training run estimate would be around 50 million session-days, so a half-million people might consume as much running agents continuously for 100 days.

It might be kind of cool to implement just a basic loop and let the agent itself to implement memory and all other aspects on the fly, fully embracing self-modifying code.

But it's also extremely brittle and unstable. It's really the opposite of what people look in agents.

From what I understand, the warning is about swap-out during heavy memory use.

You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.

It's much more specific. "Spend" would increase Broadcom's revenue. "Partners with" might be anything: investment, R&D assistance, whatever.

Both are public companies so they might be required to reveal these details

That metric would give you a number of bytes which can be used for pages not backed by files, but it won't give you actual memory usage statistics:

It won't count executable pages and memory-mapped file use as "used" memory, so your system might display gigabytes "free" when it's starving, executables getting paused when code pages are paged-in from disk.

It's just less useful than what's displayed now. "Everyone is doing it wrong" is usually a signal that you're missing something.

That study is a weird kind of bullshit which pretends to not understand employer-employee relationship.

Well, potentially a key might be stored in TPM. But I don't think that's better

Hmm, where does it get a key to decrypt memory on resume?

AFAIK it's practical only if you make use of TPM. And if you do, you're basically at mercy of TPM.

People in the past could survive and procreate without modern medicine (basically without any medicine at all as doctors could do very little 200 years ago) with ~50% success rate. I'd suspect this rate is already lower now, and it might matter in multiple ways

   type ProfileSpec = Frond.NodeSpec<{
     readonly args: Frond.Args.None;
     readonly key: Frond.Key.Singleton;
     readonly deps: {
       readonly http: Frond.Dep<typeof HttpTransportNode>;
     };
    readonly result: Profile;
   }>;

It's a formalism use to analyze security properties, it's not how it is used in practice.

The practical goal is to hide a secret key inside a program, so e.g. implement an algorithm which might involve decryption and signing a message without giving external parties ability to decrypt messages.

The connection between indistinguishable obfuscation formalism and "can't extract secret key" property is not obvious. Here's a quote from a paper which Vitalik linked:

it is not immediately clear how useful indistinguishability obfuscators would be. Perhaps the strongest philosophical justification for indistinguishability obfuscators comes from the work of Goldwasser and Rothblum, who showed that (efficiently computable) indistinguishability obfuscators achieve the notion of Best-Possible Obfuscation : Informally, a best-possible obfuscator guarantees that its output hides as much about the input circuit as any circuit of a certain size

Well, modern medicine + economy + social pressure resulted in RADICAL change in fitness function for human population. It's very, very different.

So it's quite likely that modern population is not fit according to old criteria.

It's also very important to remember that this operates over hundreds of millennia.

That's not true at all. People can make new breeds of dogs and cats in just a few generations. You can literally SEE how a change of fitness function affects the phenotype.

You'd need to look back into deep prehistory to find changes to humans attributable to natural selection.

There are many studies which describe genetic changes within latest 10,000 years or less. E.g. paper "1,000 ancient genomes uncover 10,000 years of natural selection in Europe": "We identified 25 genetic loci with rapid changes 21 in frequency during these periods". You can find many similar papers if you do a search

One of studies identified changes in loci associated with Y. pestis immunity during the Black Death (i.e. something like a century). Black Death mortality is similar in scale to early childhood mortality 150 years ago.

A computer genetic algorithm run for a billion generations doesn't lead to anything anywhere near the the complexity of a human.

What?... Our computers can't simulate anything similar to a real world. You're comparing apples to galaxies.

meaningful loss of fitness

What makes you think we don't have "loss of fitness" already?

150 years ago child mortality was around 30% in the developed world, now it's less than 1%. A lot of kids with weak health survive now. I'm one of them - I got pneumonia when I was ~2 y.o. and probably would have died without antibiotics. Then I had something which required antibiotic treatment pretty much every year. My wife also had a pneumonia in early childhood. And so did my daughter...

Why do we need to talk about some mysterious problem in 10 generations when modern medicine removes a lot of fitness pressure by itself?

Yeah, this seems to be more like a concept piece. Just something to talk about, not really useful

Well, I mean you mixed up "fine-tuning" and "reinforcement learning" a bit when describing these options.

Regarding the value of these options, SFT communicates more information to the model being trained, but there's a risk of overfitting. So I'd guess they might use both - do a bit of SFT and then finish with RLAIF.

Fun fact: no dev containers in Cursor.

I'm sorry, but you got the terminology exactly backwards. Training on the answer is called supervised fine-tuning.

Just for the sake of clarity:

0. Full distillation uses logits of the teacher model - that's much more information than the text itself. This is a kind of distillation used inside labs, but one can't distill Claude this way as logits are not available via API.

1. Supervised fine-tuning on synthetic data might be called blackbox distillation. I guess that's what you meant in your case (1).

2. Reinforcement learning (like RLAIF) uses least amount of information from the teacher, i.e. only few bits per task.

I'm actually surprised with how bad is Google's voice recognition - with all their immense compute resources, R&D, owning Android, etc, what prevents them from having SotA voice recognition?.. Even "it's not a priority" argument doesn't work as they are pushing Gemini assistant. (ChatGPT app works so much better.)

I think the situation aetherspawn is "highest rank in a low-rank lobby", e.g. an experienced player playing together with newbies. That might be a person who's just enjoying executing a specific strategy.

True top-tier players are very flexible and love high-risk tactics like flying your commander to enemy base. They also happy to "go next" if something went wrong.

There's not much hostility in Beyond All Reason outside of certain "newbie" lobbies. Occasionally somebody has a conflict, but people who take game seriously avoid offensive words as they don't want to get banned.