The hacker sent by Anthropic to calm the government's nerves about AI safety

https://www.wsj.com/tech/ai/anthropic-mythos-safety-nicholas-carlini-20bceaa3

by Brajeshwar • 1 month ago

95 117 1 month ago

wsj.com

Readable: https://www.wsj.com/tech/ai/anthropic-mythos-safety-nicholas...

boramdd• 1 month ago

Being on the other side of the AI machine changes the perspective of whether it is dangerous or not, I guess.

speedgoose• 1 month ago

Everyone has a price.

trhway2• 1 month ago

the coming IPOs will possibly create several billionaires. Standing on the top of a billion dollar pile would definitely change your perspective.

parl_match• 1 month ago

I have good friends in the AI industry who are the living embodiment of that Upton Sinclair quote.

"It is difficult to get a man to understand something, when his salary depends on his not understanding it."

You've never heard such strong one-sided cope until you've talked to an NVDA employee about AI. I'm not even against AI. It's just that a combination of intense financial incentives around a product that provides a good simulation of the Chinese Room has really fucked peoples brains up.

Simon321• 1 month ago

These are the consequences of fear mongering as hard as they did. You reap what you sow.

Now they need to convince the government that they didn't mean anything of the previous things they claimed.

thewebguyd2• 1 month ago

OpenAI is also guilty of excessive fear mongering (remember GPT 2 is too dangerous to release?)

This isn't 100% Anthropic's fault, although I'm sure that's part of it. This is the current corrupt administration executing on a grudge they have against Anthropic, and the government's new found love of picking winners and losers.

yifanl• 1 month ago

OpenAI is much more eager to jump on board with the administration than Anthropic is, Altman is a lot of things, but he definitely knows which wheels need grease.

hgoel• 1 month ago

I don't really think they're acting on a grudge against Anthropic here, I think it really is on Anthropic for describing the model's capabilities the way that they did.

IIRC Anthropic claimed to have been working with the government on securing things with Mythos, but then they seemed to have been blindsided by this.

My read is that the guys making the decision to restrict it were not the ones that Anthropic had been working with, and it's more about Anthropic getting caught between infighting within an incoherent government.

0l• 1 month ago

remember GPT 2 is too dangerous to release

FYI, this was when Dario was still at OpenAI.

lompad• 1 month ago

That was dario amodei as well, when he was still at openai. He is the primary "create hype by claiming you're dangerous"-guy.

yreg• 1 month ago

Public release of GPT (and the following models) did bring negative societal changes with it.

We now live in a world where captchas don't work, astroturfing is indistinguishable, school essays and theses don't prove any learning took place, open source maintainers gradually cease to accept stranger contributions, …

AbrahamParangi• 1 month ago

Nah, that's ridiculous. This admin is corrupt and idiotic and it's silly to pretend that Anthropic's actions matter except in so much as they didn't bribe the president like OpenAI did.

moralestapia• 1 month ago

"Nicholas Carlini recently rang the alarm about the dangers of AI—and now he’s part of a team arguing for the latest models to be released"

Many such cases, he was just hungry.

wil421• 1 month ago

Isn’t that how Anthropic started? Raise alarm bells and ride the hype train.

cmiles8• 1 month ago

The AI labs look rather naive here.

You can’t jump up and down screaming how amazing, powerful, and dangerous your new tech is and then act surprised and annoyed when the government shows up looking to regulate it.

Their new argument now seems be that this was marketing hype/fluff that backfired, in a pretty obvious and predicable way, and now they’re trying to reset the conversation.

tennfown• 1 month ago

I’m way more concerned about the loons willing to throw absurd amounts of money at the clearly naive individuals.

cyanydeez• 1 month ago

No. They got caught in a change in what it means to be "regulated".

Regulation in a functional democracy: Cool, lets figure this out, write up a bill for us, do some research in congress, lets find something that makes sense.

Regulation in a function fascism: Cool, wheres my bribe? My boots not shiny, lick it till I say stop.

See, Anthropic wasn't licking enough boot when Biden got discharged and they thought Amazon and OpenAI and Elon were just going to let them capture a market without fealty to the boot.

voidfunc• 1 month ago

This. Theres a lot of rude awakenings in the future for corporate executive types. They are no longer driving the train. Oh well.

teaearlgraycold2• 1 month ago

This is 99% petty drama between the US government and Anthropic and 1% actual safety concerns.

stvltvs• 1 month ago

But the paperclips!

I'm skeptical about the existential threat of AI, but a lot of smart people have been beating that drum for so long that people are afraid.

colonCapitalDee• 1 month ago

To be clear, this is petty drama *stirred up the US government*. It's not some sort of back and forth, the government is singling them out

matheusmoreira6• 1 month ago

We know, but it's still satisfying to see their fearmongering backfire on them.

tychez• 1 month ago

I just find this idea bizarre.

This bizarre social media meme that AI just performative when Opus 4.8 is just unbelievably good. As if it is so difficult to believe that a more capable model than Opus 4.8 might actually be dangerous and not just entirely a marketing stunt like a person waving to cars in a chicken outfit.

I think it is really this strange form of socialization that people have internalized an anonymous audience they are always performing to themselves. What is going to be the most popular and upvoted thing the anonymous audience agrees with is what I am going to think.

Why would anyone disagree and get downvoted by the anonymous audience like this post?

bellowsgulch• 1 month ago

[flagged]

binary132• 1 month ago

that seems like possibly the most unlikely outcome

dofm• 1 month ago

Dog caught the car

micromacrofoot2• 1 month ago

I suspect they're taking this as a win either way, because they're still plastering "Fable 5 unavailable" on their product and using it as an opportunity to keep themselves in the spotlight as they head to IPO.

There's really not even a ban here, they could slot in Fable under the Opus label and no one would really be able to tell. It's all part of the same show to pump up valuation.

bloppe• 1 month ago

I bet they will do a touch of RLHF and re-naming the moment OAI releases a comparable model. Otherwise, sure, they can just bask in the drama for a bit.

mips_avatar• 1 month ago

Well Anthropic would love some regulatory capture.

Bender• 1 month ago

Also a good wake-up call for investors as these big players can be benched at any moment.

reje• 1 month ago

I can’t take anyone seriously who thought otherwise.

You think you can become more powerful so much so the govt questions its own power? Don’t be stupid. They will simply send in the army to first seize the assets and then nationalise.

It almost seems as if very few people actually understand how the world works. If the govt thinks this is the tech to end all future tech, you think future money flows for invesment matter? Hahaha. No

drtz• 1 month ago

You can’t jump up and down screaming how amazing, powerful, and dangerous your new tech is and then act surprised and annoyed when the government shows up looking to regulate it.

It's entirely possible that models could be "dangerous" to fully release to the general public without guardrails and at the same time the government majorly overreacted in this case.

Releasing Mythos to selected researchers and companies at least gives those researchers a head start at addressing vulnerabilities before the model hits mainstream.

foo-bar-baz529• 1 month ago

Then why did curl only find one new vulnerability thanks to Mythos, and a low-priority one at that? It’s clear that other models are quite capable of finding largely the same vulnerabilities, and that the main key is simply running a frontier model in a good harness to find vulnerabilities.

ChadNauseam4• 1 month ago

You can’t jump up and down screaming how amazing, powerful, and dangerous your new tech is and then act surprised and annoyed when the government shows up looking to regulate it.

True, you can't. But, you can think certain regulations are helpful and certain other regulations are not. And you can be annoyed when unhelpful "regulations" are put in place.

This is like if I say that pitbulls are dangerous, and then the government comes and shoots my pitbull, who I've spent a lot of effort training to not be dangerous. Then you say "well you said pitbulls were dangerous, so you can't really complain." Well, I can complain because If you took me seriously, you wouldn't have responded by shooting only my pitbull!

Think of what incentives this creates for other people. Do you think that OpenAI will be candid about the possible dangers of their technology now? They might not even release it now, seeing that Anthropic releasing their model was what got it export-controlled.

dogleash• 1 month ago

The act of shooting the pitbull makes for good dramatics, but you would get zero sympathy from me if your local government banned pitbull ownership. e.g. Ontario bans pitbulls. I don't have a problem with that.

catigula3• 1 month ago

Sorry, this argument doesn't work. Anthropic claims Mythos is in a class of its own, the evidence corroborates this and the government believes it.

The government shot your pit bull because you were going around telling everyone who would listen that it was the most dangerous, viscous one on the cul de sac and you've trained it to kill people and they took you seriously.

claw-el• 1 month ago

If the authorities see that you publicly and widely shout out that pitbulls are dangerous, but quietly tell me that you’ve spent a lot of effort training it not to be dangerous without sharing how in public, I think it is warranted for the authorities to be skeptical.

jubilanti• 1 month ago

Well it is reasonable to expect the bare minimum of due process, or you should be able to from a government that claims to be so committed to the rule of law.

halJordan• 1 month ago

National security laws often don't require such. It's easy to meet due process when the process is 0. Voters have no one to blame but themselves

Grimblewald• 1 month ago

Remember strawberry? I do. Rememver gpt2 hype? I do.

the hype isnt real, its marketing designed to inderectly siphon capital from the less informed.

The current USA government is no stranger to a grift, so they'll get it. Not that I agree with the practumice, this bubble will hurt quite badly when it goes, but at least AI fundamentally does deliver something useful, even if it isnt infinite value as typically promised.

Think dotcom bubble and the hype and promises made surrouning that. Tge hyper will pass, the bubble will pop, and life goes back to normal as ai becomes part of the mundane everyday human environment. Like websites and domains, some will be used well, some for evil, not everyone needs it, and as we continue to move towards energy being our fundamental unit of value / exchange, if we cant make these models way more efdicient then their use case will be rather limited in scope.

tiahura• 1 month ago

They need to send lobbyists not hackers.

winstonp• 1 month ago

They are absolutely clueless about how to talk to this administration.

fnordsensei• 1 month ago

Yes, why not resolve it the same way all the others have done?

Say that Trump has weird elbows or something, Trump sues for defamation, they settle, bribe completed.

trhway2• 1 month ago

They should have taken money from Thrive Capital.

yeeetz• 1 month ago

They needed to have administration insiders on their team months if not years ago, not just now

OpenAI, Meta, SpaceX are savvy enough to play ball, but Anthropic's public posturing and government affairs has always seemed too aloof and intellectual

thewebguyd2• 1 month ago

To be fair to Anthropic for a moment (not that they deserve it), but requiring administration insiders and the greasing of palms going on should not continue to be the normal expectations of how to do business in the USA. I'm on the side of any company that refuses to capitulate to this administration. Not saying Anthropic doesn't (because they do), but let's not pretend like the blatant corruption going on should be normalized. Every single citizen should be appalled at this behavior and blatant market manipulation.

0o_MrPatrick_o0• 1 month ago

"Carlini had never before found a bug in Linux, or in Ghost. Now he had discovered many."

New guy learns nessus, now tells everyone at the bar he's basically Mr. Robot.

A pox on the labs and the government. InfosecDrama.exe just took out a frontier model because a noob learned how to use a tool.

BoorishBears2• 1 month ago

Can you imagine how cringe it would be setting up that hero image in office?

lelandfe• 1 month ago

I’m sure it wasn’t the intent but the halo really makes him look like a saint

BoorishBears2• 1 month ago

Feels very much intended.

sigmar• 1 month ago

Some administration officials have said that a resolution should include an acknowledgment on Anthropic’s part that its rollout of Fable and communication with the White House could have been improved, people familiar with the talks said.
followed initial frustration Friday among some administration officials when they couldn’t immediately get Amodei on the phone, the people said.

That he didn't drop everything to talk to them seems like the major crux? But Dario doesn't even do the day-to-day operations Daniela does. Feel like Anthropic should just hire Dean Ball to be their liason or something

theplumber2• 1 month ago

I feel Dario did enough harm. I wonder if he can do the right thing and step down. It’s really just tiresome to follow all his PR/Hype/warnings and this fiasco makes everything he says seem so silly. At the same time he’s dangerous for the industry. In the end he may get more regulation than he asked for. If the gov decides the Opus models are too powerful without KYC they are toast. And to be honest I think they deserve it.

bonsai_spool• 1 month ago

And Anthropic say they were on the phone within 15 minutes… This administration is not known for its honesty so it’s hard to take their side of things

micromacrofoot2• 1 month ago

It's because the "crisis" is a sham for publicity, like Trump's constant bullshit deals and ceasefires that aren't real, they're just happening to find more problems to keep them in the news.

FergusArgyll2• 1 month ago

Unrelated to this story but Carlini rocks.

tic tac toe in printf https://github.com/carlini/printf-tac-toe

Recently Regex Chess: A 2-ply minimax chess engine in 84,688 regular expressions https://github.com/carlini/regex-chess

https://news.ycombinator.com/item?id=48136909

FergusArgyll2• 1 month ago

I think headline readers might have thought I meant completely unrelated. It's not. Read the article...

matheusmoreira6• 1 month ago

All the government has to do is simply pull up the blog posts of Anthropic's own CEO.

jonathanstrange• 1 month ago

I'm tired of this story and the corresponding fake discussions because it's completely obvious that Anthropic was singled out because they didn't play along with the current US administration and this whole charade is just part of an extortion scheme.

james2doyle• 1 month ago

Had to disagree with that. However, I don't think you can discount how much Anthropic has been banging the drum about how AI is dangerous (specifically theirs) and an existential threat, etc. etc.

The rollout of Mythos was clearly manufactured to stoke the fears of companies that didn’t have access to it. They also bragged (for Fable) about how they "ran an external bug bounty that produced no universal jailbreaks in over 1,000 hours of testing" only for it be circumvented almost immediately.

So them standing on the high horse and saying it is _so powerful, yet so safe_ only to have that blow up in their face just made it that much easier to make an excuse to do this. Again, not disagreeing, but they made themselves the tall poppy here.

JohnnyMarcone2• 1 month ago

They also bragged (for Fable) about how they "ran an external bug bounty that produced no universal jailbreaks in over 1,000 hours of testing" only for it be circumvented almost immediately.

Where did you see there was a universal jailbreak?

How do you weigh the DOD fight against warning about Mythos' dangers when determining what made Anthropic 'the tall poppy'?

eiejeqq• 1 month ago

Are people here deluded?

Business 101 - never take on an entity who has ultimate power over you and can conduct a course of action to put your existence at threat

SpicyLemonZest2• 1 month ago

People here aren’t authoritarians, so we don’t accept your premise that you shouldn’t take on the government. That’s not how things work in the US. Perhaps you’ve encountered Trumpists who tell you that it is, but they’re lying; they routinely applaud businesses defying any government which their dictator-in-chief doesn’t control.

JohnnyMarcone2• 1 month ago

I feel like I woke up from a coma and all the sudden people are taking the administration at their word. I'm so confused.

slopinthebag4• 1 month ago

[flagged]

tomhow8• 1 month ago

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

slopinthebag4• 1 month ago

My bad, I will do better.

I know you guys are spread pretty thin managing the site, so I went through the comments for this post and collected some other comments which are also probably breaking site rules.

https://news.ycombinator.com/item?id=48576022

https://news.ycombinator.com/item?id=48576065

https://news.ycombinator.com/item?id=48576162

https://news.ycombinator.com/item?id=48576183

https://news.ycombinator.com/item?id=48575948

https://news.ycombinator.com/item?id=48575697

https://news.ycombinator.com/item?id=48575877

https://news.ycombinator.com/item?id=48576280

https://news.ycombinator.com/item?id=48576241

rurban• 1 month ago

Carlini rocks, but they should have definitely NOT sent him. He will them too much, and they'll find even more excuses to block us. This needs an experienced negotiator, ie. a manager

catigula3• 1 month ago

Nicholas Carlini recently rang the alarm about the dangers of AI—and now he’s part of a team arguing for the latest models to be released

Classic Anthropic.

robgibbons• 1 month ago

This export ban follows Anthropic refusing to provide uninhibited use of AI to the US military, and the Pentagon subsequently listing them as a supply chain threat. That in turn led to Anthropic suing the government. This most recent development is obviously just a vindictive administration doing what it can to blackball Anthropic for not kowtowing to unrestricted military use like all of the other AI giants.

sharts• 1 month ago

marketing shouldn’t exist. always exaggerated claims, half-truths.

The hacker sent by Anthropic to calm the government's nerves about AI safety

Related Stories