OpenAI illegally barred staff from airing safety risks, whistleblowers say

https://www.washingtonpost.com/technology/2024/07/13/openai-safety-risks-whistleblower-sec/

159 177 2 years ago

Matt Levine put this as a great idea for a lucrative job in his Bloomberg column (where hedge fund can be replaced with any company):

Take a job at a hedge fund.
Get handed an employment agreement on the first day that says “you agree not to disclose any of our secrets unless required by law.”
Sign. Take the agreement home with you.
Circle that sentence in red marker, write “$$$$$!!!!!” next to it and send it to the SEC.
The SEC extracts a $10 million fine.
They give you $3 million.
You can keep your job! Why not; it’s illegal to retaliate against whistleblowers. Or, you know, get a new one and do it again.

https://www.bloomberg.com/opinion/articles/2024-07-15/openai... (archive: https://archive.ph/SWLh0)

judiisis• 2 years ago

Maybe don't violate laws to be fined

nobodyandproud• 2 years ago

You skipped a step: First, enforce the rules and actually (and readily) impose the fines to indicate the laws are meaningful even for certain segments of society.

johngladtj• 2 years ago

Everything is securities fraud

potatototoo99• 2 years ago

Safety risks are not trade secrets.

helsinkiandrew4• 2 years ago

Safety risks are not trade secrets.

No, but safety risks are potential securities law violations and a confidentiality agreement that inhibits employees talking to the SEC about anything freely is a SEC violation in itself

iamacyborg• 2 years ago

Yeah that got a laugh out of me this morning

rurban• 2 years ago

The whistleblowers letter: https://www.washingtonpost.com/documents/83df0e55-546c-498a-...

Mistletoe• 2 years ago

Seems like OpenAI is going the Uber route of try to skirt the law and ignore it “because we are tech”. There is nothing new under the sun and no one is above the law.

Valodim• 2 years ago

Would have agreed on that sentiment until a couple weeks ago...

BadHumans• 2 years ago

Uber got away with it though? What consequences did executives at Uber or even Travis Kalanick suffer? He had to sell some stock, get a massive payday, and retire from the public spotlight?

sirolimus4• 2 years ago

I'm using mistral now, openai is a dying corporation in my opinion. All AI will and should be open-source and home-ran

tuwtuwtuwtuw2• 2 years ago

Why should all AI be home run? I play around with gemma/llama/sd locally but being able to pay some company to do it is very convinient.

I think most companies and people don't want to buy the hardware required to run an LLM like the ones OpenAI hosts.

carterklein132• 2 years ago

Even if AI should be home-run, it probably won't be for most people. In a more technical community, it's natural to think that most people care about "values" when it comes to tech. However, the reality is that people just want what's easiest / cheapest / most fun. It's great when what's easiest/cheapest/most fun aligns with what's best for the individual or society, but those cases are outliers. After spending a few years building a crypto startup, I left with more conviction around this theory.

whywhywhywhy• 2 years ago

Why should all AI be home run?

Because the ultimate end game for usefulness of this tech will be something akin to always on listening and understanding all your documents and personal data which I think we'd all feel better about if it happened locally.

Not necessarily saying I agree with the building of that, I just can't escape the idea that it's where we're heading.

sirolimus4• 2 years ago

Because of privacy-reasons, convenience, embedding, security and safety in regards to government monopoly. Just imagine the paranoia in using government owned AIs in less free countries. I'm happy USA is at the forefront in AI development.

matheusmoreira2• 2 years ago

All AI will and should be open-source and home-ran

I hope you're right. Technology this good should be free as in freedom.

sirolimus4• 2 years ago

Exactly, it's terrifying to imagine "AI" in 10 years in the hands of a single group of people.

jsheard2• 2 years ago

Where is the money for training big expensive open source models going to come from once the investor hype blows over and companies like Mistral actually have to try to make a profit? They currently have negligible revenue despite their $6 billion valuation, that status quo can't be maintained forever.

josefx• 2 years ago

It might be great for everyone if AI research had a reason to look into non brute force training methods.

sirolimus4• 2 years ago

I guess a parallel could be made to vue, react and svelte. Who would bother investing so much time, energy and money into developing an open-source front-end framework for free/funded by corps that use it? Vue, react and svelte don't earn anything I suppose, but then again I'm no expert in this field.

I guess like some other here have commented, techniques have to be implemented to minimize training time and I suppose the government needs to fund studies which then will benefit the general population to make it possible to train and host AI models.

But I honestly have no idea ... I'm just a simple dev running my mistral on a single RTX lol

infecto• 2 years ago

These agreements will most likely be ironed out.

What I am more interested in is the constant pressure on "safety risks" without anything that feels tangible to me so far. I believe there is indeed risk using models that could be biased but I don't believe that is a new problem. I still don't think we are at risk from a runaway AGI that is going to destroy us.

matheusmoreira2• 2 years ago

It's just the angle they're using to solidify their market position and establish moats around their technology, if not an actual monopoly. All this fearmongering over "safety"? It's to get the governments the world over to "regulate" this stuff, thereby raising barriers to entry. Such a thing would essentially make them the only AI company that's allowed to do business.

seunosewa• 2 years ago

This won't prevent their major rivals - cloud companies like Google with deep pockets - from competing with them. It only hurts the little startups and SMEs.

ADeerAppeared4• 2 years ago

What I am more interested in is the constant pressure on "safety risks" without anything that feels tangible to me so far.

It's marketing. "Our AI is so powerful it's going to destroy the world" is just a way to market "Our AI is so powerful, give us money".

The only people who sincerely care about "extinction risk" are the weird cultists. In the real world there's essentially zero chance of LLMs/Current-GenAI scaling up into AGI, nevermind AGI that'd be an extinction risk. (Yes, that phrasing is cheeky. AGI is a different kind of AI, not just specialized models made bigger. But we're only really trying to make them bigger, and not build general intelligence from the ground up)

I believe there is indeed risk using models that could be biased but I don't believe that is a new problem.

It's the same old problem with most software, and it's similarly ignored.

These models are biased, and attempts to control that bias are a shitshow. (As Google conveniently showed everyone)

The problem is twofold:

1. This bias has severe real world impact. https://www.theverge.com/21298762/face-depixelizer-ai-machin... This is an article from 2020. We still haven't fundamentally addressed problems like this. These tools are still used by law enforcement and businesses making life-impacting decisions.

2. There's a widespread sentiment that "computers can't be racist". Whenever the bias of these systems hits the news or otherwise gets attention, they're often colloquially described as "racist" (or "sexist", etc), which triggers a swift counter from many techbros going "Um akshually it's not racist, it's merely skin tone reflectivity/it's merely the data set/etc"^[1]

The argument effectively being, "It's not racism because it's not intended, it's merely sparkling discrimination". Yet, this is used as a thought terminating cliché. Nothing is done about the discrimination. Everyone just goes home. "It's not racist, we're not bad people, job done." Leaving the harm of the discrimination unsolved.

This is not without cause. The only way to really "un-bias" these systems effectively would be to extensively curate the dataset and admit that the systems are of very limited capability and should not be used in (non-research) production environments.

Both of these are "impossible". Curating the dataset for current generative-AI would take years and years. And admitting AI shouldn't be used for anything where the bias may have a material impact on the outcome kills the hype bubble. AI firms and developers don't want to address the problem, because the problem is really hard and annoying.

But despite these costs, we should still do it. Because it is the morally correct thing to do. And because regulators are going to tear every company involved a new one if they don't.

---

[1]: A footnote to pre-empt something: I don't care what side of this argument you're on. Whether you believe "racism" must include an element of intent or can be done by machines and systems without intent, there is discrimination with material harm on real people. Whether you call that discrimination "racism" or "sparkling machine discrimination" does not matter. The harm matters, and must be stopped.

rolisz2• 2 years ago

Whether you call that discrimination "racism" or "sparkling machine discrimination" does not matter.

I believe it does: if you're calling racism anything from Hitler to someone who wants to be punctual [1], the word loses it's meaning and people will start arguing about degrees instead of harmful behaviors.

[1] https://www.thompsoncoe.com/resources/myhrgenius/hr-tips/tip...

michaelt• 2 years ago

It's marketing. "Our AI is so powerful it's going to destroy the world" is just a way to market "Our AI is so powerful, give us money".

Even better - it's a way to simultaneously market your AI as very powerful and to get governments to clamp down on your competitors.

MrScruff2• 2 years ago

The only people who sincerely care about "extinction risk" are the weird cultists. In the real world there's essentially zero chance of LLMs/Current-GenAI scaling up into AGI, nevermind AGI that'd be an extinction risk. (Yes, that phrasing is cheeky. AGI is a different kind of AI, not just specialized models made bigger. But we're only really trying to make them bigger, and not build general intelligence from the ground up)

Given that a significant number of people working in the field apparently disagree with you perhaps you could give some justification for your dismissal beyond the ad-hominem? And it's very blatantly not the case that the entire field is 'only really trying to make them bigger'. The news is currently full of stories about coming advancements from OpenAI which have nothing to do with scaling and that's just a single company.

mindslight• 2 years ago

In the societal context, by the time you're focusing on racism in a decision process you've basically already lost. The best it can achieve is to make sure that oppression is uniform across whatever broad categories that end up being measured. What we're sorely missing is accountability (civil, criminal, and less of power imbalances in general) for individual unjust decisions and actions, regardless of the motivations behind them (racist or otherwise). This includes accountability for individuals and organizations who have adopted "AI" and then hide behind "the computer" or "policy" as if they are not still the responsible parties enacting those decisions (though obviously this problem is much larger and older than merely "AI").

ImHereToVote• 2 years ago

How many hours do we have once we have something that does look like AGI that is going to destroy us?

moffkalast5• 2 years ago

Probably years to make any kind of dent, while using so much power that it would be easily noticeable. Take a group of smart people, arguably the equivalent for AGI. How can they destroy the human race? We've made it exceedingly hard for anyone to do anything of the sort because there've always been groups like that. It's even harder if you have no physical form at all.

Besides, take literally any base model of any LLM made so far, they're quite the opposite of evil and destructive. Training on human data makes them embody human values and makes them a complete non threat.

lijok5• 2 years ago

Actual self-aware AGI that hasn't been completely restrained - probably a week at most before it improves itself enough that it can put things in motion which would allow it to sustain itself without humans.

But when talking about AI risks, although everyone's mind goes towards skynet, the actual risks being discussed are things like use by authorities for oppression.

pjc505• 2 years ago

We already have this problem with ordinary humans, especially since we have The Bomb. AGI can go to the back of the queue behind wars, pandemics, and climate change.

No, a far more normal threat is "how are humans going to use AI to ruin the information landscape?"

arduanika• 2 years ago

The phrasing of your question contains a dubious premise. Any answer that takes the question at face value is going to be conditional upon popular but outlandish notions like "AGI" being a meaningful concept, as opposed to a marketing term, and upon an unfounded prediction about what said entity is going to do.

ben_w4• 2 years ago

Before it can or before it has more than 50% chance of having actually done so? Because those are different things.

For the first, you don't need AGI, no matter which of the many definitions you have for what that means. The automation used by the US for their early warning radar in Thule Site J was not programmed to know that the moon does not have an IFF transponder and that's OK, while the Soviet satellite-based system was triggered by reflections of sunlight.

Likewise covid, cancer, smallpox, ebola, HIV — these things do not need "general" intelligence, they're self-replicators that hijack our bodies and have no goal besides what they do.

Some talk of "P(doom)", but (and I should blog about this) every example I have seen has a blind spot that is especially embarrassing given the circles in which this discussion proliferates is very familiar with Bayes' theorem: P(A) by itself isn't meaningful, it's P(A|B), so when it's doom, doom given what? There's too many options for what each of technology and politics will allow AI to "look like". Doom within 24 hours because an AI takes an instruction literally and without regard for our ethics, is very different to 'doom' in a million years caused by natural extinction that is never prevented because every single time we make an AI we keep finding that it builds a spaceship and flies off into the void so it doesn't need to deal with us any more.

Me, I think that when LLMs alone (again, don't need to be "AGI" by any particular measure) have the capabilities of someone with 5 years post-graduation work experience in just biology (or similar subject with similar potential for accidental harm), then we've got a reasonable chance of multiple incidents each year where idiots (and/or misanthropes) make a Covid-scale pandemic (or, if the LLM is a software rather than biology helper, a deliberately corrupting rather than greedy version of the current encryption-blackmail malware) — and that kind of scale has a combined risk of about 10% of one of the incidents being of sufficient scale that global society collapses and doesn't recover, before it's happened so often that everything gets locked down, possibly including a Butlerian Jihad. And if you think humans would "obviously" take steps to prevent this after the first failed attempt, I would point out quite how many US politicians have been shot and yet those same politicians still refuse to consider that the 2nd Amendment might possibly be not that good. (In case you think this is topical: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_... )

Remember, the AI doesn't need to actively hate you, it just needs to do something dangerous — and it doesn't matter if that danger comes from a long-term plan that requires it to engage in amoral power-seeking, or a short-term plan from a deranged monster of a human using it, and even when both it and the user are "pro human" it can still do this from simply being unaware that one part of the plan is incompatible with life.

stainablesteel3• 2 years ago

i'm in agreement that the safety/control attitude towards AI is more dangerous than anything AI will actually do

lijok5• 2 years ago

How is the safety/control attitude towards AI more dangerous than, for example, using AI to curate a list of likely future enemies of the state based on their contributions to discourse online?

dartos6• 2 years ago

I agree that I don’t think we’re in danger of runaway AGI, but tbf, in the early 2000s, there were a lot of safety risks to social media we couldn’t see yet.

“I don’t think a live message board is going to destroy society” could’ve been something someone said about Facebook or Twitter.

The danger there wasn’t in the tech, it was in the societal impact and consolidation of information control. Both, imo, are danger areas for AI as well.

exe3414• 2 years ago

did Facebook destroy society?

slibhb4• 2 years ago

The whole "social media is dangerous" is a moral panic. There's just no good evidence for it.

jrochkind14• 2 years ago

I think there probably are safety risks that I worry about too; but I don't think this article mentions them or is clear on what safety risks employees were worried about, if any, despite having the phrase "safety risks" in the headline?

pk-protect-ai• 2 years ago

I'm refusing to consider the runaway AGI a possibility within the next 20 years. However, your comment shifts the discussion away from the actual issue. Altman's hype lines were "regulate us," which is what this hypocrite rode the wave of hype with. Now he is forbidding reporting on safety risks to the regulator... which is illegal in the first place, and yes, the entire business of OpenAI is so overhyped due to Altman's manipulations.

I'm so glad that I no longer need to use OpenAI models; there are far better alternatives available, thanks to Meta and Anthropic. Private

slibhb4• 2 years ago

What I am more interested in is the constant pressure on "safety risks" without anything that feels tangible to me so far.

Exactly. I think it's the classic "I'm so smart I can reason about the world without any evidence". This is a perennial trap that very smart people fall into. The antidote is to always look for evidence or "something tangible".

pjc505• 2 years ago

"I'm so smart I can reason about the world without any evidence"

This is also a key limitation on non-embodied AIs.

Ragnarork• 2 years ago

In a statement, Hannah Wong, a spokesperson for OpenAI said, “Our whistleblower policy protects employees’ rights to make protected disclosures. Additionally, we believe rigorous debate about this technology is essential and have already made important changes to our departure process to remove nondisparagement terms.”

How can corporate communication always play that card: "Our policy on X is very good, and we believe X is very important, so that's why we're making changes right now to things that were blatantly in contradiction with X, that we wouldn't have made if this didn't make it to the press"?

neilv2• 2 years ago

Recent: OpenAI whistleblowers ask SEC to investigate alleged restrictive NDAs (reuters.com) | 76 points by JumpCrisscross 2 days ago | 17 comments | https://news.ycombinator.com/item?id=40959851

farceSpherule• 2 years ago

OMG! I am totally shocked than an arrogant egomaniac did something like this. Shocked I tell you.

keepamovin• 2 years ago

Moves and countermoves. I like the brief consideration raised by this post about all the ways that a shiny, new successful company may be attacked by its competitors surreptitiously, through the media, using lawfare by proxy, and so on. Such unadmirable deviousness!

say_it_as_it_is• 2 years ago

What kind of people are gravitating towards AI Safety roles-- activists? How does an organization avoid activists mucking up their models when a disproportionate number of applications is an activist? I guess the answer is to actively recruit for the position rather than accept applications.

mrcwinn• 2 years ago

Fear not: the guy who’s daily driving a Koenigsegg supercar has your best interests at heart.

mistermann4• 2 years ago

See also: "democracy", that sets root guidelines for everything. Funny how it manages to almost entirely evade human curiosity (besides telling stories to each other) despite it being more important than almost everything.

z3sRzPP32• 2 years ago

Very short article, still concerning, but I'd love more info.

z3sRzPP32• 2 years ago

Very short article, still concerning, but damn I'd like more info.

ChrisArchitect• 2 years ago

Some more discussion: https://news.ycombinator.com/item?id=40959851

jrochkind14• 2 years ago

The headline made me think there were going to be specific "safety risks" mentioned, but there do not seem to be. I am not sure what justifies the phrase "safety risks", "safety" especially. It makes us think of like, safety to humanity from AI or whatever, but the most focused upon thing in the article is SEC-related stuff; I'm not sure if the word "safety" is meant to refer to securities/financial related stuff, or meant to refer to other stuff?

(Note, I am personally a pretty anti-AI person honestly I went to the article hoping to get more ammunition on it, wondering what the "safety risks" employees were worried about were, disappointed to not get info on that, unclear if it was a thing or not).

ryandrake• 2 years ago

The word "safety" has been seriously overloaded lately. I'm used to "safety" meaning actual physical safety measures, like seatbelts, helmets, railings... things keeping you safe from physical injury. When OSHA talks about a "safe" working environment, they're talking about fall protection, harmful chemicals, contagious diseases, and so on.

I don't know what "safety" even means in the context of an LLM that outputs nothing but text. How is text going to cause me to fall off of a forklift or get me hit by a car? How can text injure me?

elthran• 2 years ago

Agreed - I've been unable to get at sensible meaning in my head for "AI safety" that isn't "to prevent ultron/skynet/the matrix"

jrochkind14• 2 years ago

I mean, an LLM could cause you to get hit by a car if in the future it's driving a car! I can think of various ways an LLM could effect physical safety.

But I agree it's unclear what the headline authors were even thinking when they put it in the headline (which makes it hard to say if it's justified by the article although i suspect not), in part because the word in general is often being used in unclear ways.

While not exactly or necessarily "physical safety", I could see the dangers described by this other HN front-page article about AI-powered phishing as "safety". https://news.ycombinator.com/item?id=40981067

SEC regulation violations? As "safety" without an adjective, like "financial safety" or "regularatory safety" or something (and those would still be kind of weird)? Seems weird.

batmansmk• 2 years ago

I start to feel this is all marketing. Pretend it's dangerous, so it implies it's beyond what we imagine. Because on our end, on the reality of a B2B product used daily, finding use cases for the limited OpenAI we have access to is far from trivial.

_fat_santa• 2 years ago

I'm really really not a fan of the constant talk about "safety". My issue is that it never actually points to anything tangible, anytime I read about safety it's always used in a roundabout generic way. There's so much handwaving about the issue but every time I've tried to dig into just what the hell "safety" means, it's always either refers back to itself (ie. "ai safety is about safety") or makes some vague reference to an LLM telling a mean joke.

1vuio0pswjnm7• 2 years ago

"OpenAI made staff sign employee agreements that required them to waive their federal rights to whistleblower compensation, the letter said. These agreements also required OpenAI staff to get prior consent from the company if they wished to disclose information to federal authorities. OpenAI did not create exemptions in its employee nondisparagement clauses for disclosing securities violations to the SEC."

Who drafted this employee agreement. Not a single OpenAI employees thought to have it reviewed by a lawyer before signing? Or perhaps someone did but was too frightened to tell OpenAI, "Your employee agreement needs to be fixed."

This company continues to com across as naive and amateur. Perhaps because it was never intended to become a commercial entity.

The other day I saw a Tesla with the license plate "OPENAI". No doubt there were also license plates that said "FTX".

OpenAI illegally barred staff from airing safety risks, whistleblowers say

Related Stories