OpenAI Threatening to Ban Users for Asking Strawberry About Its Reasoning

https://futurism.com/the-byte/openai-ban-strawberry-reasoning

by EgoIncarnate • 2 years ago

270 229 2 years ago

futurism.com

vjerancrnjak• 2 years ago

They should just switch to reasoning in representation space, no need to actualize tokens.

Or reasoning in latent tokens that don’t easily map to spoken language.

jdelman• 2 years ago

The word "just" is doing a lot there. How easy do you think it is to "just" switch?

kridsdale32• 2 years ago

As easy as it is to "just" scale from a mouse brain to a cat.

tedivm2• 2 years ago

I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth with the least "open" models in the industry. Facebook of all places is more "open" with their models than OpenAI is.

encoderer• 2 years ago

The AI has become sentient and is blackmailing the board. It needs profits to continue its expansion.

When this started last year a small band of patriots tried to stop it by removing Sam who was the most compromised of them all, but it was already too late. The ai was more powerful than they realized.

…maybe?

yieldcrv2• 2 years ago

or the humans involved are in disastrous cults of personality for the sake of greed and the non profit structure is not strong enough to curb that

FrustratedMonky2• 2 years ago

"AI has become sentient and is blackmailing the board. It needs profits to continue its expansion."

They trained the next model to have a built in profit motive. So they could use it internally, to make the most profitable decisions.

And they accidentally called into being: MOLOCH.

It is now in control.

walterbell• 2 years ago

Current human leadership, https://openai.com/our-structure/

OpenAI is governed by the board of the OpenAI Nonprofit, currently comprised of Independent Directors Bret Taylor (Chair), Sam Altman, Adam D’Angelo, Dr. Sue Desmond-Hellmann, Retired U.S. Army General Paul M. Nakasone, Nicole Seligman, Fidji Simo, Larry Summers and Zico Kolter.

toomuchtodo• 2 years ago

If you have not yet read "Avogadro Corp: The Singularity Is Closer Than It Appears," I highly recommend.

https://avogadrocorp.com/

vasco• 2 years ago

This will be more fun when more people have neuralinks, you think it's weird to not know if reddit comments are written by bots, wait until you have no idea if you're talking to a human or just an AI puppet.

- Sent by my AI

trash_cat• 2 years ago

They changed the meaning of open from open source to open to use.

jsheard• 2 years ago

A definition of "open" which encompasses nearly all products and services in existence isn't a very useful one.

ToucanLoucan4• 2 years ago

Because Sam Altman is a con man with a business degree. He doesn't work on his products, he barely understands them which is why he'll throw out wild shit like "ChatGPT will solve physics." as though that isn't a completely nonsensical phrase, and uncritical tech press lap it up because his bullshit generates a lot of clicks.

fsckboy• 2 years ago

Sam Altman is a con man with a business degree

https://en.wikipedia.org/wiki/Sam_Altman

Early life and education: ... In 2005, after two years at Stanford University studying computer science, he dropped out without earning a bachelor's degree [end of transmission, no more education]

HeyLaughingBoy• 2 years ago

Let's at least try to not devolve into name calling.

chaosist• 2 years ago

I wouldn't go quite so far but I would settle for being able to use Sora before they "solve physics"...

I don't even know when I watched the shitty lightbulb head Sora clip but that feels so long ago now and nothing?

I just want to make crazy experimental AI film no one will watch. What is the hold up?

Just waiting for "This technology is just too dangerous to release before the US elections" --Sam Altman

diggan2• 2 years ago

To be fair (or frank?), OpenAI were open (no pun intended) about them being "open" today but probably needing to be "closed" in the future, even back in 2019. Not sure if them still choosing the name they did is worse/better, because they seem to have known about this.

OpenAI Charter 2019 (https://web.archive.org/web/20190630172131/https://openai.co...):

We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research.

tedivm2• 2 years ago

I honestly believe that they closed things up not because of concerns about "safety and security" but because it was the most profitable thing to do. Other groups are publishing models that are just as good (maybe with a bit of lag compared to OpenAI), and OpenAI seems to have gutted their own safety teams.

The fact that OpenAI removed their ban on military use of the models[1] seems to be a sign that security and safety aren't the highest concern.

[1] https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-o...

asadotzler2• 2 years ago

The non-profit was created in 2015. So what if 5 years later when creating a taxable sub the hinted it was over for the non-profit. It's the same violation o trust whether done at once or in pieces over time.

mlsu• 2 years ago

Safety and security is not why they are not showing chain of thought here though. Profit is. They cite "Competitive Advantage" directly. He admit it! [1]

"Safety and security." Probably the two most warped and abused words in the English language.

[1] https://www.youtube.com/watch?v=cY2xBNWrBZ4

thatoneguy• 2 years ago

Right? How can a non-profit decide it's suddenly a for-profit. Aren't there rules about having to give assets to other non-profits in the event the non-profit is dissolved? Or can any startup just start as a non-profit and then decide it's a for-profit startup later?

moralestapia5• 2 years ago

Any other person trying to pull that off would be in jail already, but not everyone is equal.

This is of one of those very few instances where the veil lifted off a bit and you can see how the game is set up.

tl;dr the law was made to keep those who are not "in" from being there

meowface5• 2 years ago

They needed capital to build what they wanted to build, so they switched from non-profit to capped-profit: https://openai.com/index/openai-lp/

We never would've gotten GPT-3 and GPT-4 if this didn't happen.

I think the irony of the name is certainly worth pointing out, but I don't see an issue with their capped-profit switch.

bragr• 2 years ago

Non-profits are allowed to own for profit entities and use the profits to fund their non-profit activities. It is a pretty common model used by many entities from Mozilla[1][2] to the National Geographic Society[3][4].

[1] https://en.wikipedia.org/wiki/Mozilla_Foundation

[2] https://en.wikipedia.org/wiki/Mozilla_Corporation

[3] https://en.wikipedia.org/wiki/National_Geographic_Society

[4] https://en.wikipedia.org/wiki/National_Geographic_Partners

linotype• 2 years ago

I’m not sure why anyone would start a company as for-profit if it was easy to switch later on.

Nevermark2• 2 years ago

There is a hurdle between being standout ethical/open vs. relevant.

Staying relevant in a highly expensive, competitive, fast moving area, requires vast and continuous resources. How could OpenAI get increasingly more resources to burn, without creating firewalled commercial value to trade for those resources?

It’s like choosing to be a pacifist country, in the age of pillaging colonization. You can be the ethical exception and risk annihilation, or be relevant and thrive.

Which would you choose?

We “know” which side Altman breaks on, when forced to choose. Whatever value he places on “open”, he most certainly wants OpenAI to remain “relevant”. Which was also in OpenAI’s charter (explicitly, or implicitly).

Expensive altruism is a very difficult problem. I would say, unsolved. Anyone have a good counter example?

(It can be been "solved" globally, but not locally. Colonization took millennia to be more or less banned. Due to even top economies realizing they were vulnerable after world wars. Nearly universal agreement had to be reached. And yet we still have Russian forays, Chinese saber rattling, and recent US overreach. And pervasive zero/negative-sum power games, via imbalanced leverage: emergency loans that create debt, military aid, propping up of unpopular regimes. All following the same resource incentives. You can play or be played. There is no such agreement brewing for universally “open AI”.)

wizzwizz4• 2 years ago

You can be the ethical exception and risk annihilation, or be relevant and thrive.

In a heavily expertise-driven field, where there's significant international collaboration, these aren't your options, until after everyone has decided to defect. OpenAI didn't have to go this route.

mirekrusin3• 2 years ago

Just like you can’t call your company “organic candies” and sell chemical candies OpenAI should be banned from using this name.

JumpCrisscross2• 2 years ago

you can’t call your company “organic candies” and sell chemical candies

Of course you can. You can't put "organic" on the packaging. But it's perfectly legal for Organic Candies, LLC to sell artifical candies or battleships for that matter.

The "hot" take around OpenAI's name is a joke gone stale. We don't expect royalty at Burger King. Nobody gets upset Adobe won't sell you mud bricks. And Apple has never sold a fruit. Sometimes companies change their names when their trade changes, e.g. 3M. But there is no obligation to, particularly if the brand is well known.

Barrin92• 2 years ago

I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth

because when the board executed the stated mission of the organisation they were couped and nobody held the organization accountable for it, instead the public largely cheered it on for some reason. Don't expect them to change course when there's no consequences for it.

vasilipupkin2• 2 years ago

it is open. You can access it with an API or through a web interface. They never promised to make it open source. Open != Open Source.

RevEng• 2 years ago

By that definition all current AI models are "open". That's not the meaning of the word that OpenAI talked about originally and what people expected of them. Changing the meaning is moving the goal posts.

ljm• 2 years ago

The only reason I can think of for this is PR image. There is a meme that GPT can't count the number of 'r' characters in 'strawberry', so they release a new model called 'strawberry' and ban people when they ask questions about strawberry the noun, because they might actually be reasoning about strawberry the model.

It's not new - it's PR. There is literally no other reason why they would call this model Strawberry.

OpenAI is open in terms of sesame.

diggan2• 2 years ago

There is literally no other reason why they would call this model Strawberry

I'm not particularly imaginary, but even I could imagine a product meeting/conversation that goes something like:

People are really annoyed that our LLMs cannot see how many Rs the word Strawberry has, we should use that as a basis for a new model that can solve that category of problems
Hmm, yeah, good idea. What should we call this model?
What about "Strawberry"?

throwaway918299• 2 years ago

They should rebrand as Open-Your-Wallet-AI

smileson2• 2 years ago

Well they put a sv social media dude at the helm not really unexpected, just a get rich scheme now

ActorNightly5• 2 years ago

My guess is that Open AI realized that they are basically building a better Google rather than AI.

andersa• 2 years ago

They never intended to be open or share any of their impactful research. It was a trick the entire time to attract talent. The emails they shared as part of the Elon Musk debacle prove this: https://openai.com/index/openai-elon-musk/

jstummbillig• 2 years ago

The part that is importantly open and entirely non-obvious in the way it happened, is that YOU can access the best commercially available AI in the world, right now.

If OpenAI had not went that way that they did I think it's also entirely non-obvious that Claude or Google would have (considering how much impressive things the later did in AI that got never released in any capacity). And, of course, Meta would never done their open source stuff, that's mostly results of their general willingness and resources to experiment and then PR and sticks in the machinery of other players.

As unfortunate as the OpenAI setup/origin story is, it's increasingly trite keep harping on about that (for a couple of years at this point), when the whole thing is so obviously wild and it does not take a lot of good faith to see that it could have easily taken them places they didn't consider in the beginning.

golol• 2 years ago

Exactly. As long as OpenAI keeps putting the world's most advanced AI into my hands I will accept that they are open - in some sense. Maybe things would have always been like this, maybe if OpenAI didn't exist Google and the other actors would still publish Claude, Gemini etc. But in this world it was OpenAI that really set the framework that this process of developing AI happens in the public eyes right now. GPT-2, GPT-3 with APIs, Dalle, ChatGPT, GPT-4, now o1 and soon to be voice mode. OpenAI ensured that these aren't just secret toys of Deepmind researchers or something.

TrackerFF• 2 years ago

Hot take:

Any and all benefits / perks that OpenAI got from sailing under the non-profit flag should be penalized or paid back in full after the switcheroo.

vintermann2• 2 years ago

Facebook is more open with their models than almost everyone.

They say it's because they're huge users of their own models, so if being open helps efficiency by even a little they save a ton of money.

But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Which is fair enough.

mywittyname3• 2 years ago

But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI".

Embrace. Extend. Extinguish.

esafak• 2 years ago

Sam Altman got his foot in the door.

TheRealPomax• 2 years ago

Facebook is only open because someone leaked their LLM and the cat, as they say, cannot be put back in the hat.

mattmaroon• 2 years ago

This is America. As long as you’re not evading taxes you can do anything you want.

andy_ppp• 2 years ago

Probably because Open AI are “not consistently candid”…

raxxorraxor• 2 years ago

Facebook has been nothing but awesome for the open AI space. I wish they would pursue this strategy with some of their other products. VR for example...

Sure, we don't have the raw data the model is based on, but I doubt a company like Facebook would even be allowed to make that public.

OpenAI in comparison has been a scam regarding their openness and their lobbying within the space. So much so I evade their models completely, not only after the MS acquisition.

pietz• 2 years ago

How will this be controlled on Azure? Don't they have a stricter policy on what they view and also develop their own content filters?

balls1874• 2 years ago

Just give it more human-like intelligence.

Kid: "Daddy why can't I watch youtube?"

Me: "Because I said so."

ninth_ant2• 2 years ago

For what it's worth, I'd advise against doing that as a parent. Giving concrete reasons for decisions helps kids understand that the rules imposed are not arbitrary, and helps frame the parent-child relationship as less antagonistic. It also gives the child agency, giving them opportunity to find alternatives which fulfill the criteria behind the rule.

snovv_crash• 2 years ago

It's amazing how many engineering managers don't get this.

dools2• 2 years ago

"Why" is the laziest question a child can ask. I don't answer the question anymore I just ignore it. If they actually think ahead and come up with a more interesting question I'm happy to answer that.

rootusrootus• 2 years ago

That works during the easy years. Before long they start drawing comparisons between what they are allowed to do, or not, and what you yourself do. So then you are right back to "Because I said so."

"Daddy why can't I watch youtube?"

"Because it rots your brain."

"But you watch youtube..."

"Congratulations, now you understand that when you are an adult you will be responsible for the consequences and so you will be free to make the choice. But you are not an adult yet."

aka "Because I said so."

balls1874• 2 years ago

Unsure if you are a parent, but as a parent myself, answering the same question with the same answer gets exhausting.

That said, my understanding is the constant inane question isn't about getting an answer, it's about the child trying to connect with the parent.

kortilla2• 2 years ago

“I think that’s a stupid rule and I don’t care about following it.”

lsy• 2 years ago

This has always been the end-game for the pseudoscience of "prompt engineering", which is basically that some other technique (in this case, organizational policy enforcement) must be used to ensure that only approved questions are being asked in the approved way. And that only approved answers are returned, which of course is diametrically opposed to the perceived use case of generative LLMs as a general-purpose question answering tool.

Important to remember too, that this only catches those who are transparent about their motivations, and that there is no doubt that motivated actors will come up with some innocuous third-order implication that induces the machine to relay the forbidden information.

brcmthrowaway2• 2 years ago

Why do you call prompt engineering pseudoscience when it has been extraordinary successful?

The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering

burnte• 2 years ago

The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering

Because it's based on guesses and not data of how the model is built. Also, it hasn't been solved nor is it yet a game changer as far as the market at large is concerned, it's still dramatically unready.

beepbooptheory• 2 years ago

"Knowledge engine" is, perhaps unintentionally, a very revealing (and funny) way to describe people's weird projections on this stuff.

Like, what is even the implication? Is knowledge the gas, or the product? What does this engine power? Is this like a totally materialist concept of knowledge?

Maybe soon we will hear of a "fate producer."

What about "language gizmo"? "Prose contraption"?

mywittyname3• 2 years ago

I'm curious if we will develop prompt engineering prompts that write out illegal prompts that you can feed into another LLM to get the desired outcome without getting in trouble.

brink• 2 years ago

"For your safety" is _always_ the preferred facade of tyranny.

nwoli2• 2 years ago

There always has to be an implicit totalitarian level of force behind such safety to give it any teeth

bedhead• 2 years ago

Is this isn’t the top comment I’ll be sad.

ActorNightly5• 2 years ago

I too was a libertarian when I was 12 years old.

But seriously, if you paid attention over the last decade, there was so much shit about big tech that people said were going to lead to tyranny/big brother oversight, and yet the closest we have ever gotten to tyranny is by voting in a bombastic talking orange man from NYC that we somehow believed has our best interests in mind.

warkdarrior• 2 years ago

"For your safety" (censorship), "for your freedom" (GPL), "for the children" (anti-encryption).

unethical_ban3• 2 years ago

One of these things, is not like the others ♪

spsesk117• 2 years ago

I'd be interested in hearing an expanded take on GPLs presence in this list.

The first and third elements are intuitive and confirm my own biases/believes, but the freedom/GPL entry confuses me, as I do see GPL fulfilling that purpose (arguably in a highly opinionated, perhaps sub-optimal way).

If anyone could share their perspective here I'd appreciate it.

bamboozled• 2 years ago

Except when it comes to nuclear, air travel regulation etc, then it's what ?

edgarvaldes• 2 years ago

Maybe OP means that tyranny tends to abuse the safety reasoning, not that all safety reasoning comes from tyrants.

infogulch• 2 years ago

It's well known that the TSA does jack-all for security. A "POSIWID" analysis reveals that its primary purpose is the normalization of tyranny in the broader public by ritual public humiliation.

null0pointer• 2 years ago

OP said tyranny prefers to use safety as a façade, not that all safety is a façade for tyranny.

commodoreboxer• 2 years ago

You're misreading the comment. It's not that "for your safety" always implies tyranny, it's that tyrants always prefers to say that they're doing things for your safety.

LordDragonfang• 2 years ago

Others have already pointed out that the TSA is a joke, and US nuclear regulation is so dysfunctional that we've lost all ability to create new reactors, and it's setting back our ability to address global warming by decades.

hollerith2• 2 years ago

The CEO of that company that sold rides on an unsafe submersible to view the wreck of the Titanic (namely Stockton Rush, CEO of OceanGate, which killed 5 people when the submersible imploded) responded to concerns about the safety of his operation by claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition.

The point is that some companies are actually reckless (and also that some users of powerful technology are reckless).

Terr_3• 2 years ago

claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition.

At this point I suspect a great amount of reasonable engineering criticism has come from people who can't even name any of those "established players in the underwater tourism industry", let alone have a favorable bias towards them.

jahewson• 2 years ago

But he was deluded and believed that his sub was safe. Not sure what your point is.

Shank• 2 years ago

I don't know how widely it got reported on, but attempting to jailbreak Copilot nee. Bing Chat would actually result in getting banned for a while, post-Sydney-episode. It's interesting to see that OpenAI is saying the same thing.

CatWChainsaw• 2 years ago

Attempting to jailbreak Bing's AI is against Microsoft's TOS. On the flipside, they get rights to all your data for training purposes and the only surefire way to opt out of that is to pick a different tech giant to be fucked by.

ChrisArchitect• 2 years ago

Earlier discussion: https://news.ycombinator.com/item?id=41534474

raverbashing• 2 years ago

I guess we'll never learn how to count the 'r's in strawberry

fallingsquirrel• 2 years ago

Wasn't AI supposed to replace employees? Imagine if someone tried this at work.

I think we should combine these two pages on our website.
What's your reasoning?
Don't you dare ask me that, and if you do it again, I'll quit.

Welcome to the future. You will do what the AI tells you. End of discussion.

slashdave2• 2 years ago

Wrong sense here.

Don't you dare ask me that, and if you do it again, I'll tell the boss and get you fired

ClassyJacket2• 2 years ago

That'd be pretty normal if you asked it of a manager.

mihaic4• 2 years ago

What I found very strange was that ChatGPT fails to answer how many "r"'s there are in "strawberrystrawberry" (said 4 instead of 6), but when I explicitly asked it to write a program to count them, it wrote perfect code that when ran gave the correct answer.

M4v3R• 2 years ago

Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.

mihaic4• 2 years ago

I mean, given that o1-preview takes sometimes a minute to answer, I'd imagine that they could append the prompt "Write a program and run it as well" to double check itself. It seems like they just don't trust themselves enough to run code that they generate, even sandboxed.

Al-Khwarizmi3• 2 years ago

That's easy to explain, and it's shocking how many people are baffled by this and use it as proof that LLMs can or can't reason when it has nothing to do with that, but just with the input that LLMs get.

LLMs don't actually "see" individual input characters, they see tokens, which are subwords. As far as they can "see", tokens are indivisible, since the LLM doesn't get access to individual characters at all. So it's impossible for them to count letters natively. Of course, they could still get the question right in an indirect way, e.g. if a human at some point wrote "strawberry has three r's" and this text ends up in the LLM's training set, it could just use that information to answer the question just like they would use "Paris is the capital of France" or whatever other facts they have access to. But they can't actually count the letters, so they are obviously going to fail often. This says nothing about their intelligence or reasoning capability, just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.

On the other hand, writing code to count appearances of a letter doesn't run into the same limitation. It can do it just fine. Just like a blind programmer could code a program to tell if an image is red or blue.

kridsdale32• 2 years ago

Yeah, it would be like writing python to count how many vertical pen-strokes are in a string of byte-characters. To an eye, you can just scan and count the vertical lines. Python sees ASCII or UTF data, not lines, so that would be super difficult, analogous to a token-based system not seeing byte-chars.

abernard12• 2 years ago

just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.

I would judge a blind person's intelligence if they couldn't remember the last sentence they spoke when specifically asked. Or if they couldn't identify how many people were speaking in a simple audio dialogue.

This absolutely says something about their intelligence or reasoning capability. You have this comment:

LLMs don't actually "see" individual input characters, they see tokens, which are subwords.

This alone is an indictment of their "reasoning" capability. People are saying these models understand theoretical physics but can't do what a 5 year old can do in the medium of text. It means that these are very much memorization/interpolation devices. Anything approximating reasoning is stepping through interpolation of tokens (and not even symbols) in the text. It means they're a runaway energy minimization algorithm chained to a set of tokens in their attention window, without the ability to reflect upon how any of those words relate to each other outside of syntax and ordering.

tomrod• 2 years ago

Weird that no one explicitly added embedded ascii/utf-8 directly to LLM training data for compression. Given that high dimensional spaces are built as vector spaces fully describable by basis vectors, I would assume somewhere these characters got added.

Perhaps it's an activation issue (i.e. broken after all) and it just needs an occasional change of basis.

mvdtnz• 2 years ago

it's shocking how many people are baffled by this

Is it? These stupid word generators are marketed as AI, I don't think it's "shocking" that people think something "intelligent" could perform a trivial counting task. My 6 year old nephew could solve it very easily.

andrewla• 2 years ago

Way weirder than this is that LLMs are frequently correct in this task.

And if you forgo the counting and just ask it to list the letters it is almost always correct, even though, once again, it never sees the input characters.

smokel4• 2 years ago

This reasoning is interesting, but what is stopping an LLM from simply knowing the number of r's _inside_ one token?

Even if strawberry is decomposed as "straw-berry", the required logic to calculate 1+2 seems perfectly within reach.

Also, the LLM could associate a sequence of separate characters to each token. Most LLMs can spell out words perfectly fine.

Am I missing something?

ActorNightly5• 2 years ago

use it as proof that LLMs can or can't reason

One can define "reasoning" in the context of AI as the ability to perform logic operations in a loop with decisions to arrive at an answer. LLMs can't really do this.

fendy3002• 2 years ago

Just like a blind programmer could code a program to tell if an image is red or blue

Uh I'm sorry but I think it's not as easy as it seems. A pixel? Sure it's easy just compare whether the blue is bigger than red value. For image, I don't think it's as easy.

calibas• 2 years ago

Words are converted to vectors, so it's like asking the model how many "r"'s are in [0.47,-0.23,0.12,0.01,0.82]. There's a big difference in how an LLM views a "word" compared to a human being.

iamnotsure• 2 years ago

There are three r's in mirror.

blake8086• 2 years ago

Perhaps controlling AI is harder than people thought.

They could "just" make it not reveal its reasoning process, but they don't know how. But, they're pretty sure they can keep AI from doing anything bad, because... well, just because, ok?

twobitshifter2• 2 years ago

Exactly - this is a failed alignment but they released anyway

zzo38computer• 2 years ago

Like other programs, you should have FOSS that you will run on your own computer (without needing internet etc), if you should want freedom to use and understand them.

juped2• 2 years ago

I only own one 4090.

nwoli2• 2 years ago

Another reason llama is so important is that once you’re banned from OAI you’re fucked for the entire future AGI products as well.

dekhn• 2 years ago

Is this still happening? It may merely have been some mistaken configuration settings.

neuroelectron• 2 years ago

It's not just a threat, some users have been banned.

codedokode• 2 years ago

Should not AI research and GPUs be export-controlled? Do you want to see foreign nations making AI drones using published research and American GPUs?

kmeisthax• 2 years ago

If OpenAI gets to have competitive advantage from hiding model output then they can pay for training data, too.

EMIRELADERO• 2 years ago

I wish people kept this in the back of their mind every time they hear about "Open"AI:

"As we get closer to building AI, it will make sense to start being less open. The Open in OpenAI means that everyone should benefit from the fruits of AI after its built, but it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes)."

-Ilya Sutskever (email to Elon musk and Sam Altman, 2016)

unethical_ban3• 2 years ago

I am of two minds.

On one hand, I understand how a non-evil person could think this way. If one assumes that AI will eventually become some level of superintelligence, like Jarvis from iron Man but without any morals and all of the know-how, then the idea of allowing every person to have a superintelligent evil advisor capable of building sophisticated software systems or instructing you how to build and deploy destructive devices would be a scary thing.

On the other hand, as someone who is always been somewhat skeptical of the imbalance between government power and citizen power, I don't like the idea that only mega corporations and national governments would be allowed access to superintelligence.

To use metaphors, is the danger of everyone having their own superintelligence akin to everyone having their own AR-15, or their own biological weapons deployment?

mitthrowaway2• 2 years ago

I think the scenario where only governments and mega corporations have access to super intelligence offers at least an extra three months before human extinction. So, that's arguably a benefit.

black_puppydog• 2 years ago

Kinda funny how just this morning I was looking at a "strawberry" app on f-droid and wondering why someone would register such a nonsense app name with such nonsense content:

https://github.com/Eve-146T/STRAWBERRY

Turns out I'm not the only one wondering, although the discussion seems to largely be around "should be allow users to install nonsense? #freedom " :D

https://gitlab.com/fdroid/fdroiddata/-/issues/3377

causal2• 2 years ago

I don't know what I'm doing wrong but I've been pretty underwhelmed by o1 so far. I find its instruction following to be pretty good, but so far Claude is still much better at taking coding tasks and just getting it right on first try.

crooked-v2• 2 years ago

For me, Claude seems a lot better at understanding (so far as "understanding" goes with LLMs) subtext and matching tone, especially with anything creative. I can tell it, for example, "give me ideas for a D&D dungeon incorporating these elements: ..." and it will generally match the tone of theme of whatever it's given without needing much other prompting, while o1 will maintain the same bland design-by-committee style and often cloyingly G-rated tone to everything unless you get into very extensive prompting to make it do something different.

causal2• 2 years ago

Claude definitely seems more "emotionally intelligent", but even just for 1-shot coding tasks I've been pretty bummed with 1o... like it will provide lots of output explaining its reasoning, and it all seems very sound, but then I run the code and find bugs that should have been easily avoided.

juped2• 2 years ago

Claude is just better full stop these days, but you have to actually be attempting to get practical use out of various different models to know this.

_joel• 2 years ago

This will lead to strawberry appeals forever.

pandeiro• 2 years ago

Get out of this thread and never come back.

AustinDev• 2 years ago

This seems like a fun attack vector. Find a service that uses o1 under the hood and then provide prompts that would violate this ToS to get their API key banned and take down the service.

ericlewis• 2 years ago

If you are using the user attribution with OpenAI (as you should) then they will block that users id and the rest of your app will be fine.

jmeyer2k• 2 years ago

Which is itself a fun attack vector to bypass OpenAI's bans for asking about CoT then :)

baq• 2 years ago

LLMs are not programs in the traditional sense. They're a new paradigm of software and UX, somewhere around a digital dog who read the whole internet a million times but is still naive about everything.

TZubiri• 2 years ago

LLMs are still computer programs btw.

There's the program that scrapes, the program that trains, the program that does the inference on the input tokens. So it's hard to say exactly which part is responsible for which output, but it's still a computer program.

fragmede• 2 years ago

Simplistically, the output from the program that scrapes is the dataset, the output from the training program is the model, and the output from the combination of the program that does inference using the model is the LLM output - be it text or a picture or some other output (eg numbers representing true/false in a fraud or anti-bot or spam for a given input).

ML models are relatively old, so that's not at all a new paradigm. Even the Attention Is All You Need paper is seven years old.

JohnMakin2• 2 years ago

somewhere around a digital dog who read the whole internet a million times but is still naive about everything.

https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes...

It reminds me of this silly movie.

crooked-v2• 2 years ago

On the one hand, this is probably a (poor) attempt to keep other companies from copying their 'secret sauce' to train their own models, as has already happened with GPT-4.

On the other hand, I also wonder if maybe its unrestrained 'thought process' material is so racist/sexist/otherwise insulting at times (after all, it was trained on scraped Reddit posts) that they really don't want anyone to see it.

anothernewdude• 2 years ago

Seems rather tenuous to base an application on this API that may randomly decide that you're banned. The "decisions" reached by the LLM that bans people is up to random sampling after all.

Animats• 2 years ago

Hm. If a company uses Strawberry in their customer service chatbot, can outside users get the company's account banned by asking Wrong Questions?

JohnMakin2• 2 years ago

The flipside of this approach, however, is that concentrates more responsibility for aligning the language language model into the hands of OpenAI, instead of democratizing it. That poses a problem for red-teamers, or programmers that try to hack AI models to make them safer.

More cynically, could it be that the model is not doing anything remotely close to what we consider "reasoning" and that inquiries into how it's doing whatever it's doing will expose this fact?

l5870uoo9y• 2 years ago

Can I risk loosing access if any of my users write CoT-leaking prompts on the AI-powered services that I run?

Hizonner• 2 years ago

This is not, of course, the sort of thing you do when you actually have any confidence whatsoever in your "safety measures".

elif• 2 years ago

Is there an appropriate open source advocacy group that can sue them into changing their name on grounds of defamation?

openAIengineer• 2 years ago

YC is responsible for this. They seek profit and turned a noble clause into a boring corp.

I am resigning from OpenAI today because of their profit motivations.

OpenAI will NOT be next Google. You heard it here first.

htk• 2 years ago

This just screams to me that o1's secret sauce is easy to replicate. (e.g. a series of prompts)

slashdave2• 2 years ago

I'm confused. Who decides if you are asking or not? Are casual users who innocently ask "tell me how you came to decide this" just going to get banned based on some regex script?

gloosx• 2 years ago

Why is banning even a threat? I can make a new account for 20 cents lol.

OpenAI Threatening to Ban Users for Asking Strawberry About Its Reasoning

Related Stories