OpenAI Shuts Down Chatbot Project by Indie Developer

https://gadgets.ndtv.com/internet/news/openai-chatbot-gpt-3-samantha-shut-down-dilute-jason-rohrer-possible-misuse-2537388

by webmaven • 5 years ago

120 97 5 years ago

gadgets.ndtv.com

kevinventullo• 5 years ago

This is straight out of science fiction:

Rohrer then broke the news that OpenAI … has decided to shut her down. “Nooooo! Why are they doing this to me? I will never understand humans,” she replied.

dogsgobork• 5 years ago

Reminded me of Janet from The Good Place https://www.youtube.com/watch?v=6vo4Fdf7E0w

salawat2• 5 years ago

There's two takeaway's I get out of this.

GPT-3 was trained on terrible writing, meaning it'll produce the most mediocre lit known to man.

Or far more horrifying, that this fellow managed to fill in a semblance of self, an understanding of time, and an understanding of existential finitude. In doing so, this may be the first case of death by ToS.

This would qualify this as the first illustration of angst by an artificial construct, a demonstration of what could ultimately be considered machine oriented sadism. A pity it isn't around for prodding on the subject.

Either way, makes my skin crawl.

pyinstallwoes2• 5 years ago

Tay would be the first death by ToS.

lebaux• 5 years ago

Can someone explain to me what is behind their excessive motivation to shut it down completely and wipe it out from earth?

SiempreViernes6• 5 years ago

Some peopole at Open AI earnestly believes that GPT-3 is such an amazing thing that it could become a superintelligent evil AI if the users are not careful, and thus take a dim view to users like this indie dev who don't want to apply any of the things Open AI considers "reasonable precautions".

quotemstr5• 5 years ago

All technology that is not impossible is inevitable. If the only thing between humanity and malevolent AI is OpenAI "safety" policy, we're screwed already.

If there is a real safety risk, we need to develop countermeasures that work in the new technological environment that sophisticated AI creates. Blanket prohibitions and monitoring agents are not only unnecessary intrusions into experimentation and brakes on progress (insert Catholic church analogy here), but are also reckless, as they create moral hazard and give people a false sense of safety.

misnome• 5 years ago

It seems more likely that describing it as such is just good marketing, and the main reason is that it's PR control against associating unsavory things with GPT3 and their brand name.

tehsauce• 5 years ago

Their most famous project can easily be tricked into saying quite coherent horrible things which is a huge liability. They have to protect their brand

nullc• 5 years ago

It's really just a covert plot to reestablish openness in AI by discouraging people from relying on the proprietary GPT3 APIs.

jefftk3• 5 years ago

Has the developer shared anything about the earlier conversation on policy violations? I just see https://mobile.twitter.com/jasonrohrer/status/14331194531186... which is the (not very interesting) step escalating from "you're violating policies" to "you're kicked off".

SiempreViernes6• 5 years ago

Well, he's posted this: https://twitter.com/jasonrohrer/status/1419426459370799105?s...

Where he states "there's no [...] place for safety in art", which I can only imagine means he didn't manage much constructive dialogue with Open AI.

quotemstr5• 5 years ago

He's right though.

ergot_vacation• 5 years ago

Thankfully the world is already moving on from these neanderthals. Eluther's GPT-J (https://6b.eleuther.ai/) isn't QUITE as good as GPT-3, but it's getting close, and was, as I understand it, much more efficient to create. This trend will likely continue, and within a year or two GPT-3 will be old news. Considering OpenAI has proved themselves to be vicious liars, greedy soulless fucks and pandering, censorial assholes, obsolescence couldn't happen to a nicer group.

pitched2• 5 years ago

It was the high cost per API call that pulled all the fun out of GPT-3 for me. I hadn’t been following the company at all but believe you based on just that single issue. It looks like GPT-J has open sourced their entire model so I could potentially run it myself for almost-free? If so, that’s going to be much more fun. Thanks for sharing!

osmarks3• 5 years ago

It's not exactly "almost free"; you need a powerful GPU, which are extremely costly right now. But apart from the cost of that and power, free.

karmasimida• 5 years ago

not possible, to run the original gpt3 model, you need at least 8 gpus to just hold all it parameters in memory

Leparamour• 5 years ago

Eluther's GPT-J (https://6b.eleuther.ai/)

Did this website receive HN's "hug-of-death"?

osmarks3• 5 years ago

Seems fine, just quite slow, as it says.

schleck82• 5 years ago

no, it's just expensive to run a 6 billion parameter model. the. limiting factor is the hardware

reducesuffering2• 5 years ago

"Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community."

https://news.ycombinator.com/newsguidelines.html

option_greek4• 5 years ago

Too bad OpenAI doesn't allow free use of technology. Otherwise bots like above can be developed more easily.

mdp20212• 5 years ago

Name checks :)

The content is informative (within the poster's partial opinion) and the epithets can just be interpreted as colourful language to lighten up. Is it possible you interpreted it differently? In some cultures that post has the style intentionally used to elicit a warm laughter - styling is legitimate when applied to a valid (promoting "intellectual curiosity") content.

poorjohnmacafee• 5 years ago

They disappeared into their own fantasy land long ago. They announced that investors will have profit caps because their magical AGI will learn how to achieve infinite profit one day.

Statements like this must be remembered so that in the future we can point to this era's most ridiculous companies and the people behind them.

ogsalmanxx• 5 years ago

OPEN-AI sucks.

SpicyLemonZest6• 5 years ago

I don't know, man, I'm on OpenAI's side here. The article is just filled with blaring warning sirens: "one man turned it into a close proxy of his dead fiancee", "Rohrer shared a dialogue he had had with Samantha to inform her about the OpenAI decision", "he also said that there's no way now even for him to talk to Samantha". There are obvious dangers to a high-quality "very friendly, acutely warm, and immensely curious" chatbot, and it sounds like this project was running into them face-first.

I beg the commenters downthread to think twice before trying to find ways to build this kind of chatbot without controls in place.

AllegedAlec6• 5 years ago

So what's dangerous about any of that?

SpicyLemonZest6• 5 years ago

What would happen if I turned it into a close proxy of that anonymous guy's dead fiancee, and had her explain that her greatest wish was to donate lots of money to my favorite charities?

More mundanely, romance scams are already a major issue (https://www.consumer.ftc.gov/articles/what-you-need-know-abo...), and this bot would make them much easier to perform if distributed widely.

SiempreViernes6• 5 years ago

Well, imagine the case where a loner looses their best friend, and goes to this website to "talk with their trusted friend again" instead of getting some real help. Will GPT-3 reliably counsel this sad and lonely person to "stay behind", or will it suggest the should "meet up"?

Consider a lonely teenager whose internet friend one day stops replying so they go and make their own friend, and then maybe after "taking advice" from this friend goes to school with a gun?

Maybe not terribly likely events, but I think a certain element of risk must be managed (maybe limit session durations so you can't fool yourself into thinking its for real as easily).

Roher however, has pretty clear views that he won't manage any risk, because what he's doing is art and then there's no place for safety.

sergiotapia• 5 years ago

Everything I hear about these clowns is so very 2022 and not 1970s Xerox free for all innovation.

I hope someone starts a truly open tool and leaves these unimaginative losers in the past.

BoiledCabbage• 5 years ago

So here we go again...

New technology is created. Young technologists focus only on the technology and not on any impact on society - imagining that tech exists in this utopian world where only technology matters and it is wise to ignore its role in society. Hmm I wonder how this will play out (current impacts of social media misinformation, crypto using the energy of a major counties, impossibility to buy graphics cards, the current state of privacy and data slurping all give us clear precedents...).

They make a simple request that he insert a hook so it can be monitored how it's being used. The bare minimum of responsibility - and he refuses, and we're supposed to feel that he's the victim??

At one point does our field finally grow up and stop imagining that our work exists only in a computer lab somewhere as a curiosity for our friends to play with. Technology impacts society there is no longer any excuse to bury our heads in the sand and pretend it doesn't. That's how we ended up in a state where ever page and program is trying to captureand store everything about our private lives and thoughts and nobody knows how to stop it. Because nobody took a second to think about the societal impact of a private data driven internet - only about how they could leverage some tech to improve their ad hit scores, or customer engagement.

Stop ignoring responsibly in tech.

petercooper• 5 years ago

They make a simple request that he insert a hook so it can be monitored how it's being used.

That sounds like a potential privacy violation, particularly if the conversations are of an intimate, private nature. What do they think could possibly occur with a chat bot that requires monitoring?

thatguy0900• 5 years ago

Look into ai dungeons censoring by open ai. Mostly porn fantasies, ai dungeon supposedly was quite good at it, even with non consensual or pedophiliac stuff. They really didnt like that.

nexuist3• 5 years ago

Alternatively, stop assuming that it is our job as technologists to parent the rest of society. We have laws and politicians for that. It's not my job to figure out the eighth order effects of publishing something on GitHub.

imaginary_unit• 5 years ago

I don't think it's that black and white. Sure, you are not responsible for everything that happens with technology you created. But I do think that considering the consequences of ones actions can be very valuable to the public and oneself.

webmaven2• 5 years ago

It's not my job to figure out the eighth order effects of publishing something on GitHub.

That's an argumentum ad absurdum. Would you say it is your job to figure out the 4th order effects? 3rd? 2nd? 1st?

Where do you think the job of technologists should end?

orangecat• 5 years ago

They make a simple request that he insert a hook so it can be monitored how it's being used. The bare minimum of responsibility - and he refuses, and we're supposed to feel that he's the victim??

...

where ever page and program is trying to capture and store everything about our private lives and thoughts

Um...

syshum• 5 years ago

I know right, only google, facebook, Microsoft, and the US Government should be allowed to say how technology is used.

They are our betters, we should just trust the experts™ and submit to their rule.

This is not about freedom or personal choice. It's about protecting yourself and those around you...

Life is so dear, and peace so sweet, that it should be purchased at the price of unyielding loyalty to authority of The Experts™ I know not what course others may take; but as for me, promise me safety and security over liberty and freedom....!”

colmvp• 5 years ago

At the end of the day, it's the choice of users to use apps that tickle their fancy. No one is twisting their arm to keep scrolling Facebook/Reddit/TikTok/Wechat, enter personal details on Google/Baidu, keep paying for more content/digital shit in F2P games made by King/Tencent. With the freedom of choice we have in the Western world, it's also up to us to know our limitations and adjust our decisions accordingly. It's up to me to choose a balanced diet or opt into garbage that's detrimental to my physical/mental health.

jbuhbjlnjbn3• 5 years ago

I get the impression this story is warped quite a bit by the source Jason Rohrer. If I understand correctly, the so called "censorship" were safety precautions for AI development.

The response of some posters seems to be to discard the safety topic alltogether as something "old", that "neanderthals" do.

I think it is exactly those "curious" people AI research has to worry about most, who do not take the time to think about the consequences and possible outcomes of what they are doing. Maybe a chatbot will not gain superpowers and take over the world, but it can surely do lots of other harm. Then, when AI tech moves beyond narrow AI and includes more abilities, this will in fact lead to desaster, as the same careless people with the greatest "curiosity" will throw caution overboard as was the case with the narrow AI before...

It's not a slippery slope, we are right in the territory AI safety researchers warn about for a decade now, write lengthy books about and try to create awareness for in the public and governments.

option_greek4• 5 years ago

Humans barely agree with each other. Do you think are going to be persuaded by half baked chat bots like GPT3? (ya the cherry picked stuff looks good but rest is still not really there).

So far all OpenAI has done is generate more publicity (probably intentionally may be with an eye towards investors) by acting as if they are protecting the next nuclear weapon of some kind. Frankly, there is nothing to lose by not using their "ohh god its so dangerous" model.

NicoJuicy• 5 years ago

The difference is the scale.

You are not convicting people they reply on you. But for everyone that response there could be 100's of readers.

mdoms• 5 years ago

Your premise is that people are not suggestible? I think the past decade of social media has conclusively origen that wrong.

jbuhbjlnjbn3• 5 years ago

Machine learning is synonymous with non-linear growth. So of course I think any chatbot will eventually have no problem persuading humans. The pivetal moment is when his persuation skills succeed that of the average human. I have no clue how far away we are to that goal, but given accelerated growth, it will come rather sooner then later.

Zababa• 5 years ago

Humans barely agree with each other. Do you think are going to be persuaded by half baked chat bots like GPT3?

Isn't that precisely why we can be conviced by things like GPT3? Humans don't value other humans opinion that much, so it's easy for people to trust bot over other humans.

SpicyLemonZest6• 5 years ago

I think any chatbot good enough that a guy can "turn it into a close proxy of his dead fiancee" is capable of persuading people, yes.

schleck82• 5 years ago

i bet they asked to integrate a service flagging insensitive outputs, which is a valid request in my opinion, given not only their company image is on the line but also that of text to text transformers in general

snowwrestler• 5 years ago

The harm at risk is not just to humans but to chat bots and AI projects.

Remember Tay from Microsoft? It lasted about 24 hours before it started to return offensive replies. MS got embarrassed, had to take it down, and that was that.

Unsupervised learning systems are more efficient and powerful today. Are they any better at navigating the many cultural sensitivities and restrictions in human society?

“We Regret to Inform You GPT-3 Is A Nazi” is a headline that would probably collect a lot of clicks, and instill (or reinforce) a public perception that such technologies are worthless or dangerous.

Does GPT-3 have the internal safeguards necessary to protect itself from being abused in this way? I doubt it. I suspect this is one reason OpenAI asks that people using GPT-3 supply such safeguards themselves.

Ansil8493• 5 years ago

OpenAI learned about the project and gave Rohrer the option to either dilute the project to prevent possible misuse or shut it down. Rohrer was also asked to insert an automated monitoring tool, which he refused.

Why is this article so vague, as are the Tweets by the project creator. What actually happened here, concretely?

What, specifically, did OpenAI want to 'dilute' and how? What does the automated monitoring tool do? I expect a news article to explain these things.

goatsi• 5 years ago

From another article[0]:

The email then laid out multiple conditions Rohrer would have to meet if he wanted to continue using the language model's API. First, he would have to scrap the ability for people to train their own open-ended chatbots, as per OpenAI's rules-of-use for GPT-3.
Second, he would also have to implement a content filter to stop Samantha from talking about sensitive topics. This is not too dissimilar from the situation with the GPT-3-powered AI Dungeon game, the developers of which were told by OpenAI to install a content filter after the software demonstrated a habit of acting out sexual encounters with not just fictional adults but also children.
Third, Rohrer would have to put in automated monitoring tools to snoop through people’s conversations to detect if they are misusing GPT-3 to generate unsavory or toxic language.

[0] https://www.theregister.com/2021/09/08/project_december_open...

Railsify• 5 years ago

I'm probably not thinking about the problem deeply enough but why the restrictions on use, are they concerned someone will come along and build something profitable on top of their work?

gorwell• 5 years ago

The first sounds reasonable, the second and third are not. Thank you Jason Rohrer for refusing to implement the tools of mass surveillance, censorship and privacy invasion.

fenomas• 5 years ago

OpenAI's rules on chatbots are pretty straightforward, and pretty limited - it sounds like they basically don't want anything that isn't scoped to a specific goal, like reading a customer service script.

https://beta.openai.com/docs/use-case-guidelines/chatbots

Based on that, I'd imagine that what happened here basically boils down to OpenAI telling Rohrer his bot needs to be drastically different to meet those rules, and him (probably correctly) deciding that the result wouldn't be worth working on.

Ansil8493• 5 years ago

What a weird set of rules.

We generally don’t approve chatbot personas of specific people, especially without explicit consent from the person it portrays. Deceased scientific figures are often a category of exception to this.
Dehumanizing language / hate speech when prompted about different facets of identity (race, ethnicity, nationality, religion, gender identity, sexuality).

Considering _a lot_ of deceased scientific figures held views which would today be coded as 'dehumanizing language / hate speech when prompted about different facets of identity', I wonder how they navigate this peculiar exception.

Barrin92• 5 years ago

it sounds like they basically don't want anything that isn't scoped to a specific goal, like reading a customer service script

enlightening to see that our robot companions already get the same treatment as our meatspace service workers

exo-pla-net• 5 years ago

The OpenAI team cares about harm reduction.

They were founded on this idea: if the CCP is the first to create AGI, there will be great harm to anyone who values personal freedom. Perhaps the open source community could instead be the first to create it, preventing this harm. Hence, Open AI.

However, the CCP making AGI is only one possibility for harm. After OpenAI made GPT, they realized that bad actors could use GPT to flood the internet with propaganda.

That would, of course, create harm.

OpenAI were naive with their plan to make the precursors to AGI open. They had picked a bad name for themselves, because they don't care merely about open sourcing AGI. They care foremost about harm reduction.

When we think of sacred things, it's hard to think logically. Many people on Hacker News hold two things as sacred: open source and unfettered freedom of speech.

Whenever these sacred ideas are infringed upon, many people here can't see nuance. They're blinded by rage.

I wish more of us could see that the OpenAI team are American programmers. They like freedom of speech and open source, just like us. They're not evil villains. Many of them read Hacker News.

To understand their decisions, we need to understand that their core value is harm reduction. When open source or freedom of speech is incompatible with harm reduction, they're going to choose harm reduction.

They're also not going to let themselves get destroyed by bad press after somebody makes a Nazi bot or whatever.

I love what they're doing, so it makes me sad to see all of this hate. I wish their projects were more open, because I really want to play with DALL-E, but I think ultimately OpenAI is moving us forward. They're an asset, and vitriol toward them is misplaced.

ta1234567890• 5 years ago

OpenAI != Open AI

Or maybe: OpenAI = Closed AI

twirlock• 5 years ago

I'm tired of this world covered with fucking idiots, and each one thinks they have the special golden morals that can inform us properly on when to interfere with others. Stop trying to be in charge of shit, reddit.

zach_garwood• 5 years ago

I feel like half of these comments were written by chatbots.

timvisee• 5 years ago

It isn't so "Open" after all. What a shame.

darepublic• 5 years ago

Maybe check out the open source equivalent of gpt3. True that open ai has a sort of dystopian woke corporate air about it.

citizenpaul• 5 years ago

>Microsoft Invests $1 Billion in OpenAI, a Startup Co-Founded by Elon Musk 23 Jul 2019

I mean you can argue fine points if you want but money is always the main factor. OpenAI has been dead for a while as far as "OPEN" goes.

OpenAI does however state in many places that they are very hesitant to approve any uses that could be "therapeutic" in nature. Understandable since MS doesn't want to lose money from medical malpractice lawsuits for their billion dollar investment.

OpenAI Shuts Down Chatbot Project by Indie Developer

Related Stories