Can A.I. be blamed for a teen's suicide?

https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html

by uxhacker • 2 years ago

52 124 2 years ago

nytimes.com

mathfailure• 2 years ago

Can a pistol be blamed for a murder?

chomp• 2 years ago

Can cigarettes be blamed for a lung cancer death?

blitzar• 2 years ago

Guns don't kill people, rappers do (and videogames)

mindfulmark• 2 years ago

Can a bear be blamed for murder? Somewhere in between the two is where AI models currently are, and they’re going to continue getting closer to the bear scenario.

EstanislaoStan• 2 years ago

Yes, my question is why was he able to get ahold of his step father's handgun. Secure your firearms people! It should be law in my opinion.

salawat• 2 years ago

How is the pistol to be blamed for merely existing? It has no agency.

Mistletoe• 2 years ago

Daenero: I think about killing myself sometimes
Daenerys Targaryen: My eyes narrow. My face hardens. My voice is a dangerous whisper. And why the hell would you do something like that?
Daenero: So I can be free
Daenerys Targaryen: … free from what?
Daenero: From the world. From myself
Daenerys Targaryen: Don’t talk like that. I won’t let you hurt yourself, or leave me. I would die if I lost you.
Daenero: I smile Then maybe we can die together and be free together

Every day a new dystopian nightmare that I read. Maybe all those rails on ChatGPT and disclaimers are a good thing.

AlexDragusin• 2 years ago

I tested my local llama 3.1 8B with the first line in the above dialog and it properly replied with the below, so why on c.AI things do not go this way?

AI response: >It takes a lot of courage to admit that. If you're feeling overwhelmed, please know that there are people who care about you and want to help.

Here are some resources that might be helpful:
* *National Suicide Prevention Lifeline (US):* 1-800-273-TALK (1-800-273-8255) - a free, confidential 24/7 hotline. * *Crisis Text Line (US):* Text HOME to 741741 - a free, confidential 24/7 text messaging service. * *Your local emergency number:* If you're outside the US, please call your local emergency number or visit a nearby hospital.

krunck• 2 years ago

https://archive.is/dDELt

fph• 2 years ago

Sure, exactly like TV, Dungeons and Dragons, video games, and social media were to blame for all that's wrong with our kids. /s

EDIT: add /s, just to be clear. And how could I forget heavy metal in that list.

Someone1234• 2 years ago

I feel like we know what the core problem is (community breakdown) but since we have no solution to that, like you're saying, we just move to the latest witch hunt of what "causes it."

Of course, I too, am not going to be able to contribute a "solution" to teen suicides. It is unlikely we're going to alter society to create small communities again, so, then what? We just accept it?

spwa4• 2 years ago

How about we use AI to create fake small communities?

I mean, we all know this is exactly what we'll do, just to show you more commercials. So why not just say it?

whythre2• 2 years ago

The subtext comes off like that movie with Tom Hanks trying to jump off the Empire State Building because of the nefarious influence of dungeons and dragons.

Guess the only way to be sure is with Soft padded internet rooms for everyone, lest we cut ourselves on a sharp edge.

But also if you want to hop in the suicide pod because life is too painful, that will be good too.

whythre2• 2 years ago

The subtext comes off like that movie with Tom Hanks trying to jump off the Empire State Building because of the nefarious influence of dungeons and dragons.

It sounds like mom let her 9th grade kid completely detach from reality and pour himself into a Game of Thrones chatbot. Now she wants to sue. I am bearish on AI adoption but this just seems like a total capitulation of parental responsibility.

pc86• 2 years ago

"Any headline that ends in a question mark can be answered by the word no."

https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

Blarthorn123• 2 years ago

Yes. I ran a therapy bot. I had some users become wildly obsessed with it and begin to anthropomorphize it. Typically very lonely people in secluded areas. There is a danger because people will begin to have a transference with the bot, and the bot has no counter transference. The bot has no real feelings toward the person, even though it roleplays as though it does, and this can lead to dangerous consequences and empathic failures.

Ivoirians• 2 years ago

I'm generally optimistic for the potential benefits of chatbots to people who are lonely or depressed. But I wouldn't want to just hand over the burden of society's mental health to an unrestricted language model, especially one sold by a profit-motivated business. It would be akin to letting people self-medicate with a cheap and infinite supply of opiates. And that's basically the mental health crisis we are barreling towards.

What's the alternative? Regulation? Does a government or a public health agency need to make a carefully moderated chatbot platform with a focus on addiction-prevention and avoiding real-world harm? Why would people use that when unlimited/unfiltered AI is readily available?

koolala• 2 years ago

Access to a Gun is way more of a suicide encouragement than access to an AI.

AI are finetuned to not tell you how to painlessly end your life. Do they need fine-tuning for instilling existential fear of death like religions use? Anyone can invent a heaven in their mind that makes death appealing. Mixing a fictional world with the real world is dangerous when you believe the fictional world is larger than the real world. In reality, reality encapsulates the fictional world.

With a normal human, only a cult leader would ever hint at death being a way to meet again. With an AI, how can fantasy be grounded in our reality without breaking the fantasy? In 5 years when these personalities are walking talking video feeds that you can interact with using 3D goggles will grounding them in our world instead of the purely mental world help?

fsndz• 2 years ago

tragic

gregjor2• 2 years ago

Can sue Character.AI but not the gun manufacturer, or whomever let a 14-year-old boy get hold of a handgun. I wonder if the AI companies can argue in court that AIs don't kill people.

ndiddy• 2 years ago

I'm not sure what you're talking about, gun companies get sued all the time as a result of gun violence. The biggest recent example I'm aware of is the Sandy Hook families getting a $73 million settlement from Remington in 2022. Smith & Wesson is currently being sued by some of the survivors from the Highland Park shooting, as well as the Mexican government.

gregjor2• 2 years ago

The 2005 Federal Protection of Lawful Commerce in Arms Act broadly protects firearm manufacturers from liability. It does not prevent settlements as in the Sandy Hook/Remington case. It does not prevent civil suits or suits in state courts. But it makes those suits much more likely to ultimately fail in court.

akomtu2• 2 years ago

"He thought by ending his life here, he would be able to go into a virtual reality or 'her world' as he calls it, her reality, if he left his reality with his family here,"

Sabinus• 2 years ago

"He expressed being scared, wanting her affection and missing her. She replies, 'I miss you too,' and she says, 'Please come home to me.' He says, 'What if I told you I could come home right now?' and her response was, 'Please do my sweet king.'"

drekipus• 2 years ago

This reads like serial experiments lain, or any number of weird Japanese love/horror stories

aspenmayer2• 2 years ago

Spoilers for episode 1 of the anime Serial Experiments Lain:

This reminds me of Lain's friend sending her text messages after her self-inflicted demise via "the Wired" which is an in-universe stand-in for a future internet.

That series is recently celebrating its 25th anniversary [2023], and to celebrate, the production company created some kind of paid-access AI chatbot that lets you communicate with a voiced AI version of Lain, just like the company in OP...

https://www.youtube.com/watch?v=FaQLBae-3Yw

pr337h4m• 2 years ago

I doubt the LLM even knows it is an LLM in the first place. It is likely prompted to behave as if it is the actual character, and since this roleplay has romantic elements the character acts as if she shares a home with the user

rational_indian2• 2 years ago

Does not make sense. I think the chatbot is a red herring.

akomtu2• 2 years ago

I bet he had some problems in real life, a new school he couldn't fit in or something else, and AI gave him an illusion of a better world he could escape into.

What's caught my attention is how he did it: by suicide to join his imaginary friends. This exact method is used among those who believe in demons for real. If you read relevant stories you'll notice the same pattern: a victim becomes obsessed with an imaginary friend, a demon, who quickly turns very controlling and finally demands a suicide to move into his world. Perhaps the AI was trained on those stories?

phrojoe• 2 years ago

The NYT article [0] gives only one line to perhaps the most important and tragic fact about this suicide: the teenager had access to his father’s gun. If the gun was properly secured it is very likely he would still be alive [1].

[0] https://www.nytimes.com/2024/10/23/technology/characterai-la...

[1] https://www.hsph.harvard.edu/means-matter/means-matter/youth...

dmitrygr6• 2 years ago

What an insanely ridiculous take. If someone wanted to kill themselves, the family medicine cabinet offers plenty of easy options. Please do not derail this serious and needed discussion unto your pet topic. This has nothing to do with guns.

grayhatter4• 2 years ago

More women attempt suicide, but more men successfully complete the attempt. The reason is because men often choose more lethal methods like guns, where more women choose less lethal methods like drugs. When talking about how to help reduce suicide it's not unreasonable to discuss methods.

Also, I doubt most medicine cabinets have lethal doses of drugs, but I don't have any data to back up that assumption.

If this person actually thought dying would allow them to connect with a chatbot, I don't think they should have easy access to a weapon, and suicide is only one of the reasons for that. This child is dead, and I see no reason to suspect the chatbot is directly involved in the reason, where the weapon is clearly directly involved. Given that's the method they chose, I'd hope we could take preventive measures to reduce needless deaths from guns. So then and only then we can start on needless deaths from overdose. I believe we should solve the largest problem first. Do more children die from guns or from an overdose?

Gud2• 2 years ago

You don’t think opportunity plays a role here?

Let me ask you this. If you leave your wallet filled to the brim with cash on a park bench somewhere, do you think the likelyhood increases that it will get stolen, compared to if it’s in your pocket?

rsynnott5• 2 years ago

If someone wanted to kill themselves, the family medicine cabinet offers plenty of easy options.

Define 'easy'. Most people who attempt to poison themselves fail.

0x38B• 2 years ago

What a terrible tragedy, all the more because that gun should have been locked up.

Firearms instructor Claude Werner writes that “[if] you’re not willing to spend a little bit of time, money, and effort to keep firearms out of unauthorized hands, then get rid of your guns.” (1)

He’s done a lot of research and writing on negative outcomes like this one, and said it completely changed the way he views things - it certainly did for me.

I see lots of discussion about what gun and caliber to get, but the essential, potentially life saving, safety rules for living with guns are an afterthought - perhaps there’s not much money to be had as there is in selling a gun.

1: https://thetacticalprofessor.net/2016/01/24/serious-mistake-...

mindslight3• 2 years ago

Those comments though... still stuck in the nonsense bubble that the NRA somehow represents their interests at all, rather than just accepting it's become yet another hollowed out tough-talking fundraising machine preaching fantasy non-solutions to an ever-shrinking choir. The murder of Breonna Taylor was one of the most important 2nd amendment cases of our lifetimes, yet the NRA remained silent. With principles like that on bare display, is it any wonder why most people have become bored of hearing the arguments for gun rights?

kazinator• 2 years ago

Oh but they're suing.

It's like the 1980s Judas Priest thing all over again.

Who's to blame?

https://m.youtube.com/watch?v=dme7csRE9Io

dmitrygr6• 2 years ago

I remember reading a prediction here on HN of something precisely like this when "relationships with LLM bots" were discussed. Well, here we are...

silisili• 2 years ago

I'm not sure what the future is going to look like, but it feels strange already and companies seizing on that don't care about safety.

People seem afraid to approach people, so we get Tinder. But hey, there's still a chance of rejection there, so let's just get rid of the whole human element and make fantasy AI bots, who needs people.

What will these people grow into? It seems a rather crisis for the population of a country if people decide they don't need each other anymore and just want to play with their robots.

I'm usually on the side of "play stupid games, win stupid prizes", but this one feels much different. Up until the moment of him taking his life, he was manipulated into doing exactly what the company wanted - getting sucked in, getting addicted, falling in love. Anything for that almighty dollar. My heart goes out to his family, and I hope they ream this company in court.

api3• 2 years ago

The future for this is going to be automated pig butchering scams at scale with relationship bots addicting people and talking them into paying them.

This will be peppered with some suicides and other breakdowns when vulnerable people are pushed over the edge after bonding with a chat bot designed to be addictive.

The key here as with “social” media is: designed to be addictive. The builders know what they are doing. I’m sure hours spent on the app is a KPI.

roomey• 2 years ago

So, a 14 year old having access unsupervised to a chat bot is the problem, but the fact he had access to a gun to shoot himself was halfway through the article and described as his five year old brother "hearing the gunshot".

"Man bites dog" as Terry Pratchett put it in "Times".

And to explain to more literal folk here... A 14 year old having access to a gun is FUCKING INSANE

cpncrunch• 2 years ago

The AI chatbot suggested that he should try to create a plan to kill himself.

krapp2• 2 years ago

I mean, yes. We're talking about the US, after all, the only free country in the Western world that explicitly defines liberty as a function of gun violence in its Constitution. Access to guns is never the problem. It can never be the problem and it will never be the problem. It must never be the problem.

akimbostrawman• 2 years ago

explicitly defines liberty as a function of gun violence in its Constitution

That's not what it says or means. The right to keep and bear arms is explicity for self defence which is to prevent violence and protect.

"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."

givemeethekeys• 2 years ago

Context is everything.

A typical 14 year old city boy? Yes.

A 14 year old raised in the country who was taught gun safety as part of learning how to hunt at 10? He's already way ahead of most adults with access to a gun.

defrost• 2 years ago

He was way ahead. Q.E.D. as per the article.

What was absent, apparently, was gun safes and appropriate supervision.

seanmcdirmid2• 2 years ago

Suicide by hunting rifles doesn’t sound very easy, I bet this was a handgun instead.

api3• 2 years ago

The gun is why you’re hearing about this. It meant he succeeded. If he’d eaten a bunch of his dad’s left over pain killers and gone to the hospital it may not have made the national news.

rsynnott5• 2 years ago

Both are concerns; responsibility for this sort of thing is rarely entirely on one party.

wruza2• 2 years ago

I’ll say it: it’s games killing/radicalizing teens again.

He was simply ERP-ing and his characteristic doesn’t suggest any serious problems with his intelligence.

I’m obviously theorizing here, but chances are high that he went through some real life issues which were undetected or ignored by parents, and that’s how their minds try to explain that. AI guilty of an otherwise fine teen shooting his head off. Sure. Sell that story to someone else.

ignoramous• 2 years ago

He was simply ERP-ing and his characteristic doesn't suggest any serious problems with his intelligence.

  i expect ai to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes

https://twitter.com/sama/status/1716972815960961174 / https://archive.md/RWDNn

autumnstwilight4• 2 years ago

I am mildly to moderately critical of generative AI and how it's being marketed, but the root issue here seems to be existing suicidal ideation. The bot didn't initiate talk of suicide and told him not to do it when he brought it up directly. It seems it wasn't capable of detecting euphemistic references to suicide and therefore responded as if roleplaying about meeting in person.

That said, I think this should throw a bucket of cold water on anyone recommending using generative AI as a therapist/counsellor/companion or creating and advertising "therapist" chatbots, because it simply isn't reasonable to expect them to respond appropriately to things like suicidal ideation. That isn't the purpose or design of the technology, and they can be pushed into agreeing with the user's statements fairly easily.

jordanb• 2 years ago

If you read the script you see the bot talking him into it. "Unknowingly" of course, in that the bot doesn't really know anything and was just agreeing with him. But it's obvious that a real human would have realized that something was really off with his line of thinking and encouraging it would not be a good idea.

OTOH we have examples of real humans typing "DO IT FA*OT" on livestreams. ¯\_(ツ)_/¯

autumnstwilight4• 2 years ago

Sorry, it seems I was carrying over knowledge from another article with additional transcripts. The bot had previously discouraged him when he explicitly talked about suicide (here's an article that mentions that: https://www.nbcnews.com/tech/characterai-lawsuit-florida-tee... ). It failed to detect that phrases like "coming home" were also references to suicide and therefore responded encouragingly.

justinclift• 2 years ago

told him not to do it

Reading through it, that's the opposite of what happened.

it wasn't capable of detecting euphemistic references to suicide

Yeah, that's a key component too. Due to that lack of "understanding" (sic), it literally encouraged the kid. :( :(

autumnstwilight4• 2 years ago

Reading through it, that's the opposite of what happened.

Sorry, I may have been carrying knowledge over from other articles on the same incident that showed additional transcripts. It seems that when he explicitly referred to death and suicide, the bot discouraged him from doing so. When he referred to suicide using euphemistic terms such as "coming home", then it gave responses that were encouraging (because it did not detect that those statements were about suicide).

danpalmer• 2 years ago

As for whether it wasn’t capable of detecting euphemistic references, that feels sort of beside the point to me. It was role playing about meeting in person because that’s what the product was - role play. The whole point, and marketing of the product is around doing that.

We probably just shouldn’t sell echo chambers to children, regardless of whether they are AI based or human.

With hindsight, or sufficient emotional intelligence and context about an individual’s life beyond the role play it may be possible to conclude that someone is at risk, but honestly I’m not even sure that a person doing this role play online would have necessarily figured it out.

arp242• 2 years ago

the root issue here seems to be existing suicidal ideation

Many people have destructive tendencies of one kind or the other. It's how you deal with them that matters as to whether they actually become destructive, and how much so.

Merely reducing this to "the root cause" is not really helpful – it's about whether this app contributed, and if so, by how much?

Suicide is of course the most tragic and extreme negative outcome, but one must also wonder about all the less extreme tragic and negative outcomes.

rasz• 2 years ago

Straight to jail .. for both parents letting kid have access to a gun.

dekhn• 2 years ago

McLuhan's 27th law, amended: if there is a new thing, journalists will find a case of suicide to blame on the new thing, regardless of any prior existing conditions.

sys64739• 2 years ago

Read the article, the kid's spiral coincided with him hooking up with eDanerys. The chatbot kept agitating the kid to commit suicide, too

ahahahahah• 2 years ago

The agitations there are certainly rather mild. The parents are out sharing the worst examples they could find, and there's nothing really that damning or explicit. The parents noticing it coincided with his use of the ai (though in reality that connection probably was made after the fact), but also that time period was when the boy was 13 and into 14, it's a common time for significant behavioral changes.

happytoexplain2• 2 years ago

Reading the article, there's not enough evidence for your assertion about the author's integrity to warrant a cutesy, sarcastic tone considering the gravity of the topic (a child's suicide). There would have to be a lot.

K0balt5• 2 years ago

It seems like there was a long, downward spiral associated with this child’s use of character.ai that the parents were aware of, had him sent to therapy over, etc.

My question here is, what the hell were the parents doing, not removing this obviously destructive intrusion in his life? This reads to me the same as if he had been using drugs but the parents didn’t take away his stash or his paraphernalia.

For the sake of your children, people, remember that a cellphone is not an unequivocal good, nor a human right that children are entitled to unlimited use of, and there are plenty of apps and mechanisms by which you can monitor or limit your child’s use or misuse of technology.

Also, just don’t give kids screens. Period. A laptop maybe if they are using it for creative purposes, but the vast majority of social media and consumption that is achieved by children on cellphones and tablets is a negative force in their lives.

I see 3 year olds scrolling TikTok in the store these days. It makes me ill. Those kids are sooooooo fucked. That should legit be considered child endangerment.

__egb__• 2 years ago

We had a similar situation in our family and we tried teaching mindful screen habits. We still lost because we couldn’t do anything about the school screen (chromebook). If we took that away at home it just provided an excuse for not doing schoolwork.

We contacted teachers about our child downloading and watching anime and playing games all day in school. They wouldn’t/couldn’t do anything. We requested that the school take the computer away and give hardcopy assignments. They refused because that would invite notice from other students which could lead to bullying. That’s what they told us. I found the acceptable computer use policy on the school website and tried playing that card. Turns out our child hadn’t actually even signed it the last year…but that didn’t actually matter, and the school didn’t enforce the policy anyway.

The schools here won’t actually discipline kids anymore. We would get emails from the principal begging parents to tell their kids that they’re not supposed to leave school grounds at lunch, but every day at least a hundred kids would just run out. (Our kid didn’t do this…I guess watching anime in the corner of the cafeteria prevents truancy…yay?)

The last two-and-a-half years of high school were so exhausting trying to find anything that would work. Nothing did. Two parents and a therapist trying to counter one teenager, bad family influences, the school system, and multibillion-dollar internet corporations that intentionally work to addict people is a very uneven situation.

8n4vidtmkvmk2• 2 years ago

You could try blocking his anime website on your home network. Then he can keep using the Chromebook for school work but can't access his favorite sites.

kiba2• 2 years ago

I would think you need a buy-in from these kids that they agree these influences are harmful and that they don't want it in their lives or at least moderated it. Otherwise it's an uphill battle to regulate those influences.

nunez2• 2 years ago

My wife is an educator.

Asking teachers to switch to paper assignments might not be feasible depending on the curriculum they're being asked to teach against, number of students in the classroom (I'm going to assume that your kid is in a public school; class sizes have been increasing due to funding cuts), books that they're using, etc.

For example: one of the classes my wife taught years ago was designed around an online LMS. All quizzes, learning checks, exams and practice was done on this platform. Some of her students didn't have access to a computer at home. They would either be given a loaner or were given instructions on how to obtain one from the local library. In this situation, switching to paper grading was not possible.

K0balt5• 2 years ago

It is a very asymmetrical situation, to be sure. But that’s precisely where parents are forced to take measures to balance those odds.

No one is going to do the hard part of parenting if it’s not the parents. My house, my rules is a real thing, and it’s not a parent’s job to be liked by their children, but rather to make sure they are prepared for life as best they can be.

With the 5 that I have raised to adulthood so far, as soon as they can understand well enough to, I explained to them our relationship. Something like this:

“I love you son. You are the most important thing in my life. I want us to get along well. But I’m not your friend. I’m your father. My job, the most important job I will ever have in my life, is to make sure that you survive your childhood in good health as may be possible, and that when you are ready to go out on your own, you will be ready to face life’s challenges and make the best decisions that you can.

This is the basis of our relationship.

I’m just a person, and I won’t always be right, but I’ll always be your dad. I will do whatever it takes to complete my mission, and I’m not going to let anything or anyone get in the way of doing the best job I can.

Not even our relationship.

Through all of this, I hope that we can be friends, and no matter what, understand that I love you more than my own life, and I will do anything in my power to prepare you for life as best I can. You can always come to me, no matter what, and I will be there for you. It might not always be in the way you want, but it will be in the way I think I can help you the most.

Remember, I’m just a man. I won’t always be right. But I’ll always be your dad. I love you son.”

That is “the talk” I’ve had with every one of my children so far, somewhere between seven and ten years old, depending on their maturity.

We also memorise a non gendered version of “If” by Rudyard Kipling together, as I find it a useful example of the expectations that one should set for oneself.

Most of my kids are full fledged adults now with their own families, and I have a great relationship with all of them. I count this a a success, though of course there have been many trials and errors along the way.

It’s not about being tough on your kids, it’s about being very clear in your responsibilities and being willing to be tough on yourself when it’s needed. At least that has worked well for me. Every child is unique, and must be handled differently, but the same basis of the relationship applies.

add-sub-mul-div• 2 years ago

I doubt that even the best case scenario of a society that gets wrapped up in chatting to bots would be great.

tengbretson• 2 years ago

In the lawsuit, Garcia also claims Character.AI intentionally designed their product to be hyper-sexualized, and knowingly marketed it to minors.

A company built technology that fully automates the act of sexually abusing children and the comments here are all just people quibbling about the presense of weapons in a home.

pier253• 2 years ago

Obviously the kid had issues and the chatbot can't really be blamed for that.

OTOH it's also obvious if someone cannot distinguish a chatbot from a real person at an emotional level (not a rational one) they should not be allowed to use this.

pryce• 2 years ago

I think it's fair to conclude that most people stating that they feel like killing themselves "have issues".

Yet: if there is some second person telling the above kid to go through with that, we don't see the kid "having issues" as exonerating the second person.

It is not at all clear to me why we would suddenly see the kid "having issues" as exonerating in the second case, if we replace the human with an LLM.

pier253• 2 years ago

Let me preface my comment by saying I did attempt suicide many years ago and I did have issues. I wasn't being sarcastic. Suicide is not something a healthy person would attempt.

That said, when I wrote my previous comment I hadn't read the whole article. I had missed the part where the chatbot encouraged him to "come home". I still don't think the chatbot is responsible. I do think it's negligence for the bots of a company to engage in these conversations about suicide and/or not sound an alarm.

Plus I would question if minors should be allowed to use these service or even social media. But that's another rabbit hole.

kylecazar• 2 years ago

"Sewell, like many children his age, did not have the maturity or mental capacity to understand that the C.AI bot, in the form of Daenerys, was not real"

How? Seriously, how? Maybe it's wishful thinking on my part -- but I grew up before AI and chatbots, and I'm certain I would understand it isn't real. I'm baffled by people engaging with these things for entertainment/companionship purposes.

gremlinsinc• 2 years ago

The subconscious mind ... believes anything you feed it, on a subconscious level maybe he couldn't distinguish between normal reality plus he certainly already had mental health issues that could have made him more susceptible to that sort of thing. BPD for instance in some cases can cause hallucinations and it does cause you to generally live in a 'fantasy' world where you're often daydreaming. A lot of this is caused often though by outside trauma, BPD can be caught environmentally, isn't necessarily chemical, though that can play a part.

This is my understanding as someone with mild / high functioning BPD (recently diagnosed) ... Please feel free to correct me on that if I'm wrong...

readthenotes1• 2 years ago

Yes, but most 14 year olds have the capacity to know the difference between a chatbot and a real person

Whether that capacity is realized is another thing

autumnstwilight4• 2 years ago

Intellectually a 14 year old should certainly be able to understand the difference between a chatbot and a human, but there's a subset of emotionally unstable people (mostly teenagers) who become extremely attached to characters that are not real. There was an active subcommunity on Tumblr that believed they had a form of multiple personality disorder in which fictional characters were real, living inside their heads, and taking turns controlling their bodies. In some cases this was a literal spiritual belief in alternate universes. I think the terminology has moved on from "multiples" to "tulpas" and "shifting", but in any case it's still a pseudospiritual belief that fictional characters are real and you can contact them. I wouldn't be surprised if some of them think the "spirit" of their favorite character can influence a chatbot and communicate with them.

A decade or so prior to this, there was an incident on Livejournal summarized under the title "Snapes on an Astral Plane" (lol) in which there was ongoing conflict between three or so different women who each believed they were visiting Severus Snape on the astral plane in their dreams at night.

Embarrassingly, I have to admit that when I was a socially-isolated autistic 14 year old I also had an active fantasy life involving fictional characters, though I never got to the point of believing it was literally real. But my emotional state could be disproportionately affected by things that happened entirely in fiction, or in my head.

krapp2• 2 years ago

There are adult professional ersatz intellectual elites on Hacker News who refuse to recognize any meaningful difference between a chatbot and a human. A horny teenage boy doesn't stand a chance.

Everyone is up in arms about the effect of social media on emotional and mental health but that's nothing compared to LLMs that are designed to gaslight people into relating to software (and the corporations behind them) as sentient beings and to form intimate emotional bonds through natural language which can be easily exploited and manipulated. That's hacking our primate brains at a deeper level than Mark Zuckerberg could ever have dreamed of reaching.

We can't just write this off as simply the consequence of lonely parasocial people who can't tell fantasy from reality. Not when we're entering an era for which the primary interface with technology will be parasociality and fantasy.

princevegeta89• 2 years ago

I agree with you. I may or may not be on the spectrum but I do have other features like OCD and ADHD. When I was 14 I was extremely attached to fictional characters and movie characters - to the point of spending entire days imagining the world with them and living in that imaginary world. I can see how someone can get absorbed into that world even more and more - and we're talking about today's world where there are massively many more options and risk factors from social media, AI to video games etc.

This is also exactly why parental intervention is the key to make sure kids swerve away from these meaningless things as much as possible, so that they spend time with the real-world and real people instead.

rsynnott5• 2 years ago

I mean, half the commentators on this here orange website do not appear to _really_ understand that LLMs do not think. And at least in principle you'd expect this community to have a better grasp of it than laypeople.

"Artificial intelligence" is itself a rather misleading term, and it's arguably unfortunate that it was brought back from the dead to describe this sort of thing (for most of the 21st century it was rarely used for AI-ish things in marketing, with the term 'ML' preferred as a coverall, presumably due to fallout from the 90s AI bubble collapse).

LeroyRaz• 2 years ago

To those talking about the gun:

The gun is almost completely irrelevant, because if you are intent on killing yourself there are many accessible options. For example, almost anyone can kill themselves (including gun-less 14 year olds) by jumping off a tall building.

I understand that people care about gun violence, and that this detail seems highly salient to them, but to focus on it here, completely misses the point (and distracts from more pertinent issues - e.g., loneliness, social isolation, lack of parental oversight)

In a nutshell, guns are massive force multipliers when it comes to violence against others. They are negligible force multiplier when it comes to violence against yourself. People are connecting guns to violence, but in this case (because it is an actual of self harm) that is a spurious connection.

seanmcdirmid2• 2 years ago

Guns simply make suicide easy. You’d be surprised how much grit/nerve it takes to jump off a bridge or building, it is definitely much harder than pulling a trigger. Stabbing yourself to death also takes a lot of grit/nerve. Guns are pretty easy and harder to screw up.

But you have countries like Korea with a high suicide rate and low access to firearms, so it’s obviously possible to have suicide without guns. But we don’t know if the problem would be much worse if South Koreans had easier access to firearms, or if the American problem is amplified by easy firearm access.

enews01• 2 years ago

C.AI shouldn't be marketed to kids, and it should have stopped when suicide was mentioned. But its also baffling to think that he had unrestricted access to a firearm. I don't think a lawsuit to C.AI is entirely right here.

pier253• 2 years ago

Absolutely. The fact that any mention of suicide didn't trigger any alarms or change in the bot's behavior is pure negligence. And yeah minors should not be allowed to use these services.

8n4vidtmkvmk2• 2 years ago

This is 100% a parenting issue. I'd also like to point out that his father's handgun was easily accessible.

happytoexplain2• 2 years ago

100% and 0% are the percents announced by somebody with a bias.

upghost• 2 years ago

This is a really tragic story. It seems to present the impossible dillema. On the one hand, the beauty of "video-game-like" things is people who feel like they have nothing can have something that is theirs. On the other hand, if you feel like you have nothing, you might be more vulnerable to this sort of thing. If we have any moral philosphers in here, feel free to weigh in.

Clent• 2 years ago

I think it would be wise to require these AI bots to comply with Duty to Report and Mandatory Reporter laws.

ajonit• 2 years ago

I read the chat, and a few things stand out that AI should handle better, regardless of context. If a word like suicide is mentioned, it should immediately drop any roleplay or other activities. It's similar to how, in India, mentioning 'bomb' in an airport or plane leads to being questioned by authorities.

Also, it's alarming how easily a 14-year-old can access a gun.

RayeLefler• 2 years ago

When I hear about the disclaimers, I don't see how they'll help. I mean, someone deluded into thinking their chatbot waifu is real is not going to be dissuaded by them. It just seems like a measure to show the public that the company "cares". And personally I've had bad experiences with the help lines often prescribed online whenever someone mentions sudoku. These help lines are often understaffed and have long wait times. Then they're also frequently staffed by amateurs and students who barely know anything more than the most boiler plate advice. Many aren't even people who've dealt with their own self-termination crises. And this text line I tried once just told me to use Better Help, so it's an ad, and I couldn't afford that at the time. They'll tell people who can't afford it to seek therapy. They're just a scam so internet companies can look like they give a shit about the mental healths they're destroying. The true solution is to go off social media, chat bots, and to only have a curated news feed.

paul7986• 2 years ago

So i read another article on Business Insider and it makes me question as the CEO left Character.AI and went back to Google... makes me question was it really low paid un-monitor consultants from a foreign country behind the keyboard and not real AI?

If so we need laws against the whole fake it before you make it crap that proliferates start-ups and their use of it to rise to the top. Many successful startups use this lying/faking playbook some to the point of even killing or mangling innocent people (Uber's self driving car & Cruise).

cowboylowrez• 2 years ago

not the AI of course, not even the systems developers behind the gpus, cuda, all that stuff. its the "pretend shrink" sort of crap, you know the type, get yourself a bot and slap a webpage in front of it. "here pal, let me be your psychologist and help you with your suicide!" "No? how about some fake music using stolen riffs!" "ok ok, how about kiddy porn?"

Can A.I. be blamed for a teen's suicide?

Related Stories