Deepfakes Tore a High School Apart

https://www.404media.co/radnor-high-school-pennsylvania-ai-deepfakes-child-sexual-abuse-material/

by Brajeshwar • 2 months ago

67 130 2 months ago

ceheaaf2• 2 months ago

Didn't get past the paywall but, recently a friend had their primary school targeted by blackmailers who took public photos of children and made (I'm told) extremely explicit images with them, threatening to release them if not paid.

What's the defense? Intelligent screening of incoming messages so that the threat never reaches the blackmail target? I imagine they'll find an unprotected channel.

Don't post innocuous images of children ever? Seems like losing.

eska• 2 months ago

Don't post innocuous images of children ever? Seems like losing.

I do have to say that I find it disturbing how liberally parents seem to post pictures of their children in public though. Respect the privacy of your children, they’re not your pets.

ceheaaf2• 2 months ago

I'm more thinking, the winning school sports team, a choir performing, etc. Things that would be posted in normal institutional participation and give children positive affirmation and recognition

CalRobert8• 2 months ago

It's nice to see pictures of people. I like when parents share photos of my kids at events where they're having a great time but I couldn't make it. It's nice when the local paper shows a picture of happy graduating 8th graders.

Saying "you shouldn't do this thing that was basically harmless from the birth of photography until the ~2010's" is ignoring a lot of history and context.

jvidalv• 2 months ago

Not caring.

pjc502• 2 months ago

What's the defense?

Prosecution of AI operators for making indecent images of children?

ben_w2• 2 months ago

Although I agree this is necessary, I don't think it is sufficient.

Hackers and blackmailers on the internet can be anywhere from trivial to impossible to identify.

Even harder to prosecute when they're in a different jurisdiction, as not everywhere has extradition treaties to everywhere else. I wouldn't be overly surprised if e.g. the North Korean government runs some schemes like this (though not necessarily actually this) to bring in money.

But yes, definitely do the easy things first, like trying to stop e.g. grok from doing it and also going after users of e.g. grok who try to get around such efforts. A lot more crime happens than can be prosecuted, so raising the minimum competence threshold to commit crimes in the first place is very necessary.

Cthulhu_3• 2 months ago

Sure, but the reality is that if it happens abroad there is no legal recourse. Laws are not global, but the internet is. The only real option there is to close the internet.

Or to elect me as god-emperor and unify the world under my rule.

mittensc• 2 months ago

What's the defense? Intelligent screening of incoming messages so that the threat never reaches the blackmail target? I imagine they'll find an unprotected channel.

Same defenses that are used against fraud and other crime.

criminal prosecution of the blackmailers AND the services used to generate the pictures.

This is effectively child porn... so penalties would be pretty harsh.

There are extradition treaties to most of the world, so unless the blackmailers are in China/Russia they will end up in jail.

That same thing played out with piracy with people extradited to the US from various countries

graemep• 2 months ago

I just looked it up and according to Wikipedia distributing child porn is a crime in both, possession is a crime in China. Blackmail is a crime in both too. So even if they do not extradite the blackmailers they are likely to face jail under the their own laws.

Asooka• 2 months ago

That is a clear-cut case of extortion. The defence is having the police do their job and apprehend the criminals in question. If you screen incoming messages and stop them from reaching their target, extortionists will switch to publicly releasing one risque image with threats of releasing more explicit ones. There is always an unprotected channel as you noted.

As an aside, please do not use the b-mail word. It is insensitive towards BIPOCs.

account422• 2 months ago

As an aside, please do not use the b-mail word. It is insensitive towards BIPOCs.

No it isn't. If anything needs to stop its the ever escalating orwellian censorship of words that you're proposing.

bandrami2• 2 months ago

In the US possession of CSAM is a strict-liability crime. If the models contain the data to produce it, the modelmakers should be liable.

pbhjpbhj2• 2 months ago

Models don't have to contain imagery of a thing to produce imagery of that thing - the canonical example is the pelican riding a bicycle.

Model makers, arguably, are like pencil manufacturers but in a World of good artists.

I think you can hold people to account for what they help to create, but not what they have potential to do.

Now, if models were trained on any csam, of course, the model's owners should be held you account.

hjkl02• 2 months ago

Don't post innocuous images of children ever?

Don’t raise boys who abuse girls. That’d be a big step in the right direction.

account422• 2 months ago

Because girls can never be abusive?

arkh• 2 months ago

What's the defense?

Prosecuting every one of those blackmailers. If this kind of crime starts going into the "you're gonna get caught and jailed 100%" category, less people will try it.

d1sxeyes• 2 months ago

That's not a defence, that's a remedy.

Cthulhu_3• 2 months ago

What if the blackmailer is anonymous and abroad?

nkrisc• 2 months ago

Why a school, in this day and age, is posting pictures of children is beyond me. At my kid’s school we refused to sign the photo release clause, and any time they might be taking pictures they had to email us to ask if it was OK if our child was included. We agreed to it for the yearbook, but refused everything else.

antran22• 2 months ago

That is propagation of CSAM. We already have way of dealing with people transmitting CSAM, that is to prosecute them heavily.

You won't prevent people stabbing others with knife by banning knife or asking people to wear knife-proof vest going outside. You deter them by making everybody know that the consequence of harming somebody else is going to be a very unpleasant experience.

ale42• 2 months ago

https://archive.is/neN8H

thunderbong• 2 months ago

https://archive.is/neN8H

hjkl02• 2 months ago

This has nothing to do with “deepfakes”. These girls could have been attacked by aliens and be received with more credulity. There has to be a point where you understand that this behavior towards girls and women is simply being allowed.

One of the mothers says this quite explicitly:

Calling the event rumors and speculation when a crime occurred was almost worse than the crime

ktallett2• 2 months ago

This is sadly becoming very common and they are worringly good enough for people to assume they are real. I had a friend who had this issue, she has tried to ride it off as no clue who made them or shared them but I know it can't be easy for her. There needs to be appropriate laws for this the same way there is for revenge porn etc.

jacknews• 2 months ago

Surely this comes under defamation law or something similar, as if they photoshopped the girl's heads onto porn photos.

Anthony-G• 2 months ago

Agreed. Even if it’s not possible to know who created the images in the first place, anyone who “shares” them should be culpable.

CalRobert8• 2 months ago

I don't know how society will function when it's as easy to fake a video or image as it is to draw a stick figure. Hopefully it at least means that people view what happened here as akin to scribbling boobs on someone's photo, but we're not there yet apparently.

Sharlin3• 2 months ago

Hopefully we’ll get back to the time back when people didn’t have to worry about someone filming them all the time.

Cameras cryptographically signing all photos and videos should become the standard. Media that cannot be traced back to real camera hardware should a priori be considered fake or tampered with.

Still, deepfakes are partly a social problem and, as should be well known, social problems can rarely be solved by technological solutions.

_el1s73• 2 months ago

That won't work, a camera can also photograph another screen or a fake printed photo. And also cryptographic modules can be cracked.

CalRobert8• 2 months ago

I can't imagine that governments will end their zeal for surveillance.

sirsinsalot5• 2 months ago

This is another example of treating normal behaviour as criminal. Like banning VPNs or age verification.

How about we undo the radical individualism that has rotted out any compassion and empathy from society in the pursuit of greed? Not as easy to do granted, but it's a social technical debt that can't be quick fixed.

prodigycorp• 2 months ago

Perhaps there is reason to be optimistic. If we believe nothing online, then digital inauthenticity may cause us to yearn more for authentic, in-person interactions.

The modern web has not been without its warts. It's led society to some very unhealthy behaviors. It's telling that a lot of people think the internet is society, rather than a slice of it. Perhaps its decline is the correction society needed.

It's easy to feel like a doomer if you're thinking "it's sad that the web is being torn apart" and your framing of the web is "the magical place of promise it used to be 25 years ago."

CalRobert8• 2 months ago

Suddenly a polaroid feels better than any digital photo.

latexr5• 2 months ago

A major problem with a culture of distrust is that bad actors doing bad things get impunity by telling everyone that what they did was fake.

It’s already been tried, when deepfakes weren’t even that good.

https://www.theguardian.com/technology/2023/apr/27/elon-musk...

nunodonato3• 2 months ago

Funny, I've been thinking in the same lines as of lately. The speed-up of the lost of trust (and interest) in everything online will make us care more for the real world around us. Perhaps we will see a big shift in values.

But then again, perhaps we won't and things will just get worse.

noufalibrahim2• 2 months ago

I think there are generational aspects to this.

The older generation (gen-x) and before has been trained to believe traditional media (and by extension, anything that is published on a "website" and/or forwarded on Whatsapp). Images, pictures etc. add to the authenticity. Of course, it's easier to forward than to investigate and refute.

The younger generation is less influenced by this but also care less about generating and forwarding these. Most of these things are "just a joke". In a weird way, pushing out non-consensual nude of someone is the same as editing someone's face in a group photo with dog ears and sending it to a circle of friends.

I don't know what the solution is but this kind of thing erodes trust. Photographs used to be evidence that can be used to establish trust and we've been culturally conditioned to accept them that way. This takes a sledgehammer to that and it's not easy to untrain a whole society away from a deeply conditioned feeling so easily.

noosphr5• 2 months ago

https://en.wikipedia.org/wiki/The_Right_to_Read

You will have no compute and you will like it. The top comment here is about arresting the children for distributing child porn.

We really should have kept the normies out.

b65e8bee43c2ed03• 2 months ago

it functioned just fine for the past 50 years. digital manipulation of media had always been possible. americans simply didn't have a society-wide psychosis during the early days of personal computing and Internet.

sirsinsalot5• 2 months ago

It took some skill and time before to make anything passable as realistic. Enough skill and time to defeat a teenagers motivation. That's the difference.

"make the person in this photo naked. No mistakes"

Prompts lowered the bar. Even if you do have to install a local text to image model.

fny• 2 months ago

We need a way to prove what's real.

_el1s73• 2 months ago

Just put your phone down.

m463• 2 months ago

draw a stick figure.

I hope we don't start getting xkcd fakes.

M4v3R• 2 months ago

This kind of behavior should be met with swift and strong reaction from both the school authorities and the law enforcement. Kids responsible should be put on trial and face serious consequences including jail time for most egregious cases. Let the news cover it nationwide. "This is what will happen if you even think about creating and circulating nude pics of anyone" should be the message sent. This is the only way to prevent this behavior - not more censorship, not technical solutions

noosphr5• 2 months ago

Back in my father's day you'd get executed for drawing a dick on Stalin's forehead. It's nice to see people still care more about symbolic violence than actual violence.

numbsafari2• 2 months ago

These are teenage girls, not adult men. This isn’t graffiti, but highly realistic sexual depictions.

There’s a massive difference.

Curosinono3• 2 months ago

I do agree that shit behavour should be met with direct reaction but lets be honest here the time and effort of doing this is 5 minutes.

A proper society should educate people about things they can't get rid of anymore.

And no you can't just put teenager, which have not fully formed brains yet, into prison for fake nudity

sirsinsalot5• 2 months ago

I agree here. People are too quick to levy adult punishment and morality against children.

Teach them compassion and empathy. Teach them why their actions hurt.

latexr5• 2 months ago

the time and effort of doing this is 5 minutes.

Yet the consequences for the victims may last a lifetime and be both intrinsic and extrinsic. Sexually forcing themselves on a drunk colleague at a party or taking their dad’s gun and shooting up their classmates also only takes a few minutes, but can scar a community for years.

I’m not going to opine on what the punishment should be, though it seems clear there should be one and more than a slap on the wrist. The community where this took place (i.e. those who are living through it) should probably have a say in it. There should also probably be an investigation of how this came to pass, i.e. has this kid always been an asshole with a pattern of behaviour or have they recently been indoctrinated by manosphere bullshit and should get counselling to understand their actions and that women are people too and deserving of respect?

pibaker• 2 months ago

There are a lot more things we punish people for that take less than 5 minutes.

samlinnfer3• 2 months ago

It used to be that if you wanted to see people naked you’d just use your imagination. If you really wanted you’d draw or use photoshop, but both actually required skill. AI made it too easy. Teenagers wanting to see each other naked is normal and part of the human experience.

I don’t agree with the nanny state approach. America is already too obsessed with violence being ok and any kind of sex/nudity being completely evil. I don’t have an actual answer but having a moral panic over it is not the solution.

latexr5• 2 months ago

any kind of sex/nudity

Worth considering that this isn’t “any kind”, it’s a very specific kind. It’s nudity of minors they personally know, shared in their community. It’s on a different level than, say, consensual nudity from a stranger in a Playboy magazine.

ndsipa_pomu3• 2 months ago

I wonder if this kind of problem happens a lot less in societies that treat public nudity as normal (e.g. nude saunas, topless beaches in European/Scandinavian countries).

ChoGGi4• 2 months ago

To me the nudity isn't an issue, it's the bullying. Then again I'm not American.

GeoAtreides3• 2 months ago

let me parse what you're saying: you're saying there are no issues with male teenagers generating deepfakes of their female friends. You're saying that's is " normal and part of the human experience."

Am I getting this wrong?

barrkel• 2 months ago

Immature kids - and kids are by definition immature - make dumb decisions.

Certainly there needs to be repercussions, the kids need to learn, and need to be warned in advance, but kids will do dumb things.

You can talk about news coverage, but you can't force 12 year olds to watch the news, or understand how their actions have consequences. When the consequences do come, the focus needs to be primary on rehabilitation and restitution for the first instance.

oliwarner• 2 months ago

That's what we need. More children in prison. Now two lives are ruined. Just for the sake of retribution.

Redeemable children can make horrific mistakes. The job of parents, and pastoral adults is to maintain an environment where kids can exist without making irreparable, life-destroying mistakes. Let them learn, but limit the damage they can accidentally cajole themselves into doing.

That means limiting access to social media, phones and cameras. Turning the school clock back to 1995.

28304283409234• 2 months ago

Parents should be put on trial for giving their kids smartphones.

Henchman213• 2 months ago

Wow. I’m thinking the most charitable reading of this is “You forgot the /s”

Otherwise this is insanity.

reaperducer• 2 months ago

Otherwise this is insanity.

It's a disruption of an entire school and the learning process. Just like a bomb scare, or a fake fire alarm. Just because it's AI doesn't mean it gets a pass.

One of the points of punishment is to deter copycats.

autoexec2• 2 months ago

"This is what will happen if you even think about creating and circulating nude pics of anyone" should be the message sent. This is the only way to prevent this behavior - not more censorship

Saying "don't make a fake picture of anybody naked or you'll go to jail and your life will be ruined forever" sure seems like a form of censorship

conductr2• 2 months ago

I don’t think you understand the psychology of a teenage boys. If you do all this, all you accomplish is informing the entire nation of kids that making deepfake nudes of their classmates is possible. From there, it will proliferate much more rapidly than its current trajectory.

There’s really no way to close this Pandora’s box. So, I’d fight fire with fire instead of crying about it if I was a teen girl in this situation. Make deepfakes of the boys have micropenises and other potentially embarrassing things. And then just learn to laugh about it. If everyone is a target and victim the kids will find it boring fast enough and this will fizzle out. It’s not like these kids aren’t exposed to porn and all that early on. They all are, have been for years now, the victim here feel awful because they’re the only ones being targeted.

I know this sounds bat shit crazy but normalization of it is really all we can do now.

dmos622• 2 months ago

Throughout the night, whenever one of the speakers brought up holding Big Tech accountable, and specifically Apple and Google’s app stores, the crowd applauded.

The framing bothers me. It's shifting responsibility away from kids and the community. Deepfakes didn't do anything: kids made and used deepfakes in destructive ways.

Big tech sucks, you get no argument from me there. But, how do people so quickly get over the fact that a kid is sexually harrassing someone?

This is like when a mass shooting happens and everyone only talks about guns. Well, yeah, if you didn't have guns a mass shooting couldn't happen, but isn't the underlying issue that someone decided that hurting a bunch of people is the best thing to do?

The more people dislike a crime, the less likely they are to talk about the perpetrator's inner life, but, come on, these are kids. How do people get so side-tracked?

teaearlgraycold3• 2 months ago

That's crazy talk. You actually want the village to raise the children and have everyone well integrated and healthy?

dmos622• 2 months ago

You know, that is crazy. Disregard what I said. Down with big tech!

gorgoiler• 2 months ago

Even though they fluffed the messaging I think the school was right, in the absence of any actual deepfakes and even after the police department had scraped all the boys phones, to be wary of accusing anyone in particular of doing anything AI related. Kids are weird and one boy saying he “dropped $250” on making deepfakes of “that hoe” sounds as much like a brag about money than it could be about making fake videos. There’s clearly enough of a text message trail between these boys to give them a significant reprimand for misogyny.

The PD’s PR team changing their narrative seems to lack competence. The school board focusing on being light touch on a specific case instead of cracking down in general seems like another error of judgment. This whole article says more about the effectiveness and competency of cops and school boards as it does about generative imagery.

conductr2• 2 months ago

There’s clearly enough of a text message trail between these boys to give them a significant reprimand for misogyny

I skimmed but the worst I saw was one of them referencing her as “that hoe”. Not nice but also punishing kid’s private conversations outside of school this closely seems like an overreach. Worse, is the message it teaches the younger generation that speech should be censored and isn’t as free as they think. I feel that’s a worse outcome on a longer timeline.

teaearlgraycold3• 2 months ago

I think sharing these images is sexual harassment and it's pretty clearly wrong and deserves punishment. Other than that, I don't think it should ever be possible to commit a crime by providing inputs to an air-gapped "clean" computer and then view the output. But it seems like that's not the case.

bondarchuk3• 2 months ago

The article intentionally obfuscates the difference between images depicting sexual abuse that did not happen, and actual sexual abuse. Ironically this makes it harder to address the problem on the level where the actual abuse happens i.e. bullying, lack of empathy, teenage body image issues in the properly fucked up american high school environment.

For example here the word "abuse" technically refers to the making and spreading of these images and attendant bullying, but obviously has connotations of actual sexual abuse:

Despite all this, Radnor’s administration failed students in the days and weeks after it learned about the abuse,

This is maybe subtle but later the article gets more explicit about it:

this technology [...] bears little difference from the non-consensual intimate imagery that’s plagued young girls and teenagers since the invention of the camera.

(i.e. a straightforward equation of images of nudity that did not happen, and images produced by actually pointing a camera at someone who is physically present with their nude body)

Because the images aren’t “real,” authorities grapple with how to handle them.

Putting "real" in scare-quotes, very nice.

Dorfman is also quoted doing some nice sleight of hand, going in one sentence from "It was about the creation of the videos" to "and then share them with others as if the girls were something to be passed around." - i.e. equating the making of videos depicting unreal events with something that almost sounds like gang rape.

Anyway enough examples, article is full of them. I'm not saying there's no problem here, just that these people seem to have some kind of ideology of the sanctity of the image, i.e. a realistic image always depicts real events, and should always be interpreted as if it does. If it does not, clearly that's a problem to be corrected by reinforcing the connection between images and reality by making it impossible to produce realistic images of non-real things. It strikes me as exactly the wrong way to go about it, that can only make the bullying and harrassment (the actual problem which should be solved on the social level) worse by making its ammunition stronger.

(Slight aside but I do think the closing quote by Woelfel makes a good point)

IshKebab• 2 months ago

How did he not get expelled for that? Jesus.

Deepfakes Tore a High School Apart

Related Stories