Korean women remove pictures, videos from social media amid deepfake porn crisis

https://www.channelnewsasia.com/east-asia/south-korea-criminalise-deepfake-porn-protests-4642616

by popcalc • 2 years ago

51 94 2 years ago

channelnewsasia.com

up-n-atom• 2 years ago

Makes sense and I can see this becoming a global trend if it's not already. But the irony of the news agency taking and posting their photo/video is a facepalm.

EDIT: I can foresee a dark future where surveillance video is monetized for such salacious purpose and it becomes inescapable as the cameras are everywhere.

danparsonson• 2 years ago

That is the dark present in South Korea as they have a huge problem with cameras hidden in hotel rooms and even toilets :-/

Simulacra• 2 years ago

Could be worse than that. With all of the facial recognition and videoing that's going on by the government, if the police wanted to, they could insert you into any footage committing any crimes they want.

ssalka7• 2 years ago

I could very well be wrong about this, and usually I fall more on the e/acc side, but I think deepfakes are perhaps the gravest immediate concern of AI. Yes, they've been around for well over a decade, but never before has their creation (& thus volume) been so accessible to the average layperson. A simple 10 minutes on CivitAI will tell you all you need to know about how prolific this problem is.

IMO it is only a matter of time before this same issue comes to the forefront in the US (I'm kinda surprised it hasn't already become a bigger topic). I could easily see custom celeb LORAs/emmbeddings and realistic SD/SDXL checkpoints being banned outright for this reason. Yes, it would be an infringement on free speech, but weighed against the interests & safety of the public, the courts might see a move like that as reasonable.

transformi• 2 years ago

Banning Loras/Embedding won't help that..cause you can generate realistic images with only one reference photo these days...

ssalka7• 2 years ago

Fair point, technologists will always innovate faster than politicians are able to regulate. Cat's outta the bag now, but that only means the problem will continue to escalate until more drastic measures are deemed necessary.

It's that or we will come to accept some level of societal degradation due to the abuse/misuse of AI tools becoming normalized.

slibhb• 2 years ago

I don't think of myself as on the e/acc side, but I think that deepfakes are here to stay and we'll just have to get used to them. I don't see any way to put the genie back in the bottle but I also don't think it'll take that long to adapt.

whiplash451• 2 years ago

This is giving up a little quickly on watermarking image generation.

I do think it’s a hard problem (especially if we want robustness to compression etc) but amenable to research.

m3kw92• 2 years ago

It won’t be a problem if it gets saturated, people bored of things easily

ssalka7• 2 years ago

As AI image generation (and soon video generation) gets to be a more saturated space, we will end up in the state that you cannot trust anything you see online, even if you are educated, well-informed, and meticulously research the topic.

It has already been inching that way for a long time, but generative AI definitely is adding new fuel to the fire.

mingus882• 2 years ago

Ah yes, “waiting for the sexual abuser to get bored” sounds like a good tactic and very comforting for young girls

bobthepanda• 2 years ago

South Korea has a very bad gender relations problem partially stemming from an extremely lopsided gender ratio. In the 90s it reached as high as 115 male children to 100, and now all those extra young men are angry adults. https://theconversation.com/south-koreas-gender-imbalance-is...

comboy2• 2 years ago

It's ban photoshop all over again.

It's just the new reality. Porn is not even the worst thing. If anything it can make real photos/videos harder to get viral.

But augmentation will be (if not already is) used to make very tiny adjustments, like you make one candidate/celebrity face slightly more symmetrical and the other one less, change their gait just a little bit. These changes are non-perceptible, are not interesting enough to make a scandal, and they can have huge impact on how people perceive other people. I mean, I am not suggesting that it isn't a puppet show without those, but the game is evolving.

gjsman-10002• 2 years ago

The above sounds like the opinion of someone who doesn’t have a daughter.

ultimafan6• 2 years ago

Photoshopping something convincing has a much bigger bar to entry than (as far as I understand it) generation of deepfakes now. Lots of time/practice invested even for one convincing image.

From what I can see the issue here is that this problem with AI can potentially become much much more widespread because the amount of work required is pretty much none and gets lower as models get better. It sounds like someone using deepfakes to ie extort people can just cast a much wider net by just scraping/dumping masses of images into a generator to create a multitude of convincing pictures of many different people instead of having to focus their time and effort on picking a single target and creating fakes manually.

JumpCrisscross10• 2 years ago

ban photoshop all over again. It's just the new reality.

We didn't solve teens doctoring inappropriate images of their classmates by banning Photoshop or accepting it as the new normal. We solved it by passing laws which criminalise the behaviour and enforcing them to deter the behaviour.

Technology influences society. But plenty of social mores are almost universally respected despite being difficult to enforce. Possible and difficult to enforce doesn't force permissibility.

0x000xca0xfe• 2 years ago

Photoshop was never built as an easy tool so powerful that even grandma can produce photo-realistic fake images in seconds. The new wave of AI powered editing tools are clearly on a new level.

This is like bringing your howitzer to a gun debate because they are kinda similar.

RajT883• 2 years ago

Probably half the shorts on Facebook are pretending not to be AI generated now. And a similar number of the photos in your feed advertising groups.

2OEH8eoCRo03• 2 years ago

Yes, it would be an infringement on free speech

Freedom of speech is mainly so that we can criticize the government. How does deepfake porn or impersonation have anything to do with speech?

JumpCrisscross10• 2 years ago

Freedom of speech is mainly so that we can criticize the government

Freedom of speech is a bigger concept than the First Amendment. Technically curtailing one's ability to make AI porn of your colleagues is a restriction on speech. That said, so is every spam filter, fraud statute and retaliation against a declaration of war. A right doesn't have to be absolute to be good (or vice versa).

ssalka7• 2 years ago

Freedom of speech is there to protect objectionable/offensive speech, whether it is directed towards the government or otherwise. Objectionable speech is frequently in the crosshairs of pro-censorship folks, and many people think of deepfakes as objectionable – reasonably so.

delichon• 2 years ago

I think freedom of speech is mainly about freedom of thought and just plain freedom, and the ability to criticize the government is a side benefit. And that it should also include the freedom to augment our imaginations with the prosthetics of our choice.

pessimizer• 2 years ago

I could very well be wrong about this, and usually I fall more on the e/acc side, but I think deepfakes are perhaps the gravest immediate concern of AI.

I don't think they're a concern at all, other than that they will ruin the evidentiary value of photographs. They'll do that whether or not the general public has access to them.

The concern about fake pictures of people naked is stupid. The same process mentioned above happens to naked pictures of people: they lose evidentiary value. General access to the ability to make deepfakes not only makes it so people won't believe what they see now, but also won't believe actual revenge porn. It simply ceases to be a problem. You can claim it's a fake, and no one can disprove it. This is a reason for celebration.

The standard response for that has been for people to simply claim that it won't matter, and that people will destroy the life of some young innocent girl if a mean boy spreads fake pictures of her naked. The answer is that people who harass a young girl are guilty of harassment, and should be jailed. You shouldn't be harassing people even if naked pictures of them are real, which they often are. And if you'd attack a person over pictures that you could fake up yourself in five minutes, you're either a) a moron or b) somebody using the photos as an excuse, and it would be a benefit for society for you to be locked away from it.

I suspect a lot of people really worried about deepfake porn deep down are worried about their ability to properly identify and harass the harlots if people won't believe the pictures that you send to their family and friends anymore. From my perspective, they're literally fighting to preserve revenge porn.

ssalka7• 2 years ago

ruin the evidentiary value of photographs

This is going to be a monumental issue when the fact that we can't trust any photographs fully takes hold. Maybe watermarking will help with that. More likely people will develop tools to circumvent watermarking technology (either on the imprinting side, or on the verification side).

people won't believe what they see now, but also won't believe actual revenge porn. It simply ceases to be a problem.

This may be a comforting way to think about it, but it won't work in all cases. Consider instead of revenge porn, the case of CSAM – that will never be a solved problem until 100% of it is wiped from the face of the earth and it is never produced again. Making it less believable because there is so much AI-generated equivalents out there is a not a solution to the problem, it exacerbates the problem.

ultimafan6• 2 years ago

I think the problem is that people aren't necessarily logical or rational about such things. I know someone (family friend) who was the victim of similar harassment with (clearly) photoshopped images being sent anonymously to her work, family, friends on social media. The fact that they were fake didn't diminish the humiliation or stress she went through and I don't think she ever really recovered from the trauma of having some anonymous asshole harassing and stalking her and her family online. It eventually stopped but she was afraid of it starting again for a long time. "It's fake" isn't much of a comfort to someone being harassed and being afraid of harassment escalating in one way or another. And it getting easier and easier for crazy people to do this with less and less effort doesn't help the issue at all regardless of what laws are passed. Finding better ways to deal with these people is a good thing but enabling them to make it easier isn't, in my opinion.

I think for a lot of people it's easy to die on the hill of technological progress and freedom at any cost when the consequences haven't happened to you or a loved one.

acdha3• 2 years ago

The concern about fake pictures of people naked is stupid. The same process mentioned above happens to naked pictures of people: they lose evidentiary value.

This is too narrow: it presumes rational contexts where evidence will be evaluated critically, and it assumes a mild, uncreative attacker but harassers are good at exploiting social norms. The damage caused by this kind of harassment isn’t seeing yourself nude as much as wondering or knowing what other people will think, and there’s no way to undo that. Even if people _say_ they believe that the pictures are fake, the victim may be wondering whether that’s really true for the rest of their life.

The other thing to think about is that we’re really just getting started. Most of the deepfakes which the average person has seen are fairly mainstream porn nudes: there’s a huge control issue but in general there are worse things than people thinking you have a star’s body. We’re going to see those worse things soon: it won’t be a porn fantasy but things designed to degrade and more targeted - if the target had body issues, exaggerating them; punishing the person who rejected someone by faking them looking not like a porn star but less attractive and having degrading sex with someone unpopular, knowing that a bunch of people are going to think “well, he would say that”; make some teacher’s life hell by faking them having sex with a student; etc.

Because the attacks are being directed by humans, they’re going to adjust and find what works. Anything which forces an official investigation is going to be especially bad because we can’t just ignore things like child abuse so it’s going to be like SWATing where there’s an ugly period where the wrong teenager can get an incredibly high-leverage response with a single message.

BadHumans• 2 years ago

People on HN have a hard time understanding that the barrier to entry matters. Photoshop has existed for decades but the level of skill needed to make realistic looking nudes in Photoshop is not something your average freak with a fetish possesses. New technology puts that power into the hands of every single person.

adventured• 2 years ago

It's not your average freak with a fetish that is even the primary issue. The volume will come from average anybody (although mostly men). That's the real problem. Random co-workers in the office that want to deepfake their crush. Your average teenage boy with a decent GPU that wants to deepfake their classmates just to experiment. There have already been several instances of this catching the news in Europe for example, so it's obviously going on globally.

Oh but the average guy isn't that perverted? Yes they are. We know this from the vast amount of porn they consume and of what type they consume.

s1artibartfast2• 2 years ago

Seems like revenge porn and sexual harassment should already cover the use case of distribution.

What perverted things someone does with photos in the privacy of their own home seems like it should be their own business to me.

techjamie• 2 years ago

New technology puts that power into the hands of every single person.

I think the bar is roughly the same, all things considered. If you're going to photoshop, then it's largely a matter of combining two images with a sufficient amount of masking and blurring to make it appear good enough.

With image generators, you've got a couple different options. The more trivial would be doing img2img on your target, the more complex being making a LORA.

Knowing how tech savvy the average Andy is not, I would say anyone capable of going through the steps to do the AI method could, with some brute-forcing, also manage the Photoshop method. Setting up a local instance of any image generator is beyond the majority of the population, so the closest most can get is paid online GUIs that offer an existing service that they first must become moderately proficient in for good results.

Only real difference is that anyone who can do it is much more capable of scale over time than previously. With enough Comfy nodes, you could probably even automate it given face detection and arbitrary images.

piva00• 2 years ago

The bar is definitely not the same, editing a photo on PS/other photo editors by making a collage realistic enough to cross the uncanny valley is not simply combining two images and doing some blurring and adjusting, even less if pictures are warped from different perspectives, lenses, etc.

Doing that on an image generator is trivial, there are whole websites with the sole purpose of inputting a seed image to be transformed through a prompt, that's a much lower barrier of entry for someone trying to make a fake nude. And it scales, one can do this in a batch in a few minutes/hours, doing that manually on a photo editor requires a lot more effort and knowledge.

multimoon• 2 years ago

I believe that there are unfortunately premade websites that do this for you already, I believe those were what generated all the news stories from Europe mentioned elsewhere in this thread.

I don’t think the bar is in the same state.

BurningFrog2• 2 years ago

When I grew up "a photo never lies" was a rule you could rely on. Yes really!!

Photoshop and other advances long since killed that certainty, and now we have a healthy distrust of pictures we're not sure where they came from.

I expect videos will now walk a similar path, and life will go on with no major disasters.

singleshot_• 2 years ago

When I grew up, my friends dad had a picture taken of his British public school class taken circa 1955. In this photo, which was taken by a slowly rotating panorama camera, a lad started at the beginning of the photo, on the viewers left. Once the camera panned enough that he was out of frame, he sprinted behind the bleachers to the viewers far right, and took his place on the bleachers before the camera panned to take his picture for a second time.

Kindly, if you grew up after this picture was taken, and you believed the guideline “a photo never lies,” you were wrong.

Manuel_D8• 2 years ago

Doctored photos have been around for most of the last century.

https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...

colonelspace6• 2 years ago

The appearance of doctored photographs coincides with the appearance of photography.

colonelspace6• 2 years ago

When I grew up "a photo never lies" was a rule you could rely on.

It was a rule you thought you could rely on.

munchler• 2 years ago

I would refer you to an interesting book called “The Commissar Vanishes: The Falsification of Photographs and Art in Stalin’s Russia”.

https://en.wikipedia.org/wiki/The_Commissar_Vanishes

BurningFrog2• 2 years ago

I should have remembered that I'm writing to an audience prone to passionate literalism...

I'm of course aware that there were forged photos back in the day. But they were rare and hard to make. They were usually paper only. Producing a fake negative was really hard, and a forensic expert could often spot it.

falcolas• 2 years ago

now we have a healthy distrust of pictures we're not sure where they came from.

Respectfully, we only wish this were the case. There are, right now, AI generated images and videos of a presidential candidate in a disaster area. People want to believe they are real. And so to millions of people, they are real.

People do the same with faked nudes. They will believe in those nudes because they believe that women are inherently sinful (huzzah for "original sin") and absolutely would do such things.

Oh, and Photoshop never had a dedicated app (or 100 dedicated apps) for creating nudes from regular photos and/or videos. It took a skill a scant fraction of a percent of people had.

SmitheryB3• 2 years ago

I feel as though I'm the only one who doesn't care about this.

Curious to know if 20-30 years ago, as graphic design tools got better, were people freaking out about being able to superimpose another person's face on a the body of p*rnstar?

Superimposing people's faces on to porn star's bodies happened/s, but does anyone care really even notice or care about it now days? For the most part, of course not.

So why should we care about porn videos? What makes them so different? The whole concern smells like typical media hype (i.e. yay - we get to write about AI + Porn - instant clickbait) that nobody will be able to stop and nobody will even care about in 1-2 years: e.g. "oh, yah, another deepfake of <(politician|actress|colleague^)> in a porn film - how incredibly creative".

^ Incidentally, this was made light of in the British version of The Office where a modified image of the boss, David Brent, was sent around the office. If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.

colonelspace6• 2 years ago

It's more problematic than photoshopping a person's face was decades ago because we didn't have pervasive online social media decades ago. Computers weren't in your pocket, and an offending image couldn't be beamed to everyone you know instantly.

If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.

There are plenty of mainstream TV jokes about horrendous things. Our ability to joke about something is a poor way to gauge how concerned we should be.

SmitheryB3• 2 years ago

an offending image couldn't be beamed to everyone you know instantly

Is your concern that recipients would think it's real? Or is it that people will get an email/chat message containing NSFW material? (because the latter happens now, and is nothing more than background noise at most). What is your actual, specific concern? (genuinely, I have no idea, but want to try to understand - to me it feels like panic about a problem that isn't actually a real problem).

frognumber• 2 years ago

I feel like this is a problem without a solution, or at least without a solution more expensive than the problem:

* I suspect that, like prohibition or DMCA, criminalizing behavior will lead to worse outcomes than accepting deepfake porn sometimes happens, at least if laws are passed targeting this in isolation.

* I suspect that before acceptance, many people will be harmed, on both sides. Humiliated teen girls. Immature hormone-driven teen boys.

* I suspect that we'll see a range of bigger problems such as real scams where, for example, a deepfake child calls their elderly parents and empties their bank account. Or, perhaps, a deepfake general calls someone in a nuclear bunker.

* Deepfakes are at the tip of technological problems which need solving. We have other things, like the cheap and easy availability of technology to make the next Covid-19. The point is we have the technology to make it in a lab, and the price point is going down by an almost Moore's Law like curve.

I don't see a good solution short of completely rethinking democracy and capitalism from scratch. The first time around took the Age of Enlightenment and the Age of Revolution.

It's perhaps time to do that.

Prove me wrong. Seriously. If there are incremental solutions which stop deepfake porn, deepfake scams, personal drone strikes, and super-viruses, please let me know what they are.

ChrisArchitect• 2 years ago

Subheading on this describes the story a bit better:

South Korea’s national assembly on Thursday passed a Bill to punish people possessing, purchasing, saving or viewing deepfake sexual materials and other fabricated videos.

oska• 2 years ago

Laws that prosecute ppl for possessing, downloading and storing computer files are deeply problematic, in my opinion. Whatever the content of those files.

(Publication of the contents of the files is another matter. I appreciate how this can be cause for prosecution.)