Deepfake scams are here and we're not ready

https://twitter.com/coffeebreak_YT/status/1624868006823165953

by searchableguy • 3 years ago

67 84 3 years ago

twitter.com

rcme2• 3 years ago

No… we’re ready. Celebrity endorsements have always been meaningless. Hopefully more people realize this.

minimaxir4• 3 years ago

They may be useless to you, but you are most likely not the target market for these ads.

Celebrity/influencer marketing is one of the few marketing strategies that actually works consistently.

UncleEntity• 3 years ago

Probably used to work consistently is more apt.

What’s the world coming to if you can’t trust that the person pushing a product isn’t actually a paid corporate shill but is instead a scammer trying to separate you from your money.

There’s probably a distinction between the two, right?

Paris Hilton promoting CryptoKitties vs a fake Paris Hilton promoting CryptoKitties — the difference is obvious.

PheonixPharts• 3 years ago

I believe the subtle point being made is that the fear of AI here is that it will subvert out expectations of authenticity in order to sell us a product.

However influence culture already does this. Influencers represent identities fostered entirely for the purpose of convincing you to consume product.

You could make similar claims for the mountain of garbage writing that falls under the category of "content marketing". It's an attempt to create authentic information for the real reason of selling you a product.

The point being that influencers, content marketer and similar sophisticated advertising strategies already are artificial intelligence. Joe Rogan is a personality created to sell you products, most blogs are empty content designed to attract your attention just enough to convince you to trust them to sell a product.

The distinction between the real Joe Rogan selling you a product and an artificially generate Joe Rogan selling you a different product is only a concerning thing to people paying Joe Rogan to sell their product.

For the average consumer, they have already have been duped by AI for years now.

sandworm1012• 3 years ago

Kardashian? No. Will not impact my buying habits whatsoever. But if LTT says to stay away from a product, I would be very very hesitant to consider it. If Attenborough does a psa, I will 100% take his opinion into consideration. Not everything is about "influencers".

rib3ye• 3 years ago

Is this an argument against LTT being an influencer?

darkerside• 3 years ago

Whoever it is, I typically want to hear their reasoning anyway. They can get my attention this way, but not my compliance against my own interests.

laurencei• 3 years ago

Doesnt that just mean you need to validate the source of the video? If LTT says to stay way from a product on the LTT main youtube channel - that would seem valid?

If the video was a re-post or hosted elsewhere - then you would start to reconsider it?

kevin_nisbet• 3 years ago

I think this is the immense difficult with plausibly fake information. The target doesn't just need to be influencer did X. To follow the LTT example, maybe LTT says stay away from my product, so I do deep fakes to erode trust and viewership in LTT. And this moves the needle towards other channels that maybe aren't quite so critical, or willing to sell their souls with them VPN ad money.

And most people say they wouldn't fall for it, but for some things, you might not need to move the needle very far.

jay_kyburz• 3 years ago

Just wait till somebody deepfakes Sundar Pichai saying something dumb, tanking Googles share price for a few minutes, but long enough for that somebody to get rich.

I think it might be refreshing for everybody to completely disbelieve everything they see and hear on screens. Where we can only believe what we can hold in our hands in front of us.

It's going to be hard to run a democracy though.

minimaxir4• 3 years ago

Deepfake technology has been in a good enough state for years that a bad actor could do just that. There are many reasons they haven't. (How would you distribute a deepfake of Pichai such that the markets would take it seriously?)

Even during the crypto boom there were deepfakes to manipulate the market, which didn't even work.

haswell• 3 years ago

Huberman isn’t a celebrity though, and is known for giving out advice that is usually at least somewhat credible.

bruce511• 3 years ago

Endorsements are not the problem. They are clearly promoting a product, and we all presume they've been paid to do do.

The real problem are the not-endorsements. The "our cameras caught this when he thought his mic was off". For politics especially its the unscripted moments people believe.

Equally it offers plausible deniability to anything that is captured. Just claim its a deep-fake and the vast majority of supporters will believe you,because they want to.

Phrases like "a picture never lies" or "I'll believe it when I see it" all start falling away.

In a world where people want to be right, not informed; when anger wins more elections than hope, where fear of the future overwhelms the comfort of the present, deep fakes erode one more thing you can trust.

thatguy0900• 3 years ago

Exactly. I'm not worried at all about fakes. I'm worried about people claiming fakes. If ChatGPT can perfectly imitate anyone's speech patterns, and you can fake someone's voice and video perfectly, everyone with power in the world basically gets out of everything. Evidence will be meaningless.

orblivion• 3 years ago

I want to be caught off guard by one of these to know if I'd actually fall for it. Knowing what it is upfront, Joe Rogan sounds a little like a robot to me in this video.

fsckboy• 3 years ago

convincing me that Joe Rogan said something is going to fool me how?

Gigachad2• 3 years ago

After seeing the deep fakes of NileRed on YouTube, I constantly feel like the real videos are fake and too robotic too.

readthenotes1• 3 years ago

I nearly fell for the Queen saying "There are few things more hurtful than someone telling you they prefer the company of Canadians." And that was just a few years ago...

The best lies, and the best cons, always use the truth as a weapon

RcouF1uZ4gsC• 3 years ago

Interesting that this was posted on the date of the Super Bowl.

At last year’s Super Bowl there were real celebrity endorsements of a real scam, FTX.

Honestly, the big scams are going to spend real money and get the real endorsements. I don’t know how much difference in real life deepfakes will make.

red-iron-pine3• 3 years ago

FTX had superbowl tier money to spend, and was aiming at the whole country.

these deepfakes will be done on the cheap, and aimed at those likely to buy the BS -- i.e. your mom. Don't mean that as in insult ("yo mama..."), but like literally, these are the scams that will target elderly, non-technical women, kids, etc.

xwdv• 3 years ago

Deepfake scams to sell products is the least creative way to use deepfakes.

The real value is in creating deepfakes of politicians that can influence elections.

red-iron-pine3• 3 years ago

We've already seen how easy it is to get fake politicians made of flesh. Santos and Luna being the most recent, but no shortage of 'em beforehand. Why add tech to the mix when you can just promote idiots and liars?

impalallama3• 3 years ago

I’m at the point where I genuinely would like to see this ai stuff heavily regulated to outright banned. Any upside where Disney gets to not pay actors and instead just synthesize Princess Leia or Darth Vader for Star Wars XXIV is vastly outweighed by the sheer amount of fraud that can be committed when we have hyper realistic ai voices calling in our family members voices asking for crypto deposits.

Anyone who disagrees join my start up where we generate the voice of dead family members so we can get people to pay indulgences to escape hell.

greesil• 3 years ago

When AI is outlawed then only outlaws will have GPUs or something

tarr11• 3 years ago

Banning or regulating AI wouldn’t impact scammers at all.

throw10920• 3 years ago

Or hostile nation-states, for that matter.

janalsncm2• 3 years ago

I think it would. There’s a degree of expertise required to create a good deepfake. Banning it will reduce the pool of professional deepfake artists which 1) drives up the price of deepfakes and 2) makes it harder to secure good ones. And since the effectiveness of a deepfake scam depends on the quality of the deepfake, fewer people will be scammed.

impalallama3• 3 years ago

And yet I can’t just Google “how to build a Bomb at home” and click on the first result and get detailed instructions for how to be a little terrorist.

teolandon3• 3 years ago

I like how not paying actors is listed as an upside.

johnsillings2• 3 years ago

to be fair to the commenter you're replying to, they said it was an upside for Disney

red-iron-pine3• 3 years ago

The AI content will show up on time, not complain, not be a premadonna, not get tired or run afoul of union rules, not try to do their own stunts and then delay a production for months (impacting tons of other staff), not accidentally shoot someone on the crew, and not molest its coworkers. And that's just off the top of my head, there will be plenty more reasons.

No surprise Disney (and others) are going to be getting on that bandwagon.

anon_cow11112• 3 years ago

And now the hard part; how do you ban "AI"? it's so vague, and the law has a hard time with vague concepts.

I'd like to ban "social media", but how do you define that in a way that kills what you want (eg twitter, facebook, /r/all) and preserves less detrimental forms of it (eg HN or old phpbb forums) ?

edit- Oh and furthermore, how do you deal with every country who decides to not ban it?

impalallama3• 3 years ago

Well in my case I specifically mean hyper realistic deep fakes of people created without their consent.

Why is it the difficulties of regulating something brought up as if it means it prevents any action taken at all.

“Id like to ban murder but how do how do you define murder and ban the cases we want (murder) and allow the cases we do (self defense)? And what about countries who decide not to ban murder?”

throwawaylinux• 3 years ago

This kind of thing -- centralized action to shield and protect people from having their feelings/wallets/egos/etc., hurt always struck me as the complete opposite way to address things and yet it's always so popular. I understand governments want people to be as helpless as possible and reliant on the state controlling others in order to function so I see why it's pushed from the top down. I just don't see how it gets any traction.

People should have to learn not to be scammed rather than trying to ban scams. People should be taught that speech is not violence instead of trying to ban speech.

If this development is what it takes to finally empower people to not take their health and fitness advice from Joe Rogan and some other internet chumbo, I say it can't come soon enough. That a few additional people might fall for these things is a small price, and one I'm willing to pay.

sneak• 3 years ago

There are already laws against fraud.

What you are talking about is simple prior restraint on computer generated art, in the cases where the speech is not already covered by fraud laws.

nl4• 3 years ago

A call to "ban AI" on HN is one of the lowest point I've seen in my 15 years on this site.

Not only is the behavior that is being complained about (people fraudulently pretending to be something they aren't) already illegal in most jurisdictions the unintended consequences of such a poorly thought out policy are complete unconsidered.

The objections are obvious but to list some:

* Behavior should be regulated, not technologies. Otherwise someone will use an alternative technology to perform the same attack and it won't be covered.

* This behavior is already covered by fraud regulations

* How would you even "ban AI" when people can't define it

* The only approach that could be considered would be to take a licensing approach to writing any software. The downsides to this are considerable and hopefully self-evident to this audience.

idontwantthis2• 3 years ago

I wonder if this will kill “influencer” as a career. If I can pick between deepfake Kim Kardashian and a real tik tok star I have to pay more and actually talk to, I think I’d pick fake KK. She should start authorizing deepfakes and take in cash while literally doing nothing.

duxup2• 3 years ago

People still listen to the real personalities via their official outlets.

hackernewds• 3 years ago

KK is an influencer as well? What is your point

idontwantthis2• 3 years ago

I mean that maybe there will be no more market for anyone not already established.

duxup2• 3 years ago

I feel like fake celebrity endorsements for scam products are the least of our concerns.

johnsillings2• 3 years ago

For anyone who hasn't seen his videos, Coffeezilla's (the guy who tweeted this) Youtube channel is fantastic: https://www.youtube.com/channel/UCFQMnBA3CS502aghlcr0_aw

Javantea_• 3 years ago

In what way are we not ready? If you don't have the necessary critical reasoning skills to doubt Joe Rogan telling you about complete bullshit, no one needs technology to trick you into a scam. If you have critical reasoning skills, it's pretty hard to convince you to part with your money. Who exactly does this scam target?

jhanschoo• 3 years ago

Say you have a child or close niece or nephew with insufficient latent opsec and a lot of public social posts from which an attacker can mine their voice and compromise their account, and they call you asking for urgent help...

LightHugger• 3 years ago

This has been happening to a few relatives for about a year now. The niece calling for help from "jail" using an impersonated voice, it's quite impressive. The niece being impersonated has some shows, fairly small but enough to impersonate!

nobrains• 3 years ago

The video was created to highlight an extreme, where it should be very obvious that it was faked. However, an actual attack will be subtle.

deafpolygon• 3 years ago

Oh I'm ready. I assume everything on the 'net is a possible scam.

corbulo4• 3 years ago

We've actually been approaching being ready for a few years now. Now more than ever people have begun to add a layer of skepticism and treat information as separate from reality itself.

Remember 10-15 years ago? If it was on the news, thats because it happened. If an important politician said it that's because it's important. If the 'expert' said it then it must be true! If you need a refresher go visit old political threads on reddit from 10+ years ago. You'll recognize your old sheltered political views, and yes it was somehow even more naive and coddled than reddit of today.

Deepfakes will be a net good. It will make more ideas have to stand on their own merits (because there will be limited authority to validate the medium thats conveying it, mitigates reliance on fallacies) if you can make <famous politician> say <total opposite of their entire platform>. It will put a damper on how seriously people take everything on the internet outside of official sources (thank god). It will effectively weaponize social media's ills against itself, the more its abused the greater that effect.

People are finally starting to ask the questions they should be asking on a large scale, "where is chatGPT taking us, do we want more of this?", those kinds of meta-questions were never being asked even a few years ago. Almost no one asked that about smartphones when they came out. Better late than never.

vlunkr• 3 years ago

Now more than ever people have begun to add a layer of skepticism and treat information as separate from reality itself.

This is pretty strong blanket statement. A disturbing amount of people have bought into conspiracies like Qanon in the last few years.

corbulo4• 3 years ago

Those are a symptom of burgeoning skepticism. Just because people are acquiring newfound skepticism doesn't mean they'll believe all of the optimal and reasonable things. It goes back to normal where much like for all of previous human history, people believed ridiculous rumors all the time.

It was only the last 100-150 years or so that people have really gotten locked in to the single minded ideology wars while totally locking down their sources of information.

xboxnolifes2• 3 years ago

Believing in conspiracies is itself skepticism. The idea behind nearly all of them is that what people in power tell you is not true and there is a separate explanation.

politelemon• 3 years ago

The internet, and the information networks it could support, was supposed to be the very thing you describe, but all it did was create bubbles. All this will do is reinforce those bubbles as the echo chambers cherry pick what is convenient to their causes.

corbulo4• 3 years ago

If every bubble has a video of the same authority figure(s) (within the group or outside of the group) saying directly contrary things it has to have an effect.

The internet really was that though until the last 8-10 years. Once it reached a critical mass and started being actively gamed by world govs as a major objective it changed dramatically to what it is now. Now the masses are becoming just as paranoid and skeptical as a baseline as the cypherpunks were.

danaris• 3 years ago

It will make more ideas have to stand on their own merits

This is a reasonable position to take, but I'm afraid it's somewhat at odds with the current reality on the ground.

The problem being stated here isn't that "absent the ability to provide image/video/audio evidence and trust it as untampered, we will be left adrift in a sea of confusion", it's that we aren't ready to compensate for the problems that lack of trust creates yet.

For the vast majority of people in our society, nothing has changed since 15 years ago. They still trust that everything they see on the news (at least, the news source of their choice) is real. (This despite the fact that it's been terrifyingly easy for decades to create a narrative with no actual alteration of the source footage—just clever editing. See, eg, "The Illusion of Truth", from the 4th season of Babylon 5, aired in 1997, for a great illustration of this effect. Not to mention a great example of how science fiction draws attention to real social issues of the time, but can't easily predict what technologies we'll have available—or they just would've made deepfakes of John and Delenn there instead.)

All in all, I hope you're right—and I think that ultimately, we will develop a more robust ability, as a society, to make up our minds based on principles and values rather than on who says what...but I'm afraid that we will have some very rough times in the near-to-medium term before we can get there. Because we're not ready yet.

corbulo4• 3 years ago

I agree wholeheartedly with your general points. We are about to face an intense chaos, if we're not already in the midst of it.

I'm less pessimistic on how many people believe media at face value. There's been large reduction in people who do, and the beginning of that shift is willingness to doubt their 'trusted' sources.

When someone says 'this happened' and you say 'actually that turned out to be fake news/not the whole truth' there's a much greater shift in reaction to that than a decade+ ago. The kneejerk is increasingly less condemning.

smitty1e• 3 years ago

There are always the traditional vistas of retail scams.

Zetobal• 3 years ago

I don't know grandpa's "If it sounds too good to be true it's probably not true." Still works quite well.

Deepfake scams are here and we're not ready

Related Stories