It sounds like science fiction but it’s not: AI can destroy your business

https://www.theguardian.com/business/2023/apr/09/it-sounds-like-science-fiction-but-its-not-ai-can-financially-destroy-your-business

by zqna • 3 years ago

72 71 3 years ago

theguardian.com

martyvis• 3 years ago

And with a lot of people uploading reviews or DIY videos to YouTube it's going to be pretty easy to harvest enough of a voice print. If you have written comments on forums and other social media such as right here on HN they also have your writing style to work with.

jaidhyani• 3 years ago

I think we're on the verge of few-shot believable voice impersonation. Between that, realtime deepfake videos, and AIs being more than good enough to solve CAPTCHAs, it seems like we're at most a few years from having no means of verifying a human on the other end of any given digital communication unless someone figures out and implements a new solution quickly.

ChatGTP2• 3 years ago

We will just get ChatGPT 6 to solve it for us. Done.

ghaff2• 3 years ago

There are (mostly) solutions but I lot of people won't like them. As with things today like notarized signatures or just transacting in person, they basically depend on some sort of in-person attestation by a reliable authority. Of course, that means adding a lot more friction to certain types of transactions.

mdp20219• 3 years ago

few-shot believable voice impersonation

We have been there for the past few years. And yes, it has been actually used for scamming.

Submitted only a couple of days ago: Voice scams: a great reason to be fearful of AI - https://news.ycombinator.com/item?id=35459261

RandomLensman2• 3 years ago

Why would passwords, personal devices, policed platforms etc. fail as an authentication method between known counterparties? Between unknown counterparties the issue is much bigger than just about being a human or not.

beefield• 3 years ago

I can see how that might destroy many business models. But from the top of my head I can't come up with any whose loss would have a dramatic negative effect on my wellbeing. Could someone elaborate why I should be worried?

courseofaction5• 3 years ago

If a common-sense LLM is listening to grandma's calls (privacy alarms going off but hear me out), it can stop her from wiring her life savings to an untrustworthy destination, without having seen that particular scam before.

throwaway4260794• 3 years ago

Article is about deepfake-based fraud. Good to be reminded of this risk, but the headline is riding the mostly separate wave of chatgpt interest.

stareatgoats• 3 years ago

Came here to say this. It's clickbait, but a good reminder nonetheless.

skilled• 3 years ago

It sounds like you are saying that only ChatGPT can be called AI.

throwaway4260794• 3 years ago

No such implication intended.

jacquesm• 3 years ago

There are plenty of large databases of image, video and audio material of private individuals tied to their identity. Youtube, instagram, Facebook, LinkedIn etc, all of these give you access to extremely high quality material as well as a lot of context that you could use to make a deepfake more believable. And that's before we get into non-public accessible databases, of which there are plenty.

throwaway4260794• 3 years ago

Yes. Several years ago when banks started using voice pattern recognition as part of telephone banking authentication that was a pretty cool advance, but now it's looking redundant.

courseofaction5• 3 years ago

As concerning as this is, I'm wondering if LLMs like ChatGPT can be more powerful fraud-detectors than fraud-makers.

Unlike scam detection and filtering methods based on pattern recognition or reputation, LLMs can analyze the content of a scam rather than its artifacts. By that I mean, you can ask the AI "does this make sense, or is there reason to believe it's a scam? If so, how can we verify that it's not a scam, knowing that deepfakes exist" rather than "does this fit x y z pattern that we have seen before?"

I'm hoping, and I may be wrong, that most scams rely on emotional hooks, unsuspecting victims, or desperate victims. LLMs aren't vulnerable in the same way.

Maybe.

mdp20219• 3 years ago

can be more powerful _-detectors than _-makers

Surely you have witnessed (Generative) Adversarial Networks. It's called "an escalation" (attack vs defense, etc.).

More to the substantive proposal in the post: remember that incidents during surgery are dramatically reduced just by using checklists - you can already reduce incidents through deterministic regulation of action. And you do not require "fuzzy" systems like ANNs for that.

courseofaction5• 3 years ago

I agree, but the thing is, can the escalation potential of a scam which is essentially "send me your money for nothing", with extra steps, confound the understanding of an advanced LLM?

The scam has to be both human-comprehensible and plausible in order to be delivered to the victim in the first place, which places a limit on its complexity which the adversarial anti-scam LLM isn't bound to.

WithinReason3• 3 years ago

To train GANs you need access to the internal states of the generator, which you wouldn't have with these LLMs.

otikik• 3 years ago

Weapons race. You deploy a more intelligent scam detector AI, I deploy a more powerful scam generator AI.

The only ones that benefit really are the AI makers.

courseofaction5• 3 years ago

I guess this is an asymmetrical kind of warfare, not simply two AIs duking it out. It's one AI targeting a human with a proposed transaction that results in an absolute loss for the human, and one AI trying to protect against that loss by verifying the plausibility of the transaction in context.

Most scams are fairly transparent and rely on wetware failures, right? The anti-scam AI doesn't have to overcome the scam AI, it has to compensate for the emotional responses or unpreparedness of the target. I think that's less of an ask than generating bulletproof scams. I could be completely wrong.

totetsu• 3 years ago

What if we make the scam detector AI too good and it starts to point out the hypocrisy inherent to our Society

jbandela1• 3 years ago

I wonder if some of the AI fraud will drive to more in person interactions.

If you meet with somebody in person, you know they are not a bot and you know what they look and sound like directly.

hallway_monitor2• 3 years ago

I hope so.

You already need to meet someone in person to get a true feel for them. There are so many things that don't come throug video. Smell, for instance, generally plays an important role in our perception.

dgellow• 3 years ago

Quite sure lot of people would be happy to smell other people less :)

LinuxBender• 3 years ago

Some financial institutions already support this concept. Accounts can be flagged to require a live video chat to transfer money or even require being in-person and showing state ID.

gonzo41• 3 years ago

It'd be so cosmically funny to end up having to go do banking in actual banks because silicon valley innovated us into a dead end with AI.

tekknik3• 3 years ago

History is circular. Another example, we have cars and yet are pushing to move back to not having them.

FpUser2• 3 years ago

First - it is a clickbait. Second - if am being asked to transfer money by say my fake relative I would surely do some verbal grilling first. Are they saying that listening part of AI can perfectly decipher all the questions and give proper answers? I doubt it. And even then I will make bank transfer to a relative. How would scammer actually get it?

As for bank manager transferring hefty sum without some proper verification - the problem is not scammers but the manager themselves.

vba616• 3 years ago

even then I will make bank transfer to a relative

The way the scam is said to work is that someone pretending to be your relative is telling you they are in an unusual situation away from home and it is very urgent so you must send them money in a different way.

e.g.

"The caller said she was locked up for a misunderstanding between herself and the Mexican state police of Quintana Roo. They needed money for a Mexican lawyer and court costs"

(That particular incident, apparently they were impersonating someone who had been to Mexico, but had - unfortunately for the scammer - returned prior to the call)

https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-us...

tekknik3• 3 years ago

I also certainly don’t know my own account numbers, much less a relatives. How am I going to know if they’ve given me the correct account details?

brightlancer• 3 years ago

Second - if am being asked to transfer money by say my fake relative I would surely do some verbal grilling first.

Sadly, you are only one person on a planet of more than 7 billion.

I worked with someone whose mother had been scammed multiple times by folks claiming to be his (deadbeat) brother. Social engineering works, especially as the attacks get more sophisticated.

derbOac• 3 years ago

I wonder if this will drive adoption of secure voice communication ala Signal or something.

mdp20219• 3 years ago

secure voice communication

How would you implement that? You cannot certify a timbre you can simulate.

SAI_Peregrinus• 3 years ago

You can certify the endpoint hasn't changed its keys, as Signal currently does.

Nasrudith• 3 years ago

Only bit I could see helping is certificate based identity verification. It wouldn't help with "Biden admitting he is a space alien" type content verification.

eddhead• 3 years ago

Yes it can. Especially tabloids like The Guardian. Fairly easy to come up with the soft of stuff they wrote. Including this hyperbole.

mdp20219• 3 years ago

Especially tabloids like

What about The Spectator - https://news.ycombinator.com/item?id=35459261

The very agents that can be scammed into damaging you are not aware of the state of facts. (They may read those "«tabloids»". It's educational.)

dr_pardee2• 3 years ago

While I am aware that it is possible to use software to clone someone's voice, I am uncertain if it is currently feasible to utilize the cloned voice in real-time situations.

I am curious if these scammers are pre-recording conversations and playing them back in real-time or if they are actually able to speak using the cloned voice technology.

tylergetsay• 3 years ago

near real time is possible, there's lots of "v-tubers" doing it

aqme28• 3 years ago

If it's not now, it will be soon.

mdp20219• 3 years ago

feasible to utilize the cloned voice in real-time situations

Yes, it has been so for years. The attack to the Dubai Bank in Hong Kong was effected in January 2020.

https://news.ycombinator.com/item?id=29711876

dr_pardee2• 3 years ago

From the article https://www.unite.ai/deepfaked-voice-enabled-35-million-bank...

"Though users at the Audio Fakes Discord and the DeepFaceLab Discord are intensely interested in combining the two technologies into a single video+voice live deepfake architecture, no such product has publicly emerged as yet."

justinclift• 3 years ago

The video's shown on YouTube have a number of "plausible" common responses pre-created, that can be played back by pushing a button.

In operation in the video's it seems to work pretty well. :(

JayStavis• 3 years ago

60ms here: https://twitter.com/nearcyan/status/1643419486400159744?s=46...

m0llusk• 3 years ago

Some more descriptive terms might be helpful. AI sounds like something that has agency and makes decisions. This is about ML models to generate speech which is at a different level.

flappyeagle• 3 years ago

The election season is going to be out of control.

Fake ads for your opponent in their voice spouting bonkers positions and nonsense. Everywhere. Online, voicemails, far as the eye can see.

It’s gonna be rough.

ManlyBread• 3 years ago

It's going to be taken about as seriously as random photoshopped images were until now.

mdp20219• 3 years ago

I had in fact swiftly daydreamt (day-nightmared, more like) the scene of the watch-the-world-burn'ers discussing plans of the prank while on this page.

But the education of the public has already started, with the deepfakes that have circulated in the past few weeks, from odd or dreamed depiction high "authorities" to discussions about the status of documents related to e.g. the protests in France.

blibble• 3 years ago

this one will be broadly recognisable

personally I'm more worried about the next one where the AI infestation will be far more subtle

speed_spread• 3 years ago

Your business? AI can destroy your whole country. Just wait for the next elections.

JoeAltmaier• 3 years ago

Already got people fired. Copy-writers are being replaced by a manager getting an AI to write bland drivel. Also model-designers for software (games). It's only going to expand.

rcarr• 3 years ago

I watched a video on how they made the mandalorian today and Hollywood is basically moving away from green screens towards gigantic curved lcd screens which have scenes created in a 3d games engine. It produces more natural lighting results and is better for actors and cinematographer to work with. It’s far cheaper than shooting on location but I wonder if there’s going to be a net loss in jobs when when you can just have a couple of prompt engineers and maybe one or two artists to make tweaks to get any shot you want.

Video for anyone interested:

https://youtu.be/Ufp8weYYDE8

It sounds like science fiction but it’s not: AI can destroy your business

Related Stories