The Terrifying A.I. Scam That Uses Your Loved One's Voice

https://www.newyorker.com/science/annals-of-artificial-intelligence/the-terrifying-ai-scam-that-uses-your-loved-ones-voice

by cromulent • 2 years ago

60 38 2 years ago

newyorker.com

treme• 2 years ago

whatever happened to that yc company that was providing this service

minimaxir• 2 years ago

ElevenLabs (not YC), which powers most of the popular voice cloning controversies that make mainstream media, just raised a round at a valuation of $1.1B: https://www.reuters.com/technology/voice-ai-startup-elevenla...

simultsop• 2 years ago

https://shorturl.at/iHPQW

shrubble2• 2 years ago

It's right out of Terminator 2, except for the actually dead foster parents part.

MrVandemar• 2 years ago

It's right out of Terminator.

The voice-altering scene in Terminator 2 was a replay of the original.

shrubble2• 2 years ago

You're right! Somehow the T2 scene stuck in my mind more...

rKarpinski• 2 years ago

What's wrong with Wolfie?

blibble• 2 years ago

OpenAI's military partners are likely working on that bit

devjab• 2 years ago

As horrible as this is, I suspect this will be far worse in enterprise companies. We’ve already seen issues with email impersonation where top managers or CEOs order money transfers, and since this works better than it should, we now use confirmation of some sort. Imagine if you’d get both the email and a call, maybe even a video call in the future with the confirmation… Especially CEOs will be very “easy” to impersonate since there is often quite a lot of video footage freely available of them.

You’re going to have to rely on personal confirmation, or digital signatures to do these sort of transfers. We use the digital signature since we have a system for that in Denmark, but we can’t use that for our Japanese department as an example as they don’t have a similar system which integrates with our national digital ID for companies. I guess our national system will eventually be extendable to Japanese citizens in some way, but it’ll likely never be extended to al the countries we operate in for various reasons.

And this is us being sort of ahead of the curve on this, even for most companies around here in the worlds most digitalised country (unless that’s Estonia now). Interesting times.

BOOSTERHIDROGEN• 2 years ago

Interesting any article about the systems on denmark

piva00• 2 years ago

It's more than Denmark, the Nordics in general use digital signatures for accessing banking, government services (tax agencies, healthcare, pension systems, etc.), online payment ID checks, and so on.

In Denmark I believe NemID/MitID is the most used one, here in Sweden we have BankID (most used), and Freja eID, there are a few smaller ones as well; Norway has their own BankID that is independent from the Swedish one; Finland has their own as well (and I heard it's been live before the 2000s).

codingminds• 2 years ago

I don't know about the danish system, but I guess it's similar to the ones used by Norway and Sweden

https://bankid.no/ (Norway) https://www.bankid.com/ (Sweden)

macrolime• 2 years ago

You can use digital signatures without relying on a national system. There are plenty of available digital signature systems available. While that may not work well for communicating with external actors, within an enterprise it should be no problem.

hiddencost• 2 years ago

Japan is an interesting example. My understanding is that Japan relies heavily on the Hanko (a physical stamp), which makes business difficult as agreements need to be stamped in person.

_visgean• 2 years ago

I wonder if the problem is with impersonating C-level executives or if the problem is that they have power to just send money.. In recent John Oliver episode there was banker who simply bankrupted his bank by sending money to the scammer.

Or this guy https://www.npr.org/2019/03/25/706715377/man-pleads-guilty-t.... Do these companies not have proper process for sending so much money with multiple levels of people involved?

Turing_Machine2• 2 years ago

I wonder if the problem is with impersonating C-level executives or if the problem is that they have power to just send money.

Yeah, I guess I'm surprised that executives are allowed to do this without a "second opinion", so to speak.

In my (thankfully) long-ago days as a manager, I had a petty cash fund (used for stuff like office supplies and coffee filters and such). I'd spend that as I saw fit, and just send in my receipts once in a while (usually only when the petty cash box got low). No one got too worried about that.

Above that, I could make medium-sized purchases without approval, but I had to send in the paperwork immediately, along with a justification of why I'd spent the money.

Anything above that required approval at a higher level.

At the very least, it seems like both the CEO and the CFO should have to sign off on any money transfers of a potentially company-bankrupting size.

Maybe even add the CTO (or some other executive), just to have a third watchdog on the case.

vermorel• 2 years ago

As a CEO, this is one of the many reasons why I do not do money transfers myself.

Ps: your insights are spot on. Been plagued by this issue for years. Deep fake is merely a refinement of a pre-existing fraud pattern.

saliagato• 2 years ago

This sounds so fictionalized that may be fake

notTooFarGone• 2 years ago

Anyone believing AI will be a nice thing in the future: With data from social media it's a fucking nightmare - billions of people having their relationships visible to everyone and a singular voice recording/video is enough to scam your friends, family, workplace.

I don't see a future of the current communication channels without strong authentication measures - people already get scammed with shitty tactics by billions of dollars per year and now you can automate it.

lynx232• 2 years ago

I doubt a single voice recording could generate speech with enough emotional depth that I'd be fooled by a fake call from my better half. Do people really have such a shallow connections to their loved ones that they wouldn't instinctively feel that something isn't right? C'mon, we humans operate with a lot of extra clues. Its not like we just STT and "read" the resulting output.

idank• 2 years ago

It doesn't have to work 100% of the time to be a successful scam.

godelski3• 2 years ago

You're right to be suspicious but wrong in your interpretation of reality. The issue is that there's a big difference in how you'll interpret things in different environments and stress levels. Under pressure a lot of your normal checks will go out the window. This is why conmen often put you under time pressures, because they're stressing your system. Under normal circumstances, you're prefectly right, but under duress weird things seem normal. I mean duress is abnormal to begin with.

Also enshitification is an important aspect here. We readily overlook things like sharp cuts in video, degraded signals in video and voice, distortion, etc. Because that stuff happens normally. Post hoc it is often easy to see scams and think how dumb someone must have been to fall for something so obvious, but that's actually a tool of the scammers themselves. They love that because the more embarrassed you are, the less likely you will report it due to shame. Be careful with over confidence, because that's what they target.

nickfromseattle• 2 years ago

that I'd be fooled by a fake call

Can you say the same thing about all of your friends and loved ones?

You don't have to personally fall for the scam for it to negatively impact you. Maybe your mom falls for it, and now she can't pay her mortgage.

Fake call scams already generate billions of dollars in stolen money. Telecoms let spam phone calls run rampant for a decade before being forced to fix it. AI video/audio deepfakes at scale could 10x this industry.

beej71• 2 years ago

There was an interesting paper about a year ago that talked about synthesizing voices with minimal training data, iirc on the order of 30 seconds. The results were imperfect but wholly impressive given the input.

"Think of what will be possible just two more papers down the line."

Ferret7446• 2 years ago

From experience, it is the people who are confident they won't be tricked that are most vulnerable to being tricked.

Turing_Machine2• 2 years ago

The spouse of a former boss of mine regularly mistook me for him, if I happened to answer his phone while he was at lunch or in the toilet or something (this was long before people had personal cell phones).

Once she even said "Hi! This is your little snuggle bunny!". Mutual embarrassment ensued.

throwaway74432• 2 years ago

I recommend that everyone come up with a predetermined secret question that has both a real answer and a duress answer. Decide this question/answers with your loved ones in advance. If you're ever in a situation like this, you can ask the question to determine if its real and if they are being coerced.

atleastoptimal2• 2 years ago

this is why i believe the only possible route forward for humanity in the AI age is a biometric digital footprint

bboygravity• 2 years ago

And a biometric digital footprint cannot be stolen or replicated by AI because?

atleastoptimal2• 2 years ago

Well it has to be something tied to the body, but also unique to the individual, like a fingerprint mixed with DNA

godelski3• 2 years ago

Biological fingerprints are not that unique. Besides, how are you operating on it without digitizing it? What data cannot be replicated? The quantum no cloning theorem? Good luck getting that level of precision.

verisimi• 2 years ago

Well, I feel sorry for real kidnappers. Who's going to take them seriously now, when it is so easy to fake?! :)

In all seriousness, the solution to this and other tech dystopia issues seems obvious to me. It's un-teching things. Having to go to the bank in person, etc. Not believing screens, even corporate or government ones. (We never should have believed those anyway, so this should be a net gain for humanity!)

godelski3• 2 years ago

I wonder if these things will be the breaking point where we start to take security and identity verification seriously. I mean banks are still using SMS 2FA. Will tools like Keybase become well known and used? Yubi keys? Push into homomorphic encryption? Signal? Matrix? Don't get me started on SSNs...

I think one of the unfortunate things about our societies is that we so strongly believe "if it ain't broke, don't fix it." It can be a useful piece of advice, but we should always be looking to improve things. Hell, it's what low level research is about. Just technically everything is broken, just how much. Similarly like "move fast and break things." It can be great but it leaves a wake of destruction and tech debt. Sometimes you go faster by fixing things first instead of hacking around things and compounding.

How I think about it is that it is cheaper to perform maintenance rather than fixing things. Yet it is always easier to push off maintenance because other things are high priority. But in the long run this is far more costly. So we should often be fixing things before they are broken. But like all things, nothing is absolute ;)

wreckdropibex• 2 years ago

Identifying people online has already been solved https://news.ycombinator.com/item?id=39638744

djur• 2 years ago

This scam was being done years before AI voice cloning was possible (it's called a "grandparent scam") and so far I have not seen evidence that such technology is being used. The victims claim it was exactly like their relative's voice, but victims were saying the same thing two decades ago. The attorney mentioned in the article testified that his "son" was sobbing and claimed to have had his nose broken in the accident. Most people are not going to be able to confidently tell apart even a loved one's voice from someone around the same age if they're crying, screaming, being distorted by the phone transmission, etc.

I think it's possible that scams like this could be done with voice cloning, don't get me wrong, but it's not necessary. It's probably good that people are alert to the risk of these scams, regardless.

scrps• 2 years ago

Use shibboleths. A friend and I have a few, not due specifically to cloning, mostly inside references, things only we would get. I could think of a few with family members I could use as well.

The Terrifying A.I. Scam That Uses Your Loved One's Voice

Related Stories