Candy Wrappers Are Listening (2014)

https://spectrum.ieee.org/tech-talk/consumer-electronics/audiovideo/your-candy-wrappers-are-listening

by wglb • 9 years ago

109 39 9 years ago

spectrum.ieee.org

blt• 9 years ago

Bill Freeman, the PhD supervisor on this project, is a really creative computer vision researcher. Read through his CV, he has a great ability to pick interesting problems slightly outside the mainstream. I wish for that ability.

tristanj• 9 years ago

Fun tidbit: Three letter intelligence agencies have known about this attack vector since before this paper was published, and are worried it could be exploited in the wild.

https://www.theguardian.com/world/2013/aug/20/nsa-snowden-fi...

[British government officials] expressed fears that foreign governments, in particular Russia or China, could hack into the Guardian's IT network. But the Guardian explained the security surrounding the documents, which were held in isolation and not stored on any Guardian system.

However, in a subsequent meeting, an intelligence agency expert argued that the material was still vulnerable. He said by way of example that if there was a plastic cup in the room where the work was being carried out foreign agents could train a laser on it to pick up the vibrations of what was being said. Vibrations on windows could similarly be monitored remotely by laser.

Judgmentality• 9 years ago

I've actually done this in a lab with a laser - it's a fun trick. What the original article talks about though is passively measuring sound from objects using a camera - no laser. Although they're measuring the same thing (physical vibrations of objects) the way they're doing it is actually quite different.

petertodd• 9 years ago

A friend of mine used that technique while working in an entomology lab studying the mating habits of cockroaches. They were trying to figure out what kinds of vocalizations the cockroaches were making while mating, and bouncing a laser off their shell was by far the easiest way to record those very quiet sounds.

It's been a few years, but I think this was the manufacturer: https://www.polytec.com/us/vibrometry/products/

I wouldn't be surprised if the non-laser techniques eventually also show up as off-the-shelf products too, simply to reduce costs - a simple camera could be cheaper than a laser, and might have advantages in being able to record sounds off multiple points at once.

pilsetnieks• 9 years ago

The laser thing has been around for a while and is quite well known. The innovation here is that they found a way to do it without any lasers and possibly even with consumer cameras.

joecool1029• 9 years ago

Laser vibrometers are neat. They can also be used to sniff keystrokes remotely by pointing them at a laptop lid. Research also released before this paper.[1]

[1]https://www.youtube.com/watch?v=xKSq9efXmh8

larkeith2• 9 years ago

The linked article describes a passive monitoring system achieving the same result, no laser required.

larkeith2• 9 years ago

For more details, full paper can be found at https://dspace.mit.edu/handle/1721.1/100023

sopooneo• 9 years ago

I'm assuming it would need to be extremely high frame rate video to extreact decipherable voice data. Because even at 60 frames per second, which I understand is at the high end for normal video, that gives you a maximum frequency of only 30Hz, which is nearing the low end of human perception.

So it's interesting and a definite risk if an attacker is supplying their own camera, but definitely does NOT mean you can pull voice data from the vibrating tablecloth in a YouTube video, right?

craigds• 9 years ago

Depends what's on camera and how smart your software is. A full wavelength of audible sound is between a few centimetres and a few tens of metres. If you have a few objects in scene at such known distances, you can calculate how far out of phase they are, interpolate them and piece together a much higher effective sampling rate.

KnightOfWords• 9 years ago

They show a technique which exploits the rolling shutter effect to recover frequencies at up to five times the frame rate. On the other hand, they also mention that artifacts in highly compressed video stop their techniques from working, so any secrets in your YouTube vids are probably safe.

function_seven• 9 years ago

Not from a YouTube video, due to the video compression obliterating the small details needed to get the frequency. But the article notes that a rolling shutter allows even a 60fps camera to capture higher frequencies. Each scan line of the captured video is at a sub-frame time slice. If the object occupies a hundred scan lines, then you have 100 slightly shifted 60 hz samples that you can combine to reconstruct higher frequencies.

matte_black2• 9 years ago

What is a viable defense against this attack? Some kind of white noise machine?

azernik• 9 years ago

A private windowless room.

0xdeadbeefbabe2• 9 years ago

Skype with headphones and a keyboard for the secret stuff?

matte_black2• 9 years ago

You can point a laser at a keyboard and log the keystrokes, probably can do the same by pointing one at a headphone.

regularfry2• 9 years ago

Abstain from candy. And coke. And checks article again bricks.

spyder• 9 years ago

Even more "scary" (if it really works): using reflected microwaves in a windowless room in a similar way: https://www.eetimes.com/document.asp?doc_id=1274746

titzer• 9 years ago

Also a really cool application of this rolling shutter is video of guitar strings:

https://www.youtube.com/watch?v=8YGQmV3NxMI

(Note that real guitar strings don't vibrate in this way; they vibrate across their entire length, with limited higher order harmonics; the visible waveforms here are an artifact of the CCD capture process).

thedrake• 9 years ago

This will help to solve debates like the one about the moon landing. Use this video (https://www.nasa.gov/multimedia/hd/apollo11_hdpage.html - the highest HD I could find) of the flag moving and extract sound. Then extract the sound (if there is any) http://people.csail.mit.edu/mrub/vidmag/ is the site that he mentions at the end or the ted talk given that goes into more about the workflow.

Not a programmer so not sure how to use the code to implement it. Any takers?

Sir_Cmpwn• 9 years ago

This is going to do nothing to solve the "debate". Moon landing deniers will find an excuse to dismiss this evidence like they have for the rest.

jwilk• 9 years ago

The conspirators purposefully made the flag from nylon, rather than candy wrappers, in order to foil future attempts of sound extraction.

333c• 9 years ago

Am I missing something? Wouldn't there be no sound to extract because the moon has no atmosphere?

Or maybe that's your point — if it were faked, there would be sound, otherwise not.

mikejmoffitt• 9 years ago

As the original footage was done on film, the time difference between the top and bottom of a single frame is zero, unlike a "rolling shutter" progressively scanned sensor. The framerate is also much lower than the 60fps phone camera used in the last example from the MIT video, which was only barely enough to get meaningful data out.

So unfortunately, we probably won't be using this to get much out of old traditionally archived footage - not because of resolution of the frame, but time resolution.

amelius2• 9 years ago

If smartphone accelerometers become more advanced, perhaps they can be used in the same way :)

delinka2• 9 years ago

I'm trying to work this one out in my head ...

You've left a smartphone lying around and you want to read accelerometer data to determine what was said in proximity. Just turn on the mic instead.

You've left a smartphone lying around to remotely read light bouncing off it ... accelerometer not required. Also, just turn the mic on instead.

amelius2• 9 years ago

Apps don't have mic access universally. It has to be granted by the user.

breakingcups• 9 years ago

I wonder how much archival footage we might be able to give a second look knowing what we know now.

0xdeadbeefbabe2• 9 years ago

Well fine, but who do you listen to? Is face to face interaction still a thing?

mcknz• 9 years ago

mmagin• 9 years ago

I thought this was as old as continuous-wave lasers, albeit the original idea was bouncing it off a window.

efader• 8 years ago

creepy

Candy Wrappers Are Listening (2014)

Related Stories