Guessing the pressed keyboard keys by analyzing the audio from the microphone

https://github.com/ggerganov/kbd-audio

by Osiris30 • 8 years ago

194 33 8 years ago

github.com

daenz• 8 years ago

Now determine the keypresses by filming the vibrations of a potato chip bag[0]

[0]http://news.mit.edu/2014/algorithm-recovers-speech-from-vibr...

hashmap• 8 years ago

They manage to do it (just sound, not keypresses) with a 60fps DSLR camera at the end by examining the rows of the video - does this mean that for any videos already in existence, sound can be decoded from the images?

singularity2001• 8 years ago

For any old 60fps homemovie of your potato chips: maybe.

aaaaaaaaaab2• 8 years ago

Not for handheld videos.

rapnie• 8 years ago

Primary use case is more tracking and surveillance, I presume?

IIAOPSW• 8 years ago

Primary usecase is hacking. A keylogger without any software or trace on the target computer.

spockz• 8 years ago

This is another reason why you want 2FA.

_Nat_• 8 years ago

For the technology itself, sure.

For this particular open-source hobbyist project, not likely; those who're most likely to attempt to use such a tactic maliciously are likely to keep it to themselves rather than demonstrate that the security concern exists.

scarlac• 8 years ago

Attempted to train it by typing for ~2 minutes. Basically typed everything on the GitHub page and tried getting predictions. Results were disappointing. I didn't see any accurate predictions. Even with the default p/q program I see mostly random results.

I tested on a 15" Macbook Pro 2018 (latest version of keyboard that is softer to type of / less noisy)

ggerganov3• 8 years ago

The p/q test fails on my Macbook Pro 2013 as well. Plugging an external mechanical keyboard makes it work, so it's definitely due to the soft type of keyboard and not the mic.

carbocation• 8 years ago

Sounds like the author has the same experience: if the keyboard is not mechanical, then this library doesn’t work:

https://github.com/ggerganov/kbd-audio/issues/3

ggerganov3• 8 years ago

Correct. I just made a short video to demonstrate what a working setup looks/sounds like:

https://www.youtube.com/watch?v=2OjzI9m7W10

kalmi10• 8 years ago

Macbooks usually have multiple mics for noise filtering. That could also be an issue.

amelius• 8 years ago

Still, I suspect that certain words, when quickly typed, have specific sounds.

cbg0• 8 years ago

I've tested this on my Filco Majestouch 2 with MX Browns but it barely managed to detect anything, and even then it didn't do very well. I think this is also highly dependent on the microphone used (I used the one in my Logitech webcam).

0db532a0• 8 years ago

Could you maybe do away with training beforehand on a single source and instead use multiple, very sensitive microphones and triangulate the locations of the keys being pressed? The estimations might not be accurate, but you could put the results through a smartphone typo correction algorithm.

switz• 8 years ago

I've always thought Twitch streamers were opening up an attack vector through this exact method.

Cool to see someone follow through with it. Any streamers out there should figure out a way of avoiding keypress bleed or muting their mics when typing sensitive info (e.g. passwords).

nmstoker• 8 years ago

Yes, countermeasures, such as playing other sounds around those frequencies and/or filtering the microphone at those frequencies would be interesting to explore. Filtering the mic seems the less annoying from a user perspective!

drankula3• 8 years ago

Most streamers, myself included, use a filter called a noise gate, which requires the volume of the mic reach a certain threshold before being broadcast. This filters out the majority of background noise on the stream.

TazeTSchnitzel• 8 years ago

Don’t Skype & Type! Acoustic Eavesdropping in Voice-Over-IP

https://www.math.unipd.it/~dlain/papers/2017-skype.pdf

mad_hominem• 8 years ago

Perhaps accidentally, a yubikey or something similar would avoid this attack.

pbhjpbhj2• 8 years ago

If someone hunts-and-pecks how close can you get with gaze tracking?

syntaxing• 8 years ago

This is super cool. Very similar to timing attacks. I wonder if there is a way to tune the model. Essentially cater the probability of each key to a person with maybe a sentence or something.

sdenton4• 8 years ago

Yeah, this is exactly the sort of thing where augmenting the base model with a language model will have some good returns...

ggerganov3• 8 years ago

In keytap2 I'm trying to make use of the statistical distribution of n-grams in the language. The idea is to first group the unknown keys into clusters based on how similar they sound. The prediction then is performed by breaking the obtained substitution cypher (assuming each cluster corresponds to a letter).

sentrysapper• 8 years ago

Can you combine this with or does it use relative positioning of the microphone? Seems like a good way to map where keys are (measure lower decibels which would be keys further from the mic).

danschumann• 8 years ago

Someone has seen the movie sneakers! The blind guy did this in that movie.

hnmonkey• 8 years ago

In what scene? I don't think he actually did... I'm pretty sure when they're trying to crack the guy's password they're using video to record and zoom in to watch him and he's blocking it.

frakr• 8 years ago

definitely keep buying mechanical keyboards....

ppod• 8 years ago

There is more research on this than what you have cited here, unfortunately the literature goes under the catchy keyword "acoustic keyboard emanations", try that in Google scholar.

Guessing the pressed keyboard keys by analyzing the audio from the microphone

Related Stories