Control Linux based distros using hand gestures using OpenCV, GTK, Mediapipe

https://github.com/flying-pizza-69/GestureX

by politelemon • 2 years ago

79 19 2 years ago

donatj• 2 years ago

Starting in the early aughts and extending into the release of the Wiimote there were a handful of projects like this, largely for Mac. Never amounted to much more than a proof of concept most of the time but I always found the concept exciting. I had a lot of hope for Magic Leaps initial stand-alone hand tracking device before they pivoted to AR

NegativeK• 2 years ago

At the end of the day, for my computers, my favorite hand gestures all involve the keyboard.

bee_rider2• 2 years ago

Yeah I keep a webcam pointed at my keyboard too. That way the computer knows I’m subtly flipping it off when I undo in vim.

dosinga• 2 years ago

What? Not using LLMs or an expensive cloud based API?

brutal_chaos_• 2 years ago

Quite pleasant, eh? I hope we see more projects without them.

anon1566541• 2 years ago

This project utilizes the hand gesture recognition model provided by the Google MediaPipe project.

The rest is just some glue.

thedanbob• 2 years ago

Suggestion: middle finger mapped to "run last command with sudo".

ToDougie• 2 years ago

I would prefer looking up towards the ceiling combined with a deep breath, followed by muttering a four-letter word.

Lukas_Skywalker• 2 years ago

Are you by chance interested in a command named after the four-letter word, which, when invoked, automatically fixes typos in the last run command and reruns it: https://github.com/nvbn/thefuck

aidenn02• 2 years ago

For those who don't know, this works in many shells:

    sudo !!

bee_rider2• 2 years ago

In the screenshot they’ve mapped fist to “shutdown now” which seems to either indicate high confidence in the gesture recognition capabilities, or a good sense of humor about the whole thing.

Josh5• 2 years ago

Some people want to live life on the edge. Expert mode would be a gesture that nukes your installation

sigmonsays• 2 years ago

I love this, I can't tell if it's really true or not

I feel like a few gestures to lock or switch workspaces would be really ideal.

skeledrew2• 2 years ago

I worked on something very similar to this a couple years ago. My main issue with it why I didn't go beyond the PoC stage was that raising my hands up to the camera soon became a pain (I use a laptop), esp as I have it at eye level. It also had a hard time keeping up with my motions, so most of the time it was getting a blur.

mncharity• 2 years ago

I had a overhead-ish camera on a stick, that flopped up from the laptop screen back, to point straight down at the keyboard. Alternately, a phone clip-on fish-eye lens was low-res but ok, iirc. Kludgy little mirrors were iffyer, but there's a commercial one for iPad I didn't try.

Blur wise, perhaps it was 30 fps? I was mostly ok with 60 fps. Would have liked 4k resolution for markers and precision though.

More problematic was integrating high-latency "wait for the camera frame and then for mediapipe" with low-latency "fast keypress response". Drove me towards replayable event streams, and thus custom apps. But I was using it for custom XR-ish apps anyway.

Running high-res mediapipe etal on multiple cameras kept a thinkpad doorstop laptop warm even before starting XR. Keyboard feel got a bit toasted. But TFA doing gestures-only fixed-poses low-res single-camera is far lighter.

skeledrew2• 2 years ago

It's a pretty old system; had it over 10 years now, so I didn't expect too much re the blur. I'll likely revisit after I've upgraded to something more modern. Maybe by then I'll have the camera position figured as well.

Love to get into the XR space myself.

teo_zero• 2 years ago

What is, exactly, a "smile hand gesture"?

aidenn02• 2 years ago

By comparing the list to the icons, it would be what I would call a "Finger Gun[1]"

1: https://en.wikipedia.org/wiki/Finger_gun

yoavm• 2 years ago

similar but different, I've used MediaPipe to create Eyeroll, a WebExtension that lets you scroll up and down webpages by blinking green your eyes.

https://github.com/bjesus/eyeroll/

Control Linux based distros using hand gestures using OpenCV, GTK, Mediapipe

Related Stories