Handsfree.js – integrate face, hand, and/or pose tracking to front end projects

https://github.com/midiblocks/handsfree

by detaro • 5 years ago

187 35 5 years ago

github.com

twobitshifter• 5 years ago

What are some of the applications for this?

Gys• 5 years ago

https://github.com/midiblocks/handsfree#examples

danielskogly• 5 years ago

I found this demo of him playing "Into the Breach" with only Face Gestures very impressive! https://twitter.com/GoingHandsfree/status/140022171083874713...

umvi• 5 years ago

You could integrate this with adtech to help make advertisements more interactive and to analyze how the viewer is receiving the ad by their pose and facial expressions. Based on that you could have an algorithm serve only ads that improve facial expressions and poses to positive ones

Solocomplex• 5 years ago

Another excellent reason to disable JavaScript in your browser

jcdietrich• 5 years ago

Could play punch the monkey!

lolinder• 5 years ago

I hope we haven't yet gotten to the point where people will mindlessly click "Yes!" when a random website asks for webcam access.

notjustanymike• 5 years ago

WebXR comes to mind.

blensor3• 5 years ago

I can't speak about this specific project but pose tracking in the browser can actually be really useful to get it on mobile without the need for dedicated apps.

We are currently experimenting with using phones (or tablets or basically anything with a browser and a camera) as a tracking source for full body tracking on the Oculus Quest ([1]) and being able to just use a Tensorflow model in the browser has cut down the time of development on the mobile device side to almost nothing.

There are of course a whole lot of other issues one buys into with that approach (browser security policies, the whole thing not working in certain countries due to problems with loading the model) but that's manageable

[1] https://www.youtube.com/watch?v=zz9S5hgrWpM (relevant part starts at 0:29)

tppiotrowski• 5 years ago

I made a slouching detector a while back. It calibrated the distance of your face from your laptop and if you start leaning in it’ll give you a notification to sit back. It’s a bit dated but still works on Chrome and FF, I believe:

https://slouchcam.com

ranguna• 5 years ago

Wow this is amazing! I remember a few years ago in my thesis I was trying to use something similar in ROS to detect hand gestures and make a robot drive toward you, it's incredible to see this in the browser!

GoingHandsfree5• 5 years ago

Oh that would be so epic! I did something similar during my residency for Handsfree.js but for "puppeteering" a robot: https://giphy.com/gifs/1XE2rnMPk6BFu8VQRr

albybisy• 5 years ago

the library and the demos are not loading for me.

rgovostes• 5 years ago

On Safari for Mac, you must enable Develop → Experimental Features → WebGL 2.0.

gregsadetsky2• 5 years ago

I'll use this post / opportunity to ask the HN community

(I understand that this is a technically complicated, and potentially sensitive subject)

I've been approached by a few people who are doing what they can to lower the cost of prescription glasses. It's a mix of (licensed) ophthalmologists, people interested in offering community health services, etc.

What is the current state of the art for pupillary distance (PD) measurement based on face detection / pupil tracking using laptop webcams or mobile (front) cameras?

I imagine (obviously?) that there must be scholar research on the acceptable error margins for a PD measurement (depending on the type of vision condition, i.e. farsightedness, etc.)?

Would using something like Handsfree.js or https://github.com/esimov/pigo (which has pupil detection) be a good start, or would these be an ~order of magnitude off in terms of the necessary margins?

Thanks a lot.

notum• 5 years ago

It comes down to physics. Cameras and their lenses come in all shapes and sizes, most of webcams don't expose their geometry in a programmatically accessible way, even if they did, most of them don's have means of determining distance to the subject.

We can detect faces, eyes and pupils, but with no reference points we have no idea how big or far apart they are.

gregsadetsky2• 5 years ago

Thanks!

What about sites (such as Warby Parker) that make you hold up a credit card (which has a standard size) and use that to determine (via what I assume is a rule of three) the PD? [0]

I can see of course problems with the card not being “perpendicular” and other 3d distortions.

[0] https://www.warbyparker.com/pd/instructions

mncharity• 5 years ago

Fwiw, there's https://www.eyeque.com/pdcheck/ .

And Google's MediaPipe Iris used iris diameter for distance estimation.[1]

EyeQue is perhaps a cautionary tale. "Change the world" "few dollars of optics + phone = D+CYL" become $45 one-person-per-subscription "vision tracker".

[1] https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...

fny• 5 years ago

There are a lot of problems with measuring pd accurately with a photo. Ideally you need to ensure the pupils are aligned (i.e. the head can't be turned) which is really hard to tell from a photo (there's no depth).

The credit card trick somewhat works since you can guage depth given the size and positioning of the card, but the errors still can end up in the 2-3mm range fairly easily.

The best mechanism would be to create something like a Google cardboard to guarantee the head is properly positioned and then perform the measurements.

epmaybe• 5 years ago

This probably isn't very helpful, but I'd imagine underestimating the PD is going to lead to happier patients even if not all that accurate. We can accommodate with crossing our eyes, but the opposite is not usually true.

GoingHandsfree5• 5 years ago

Hello! Handsfree.js doesn't yet have pupillary distance, but I do plan to implement MediaPipe Iris very soon which does: https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...

joshuamcginnis• 5 years ago

If my memory serves me correctly, I once a used a digital try-on app from an online glasses retailer and they had me hold a common item with a known width (something like a credit card or drivers license) in front of the cam within a bounding box. It could guess the location of my pupils but allowed me to drag the pupil location markers to the exact centers. Calculating pupillary distance was then just a matter of using the distance in pixels between the pupils and converting them to mm using the ratio of pixels to mm of the card in the previous photo.

esens• 5 years ago

I have tried the underlying library, MediaPipe/TensorFlowJS, and it is only okay. Once you try to use it for visual augmentation, I found it doesn't track that well and it has problems with scale.

So, while it may look good with the skeletons, notice they do not have useful demos, mostly just proof of concept demos that do almost nothing.

blensor3• 5 years ago

Have you tried the newest MoveNet from google? That thing works really well. I only started experimenting with the Lighting model (not Thunder) for our VR game but even that is pretty robust.

If you want to see how stable it is, have a look at that: https://youtu.be/zz9S5hgrWpM?t=147

It's tracking me with the Oculus Quest on the head, so it does not even have facial features to go on

jonplackett• 5 years ago

Thanks for the link - Does MoveNet give you 3D positions / rotations for each joint or just a 2D position on the image?

GoingHandsfree5• 5 years ago

I'm very sorry you don't think Handsfree.js has useful demos. That kind of hurts because I put a great deal of effort into making them useful as starting points (I started all this while I was homeless to help someone). I work on Handsfree.js full time and my goal is really to create simple demos so that others like yourself can run with it!

Maybe you'll find some of these more useful?

- Face Pointer, and accessibility tool to help people who can't use a mouse/keyboard: https://handsfree.js.org/ref/plugin/facePointer.html

- Palm Pointers, scroll the page and even multiple scroll areas at once: https://handsfree.js.org/ref/plugin/palmPointers.html

- Gesture Mapper, map static poses in seconds: https://handsfree.dev/tools/gesture-mapper/

- Face Coding (not ready), to help people with disabilities code by snapping blocks together: https://twitter.com/GoingHandsfree/status/140056015231972966...

- The library itself doubles as a Chrome Extension, here are some examples of what you can do with it: https://handsfree.dev/sites/

I have dozens and dozens of more experiments. Most of them are pretty basic, but the point is to show you what's possible :)

GoingHandsfree5• 5 years ago

Also, here is a simple demo that has been helping some of my friends with disabilities. It uses the Face Pointer to play desktop games: https://twitter.com/GoingHandsfree/status/140022171083874713...

bjornstar2• 5 years ago

There's a nice demo of using handsfree.js to drive a car in a racing game -- https://twitter.com/GoingHandsfree/status/140032233301379891...

amelius• 5 years ago

His comment: The controls are whacky as hell and it took 5min of practice but it's just a tech demo

...

GoingHandsfree5• 5 years ago

haha yeah, the idea is to do one radically different use case every single day for 100 days straight so I don't always have time to make them smoother

I did do another whacky one today that is a lot easier to do. It's called "Flintstone Mode": https://twitter.com/GoingHandsfree/status/140136648492059033...

bjornstar2• 5 years ago

He just posted another demo where you kick your feet to accelerate a la The Flintstones -- https://twitter.com/GoingHandsfree/status/140136648492059033...

Pretty fun to watch!

FabianCarbonara• 5 years ago

Very cool project! This looks like so much fun to build. Thank you for the great documentation of your endeavour!

powerslacker• 5 years ago

Probably the coolest library I've seen in a while.

Handsfree.js – integrate face, hand, and/or pose tracking to front end projects

Related Stories