HN user
I'm working on a web app that animates hand drawn human characters. You try it here without any login or anything:
It doesn't use generative AI, instead it auto-rigs the drawings in just a few seconds.
Thank you! It’s a web app, but we designed it to be mobile first. You can absolutely visit the website, take a picture, and show the animation to your daughter all on your phone.
If you do give it a try, please let me know how it goes!
That's the plan! Glad to hear DoodleMate brought a smile to your face.
That's a good question. I would say that the two approaches are quite different and bring different strengths to the table. The major strength of genAI is that it is open-ended.
But it comes with costs. GenAI video is expensive to generate, and most tools constrain your animation to a handful of seconds, not long enough to tell a real story. You can generate multiple clips and stitch them together, but then you'll run up again another limit of GenAI- subject consistency (especially with non-realistic subjects, like doodles).
It's also difficult to finely control genAI outputs, which I argue limits the creative expressivity of the human. And if you generate numerous clips to try to get things perfect, it can get expensive.
Our approach is limited by the motion/visual/audio assets we have access to. But, when we release DoodleMateStudio users will be able to upload their own visuals, record their own audio, capture their own motions, and specify their own high-level story scenes. This should be enough to let people tell expressive and personal stories. And if we get things right, it will also be a lot more fun than refining a prompt.
I hope she likes!
Please let me know how it goes and if there's any functionality she/you would want.
That's a good question. The distinction is between generative AI (which takes in a prompt or image and generates every pixel of a new video) and non-generative AI models (e.g. classifiers, segmentation models, and pose estimation models). The second category helps us to infer characteristics about the input drawing, but it doesn't try to 'recreate' anything.
We use non-generative AI models to quickly auto-rig the character when it's uploaded. In a traditional computer graphics animation pipeline this would be done by hand and would be a slow process; we use these models to speed that step up. The resulting animations don't use any AI at all(generative or otherwise).
I'm a student of good animation myself (know the 12 principles of animation by heart, got a copy of The Illusion of Life right next to my desk, etc.) and have great respect for what professional animators can do.
And while I (maybe more than anyone else) am highly aware of all the ways this tool doesn't create high-quality animation, I'm okay with it. The point here isn't to create animations that rival those of professional animators; it's to lower the barrier-to-entry to the point where almost anyone can animate their own character.
And if a tiny number of users become more interested in animation through this tool, learn how to do it the proper way, and create animations your father would approve of... well I think that would be a phenomenal outcome.
Yeah, I think that'll happen. I think it's already happening. And if someone wraps it in a terrible app with ads, makes millions of dollars, and I never see any of the money, yes I'll be very sad. But I'm trying to take a bigger perspective here. There's a lot of experience and tools that could be built off this, and I can't possibly build them all.
And if I hid the code away so only I could use it, there's a good chance no one would ever create any of those experiences. Creating a successful app requires mobile UI people, devops, designers, marketing/SEO, monetization, etc. I don't want to do ANY of that... in the land of app creation, I'm more of a Tom Bombadil-type.
This is also my first project out of graduate school. I feel like I'll probably have other ideas down the road, and maybe I'll try to monetize those ones with what I've learned from this adventure.
I’m curious to learn more.
Sounds awesome! Let’s do it.
Love the idea. Feel free to reach out if you think there’s room for collaboration.
You're the first stranger to ever compliment my coding style. I feel all warm and fuzzy now.
Yes. I did most of this as a postdoc at FAIR, Meta's AI research group.
It’s not metadata, you would see that when you upload the photo at the beginning.
I’m guessing that the nose key point is located below the shoulders in your child’s drawing?
That’s a known issue caused by how we ‘apply’ poses onto the characters. It’s fixed in the GitHub code but not in the browser version. Try moving the nose keypoint to the top of the characters head and that should resolve it
If you post your creation to social media, please use #FAIRAnimatedDrawings. I'd love to see what you create!
As written now, it expects 3D motion data. Extending it to support 2D motion data is doable, but will take time. Feel free to open a github issue requesting it if you'd like me to prioritize it.
It currently only outputs .mp4 and .gif files. You could probably use these as game assets, but I don't think it would be optimal.
The rendering code utilizes an MVC framework. If you need the animation output in a .obj format or something, it's as simple as creating a new View that supports it. Feel free to request this in a Github issue if you want me to prioritize it.
You might check out the technical paper, the background section lists some relevant related works on non-photorealistic pose estimation.
One work that seems particularly pertinent to your use case is:
Gesture3D: Posing 3D Characters via Gesture Drawings http://www.cs.ubc.ca/labs/imager/tr/2016/Gesture3D/
Glad to hear it was easy for you to get up and running, but where did come across the repo? It's been in 'stealth release' mode for a few months now, but nothing was publicly announced until today.
It's equally important to stretch before coding and stretch before dancing.
I don't know who would speak out against it. The MIT license certainly allows for it, and I don't actually own it... I just wrote it :)
Glad you got it working.
And glad you appreciate my sick dance moves :)
I assume you're referring to the demo at sketch.metademolab.com? The server is returning an .mp4 file that is displayed by your browser, so you should be able to download it. On Chrome, if I maximize the video and click the three vertical dots, I get the option to download the video.
Hey! That’s my project!
Code and dataset are here: https://github.com/facebookresearch/AnimatedDrawings
And a browser-based version of it is here: http://sketch.metademolab.com/
If it makes you feel better, there's no data being collected other than the images and annotation fixes (and that only occurs if you click 'agree' on the consent page).
You also don't need to be signed in Facebook or anything to use the demo- there's no way to link the uploaded images with personal information about the artist/uploader.
That's a good point. FB/Meta did a blog post talking about the tech. There are some failure examples there:
https://ai.facebook.com/blog/using-ai-to-bring-childrens-dra...