Face swapping with Python, dlib, and OpenCV

http://matthewearl.github.io/2015/07/28/switching-eds-with-python/

by bemmu • 11 years ago

296 94 11 years ago

matthewearl.github.io

JosephRedfern4• 11 years ago

How long does this take to run? Could it be done in real-time?

taejo2• 11 years ago

The result doesn't look like Ed Milliband, though...

thomasfoster96• 11 years ago

Well, it looks something between Milliband and Voldemort...

bsenftner16• 11 years ago

Nice work! But people should be aware that this type of thing is patented left right and sideways. I wrote a patent for automated actor replacement in filmed media in '05, filed it globally, and it began being awarded in '08 in the various regions it was filed. Here's one of the US versions: http://www.google.com/patents/WO2007035558A2?cl=en I no longer own it; now it is part of a broad family of patents collectively forming a "Personalized Media" infrastructure. It is owned by a major media company I'm not allowed to disclose.

Essentially, it is possible to replace people, objects, environments, & audio elements automatically through a series of techniques. Those techniques form the basis of a powerful advertising and message delivery infrastructure. And the collective IP is already owned by a major media conglomerate.

JosephRedfern4• 11 years ago

Bleurgh.

bsenftner16• 11 years ago

Would you rather people spend time developing in this area only to learn they can't use their work?

fnordfnordfnord5• 11 years ago

It is owned by a major media company I'm not allowed to disclose.

Why would a patent owner want to remain unknown?

unavoidable2• 11 years ago

Come on guys. Like all patents, the owner must be disclosed to the USPTO. A quick Google search turns up Flixor as the original owner, and "Big Tent Entertainment, LLC" as the current owner of some of the recent patents in this family. A further search of their website indicates it licenses various media technology to companies including "Concept One, Nintendo, Fifth Sun, Parragon Publishing, Harry Abrams, Kids Station, Merch Source and more".

srean2• 11 years ago

For a patent troll it makes perfect sense. Rig the line and wait for a bite.

rasz_pl2• 11 years ago

trolling

spdustin• 11 years ago

It's a patent and you're not allowed to disclose its owner? Is that a thing? Secret patents?

bsenftner16• 11 years ago

Those were the terms of the sale. I was not in a position to argue.

tdicola• 11 years ago

Even if it's patented you can still play around with the technology yourself. Just don't sell it or make a product out of it.

the-dude• 11 years ago

I remember that even 'having' the technology is in breach of the patent if there is no license.

Could be wrong though.

witty_username• 11 years ago

As far as I can tell, this is just a patent troll's patent. It seems to be a very generic process for replacing people, objects, environments, etc.

richmarr2• 11 years ago

Just to echo this a bit, I spent 3 years working on image similarity tech and a lot of our research was challenging, not because of the intrinsic complexity of the problem, but because we had to implement it in such a way as to step around existing patents.

wjnc2• 11 years ago

I guess you found the math combined with the search for alternative routes interesting? But did you also try to find prior art? My first thought would be that for unreasonably broad patents prior art is the easiest way to be able to use the tech?

srean2• 11 years ago

Company I am not allowed to disclose.

Aren't patents supposed to be public information ? One of the (rather hollow in practice) justification for patents is to (also) encourage eventual disclosure of knowledge.

bsenftner16• 11 years ago

The patents are, but their ownership does not have to be.

dragonwriter• 11 years ago

Aren't patents supposed to be public information ?

Private agreements can restrict a person's comment on things without regard to whether the information is already public.

wjnc2• 11 years ago

What's your view on the patent? Pretty broad? Or a sign of very quick progress that in 10 yrs something innovative (hence, a patent) turns into a few open source libraries and few lines of code to replicate. Most responses (including my thought) call 'patent troll' etc, but perhaps there was substantial thought and effort in the techniques below the patent.

bsenftner16• 11 years ago

The majority of these "Personalized Media" patents rely on existing integrated media - meaning imagery and audio composed of multiple elements - and the ability to isolate one element, remove or modify it, and/or replace it with something else in a manner that the original media's intent and message is retained, only now it is enhanced for a specific audience.

Think product placements that change for every single viewer.

Real time rendering, VR/AR and that entire enchilada of media techniques completely sidestep these patents because there is no original media, it's all created and composited on the fly, dynamically for the single specific consumer of the media.

shadowmint• 11 years ago

This is literally what's wrong with the patent system.

Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.

I bet there are a million patents out there for 'automated face replacement'. I just hope very very much, that somewhere there is some prior art that makes them all completely worthless and bankrupts whomever is making any money from them.

JonFish85• 11 years ago

> Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.

Who's discouraging? Being aware that it's highly patented doesn't mean you shouldn't play around with it, it just means that you probably shouldn't be trying to make money off of someone else's work.

bsenftner16• 11 years ago

The automated face replacement patents begin in the late 70's. By '05 the process had to be entire in-motion actor replacement to be patentable. To some degree, I'd say patents force people to work in uncharted areas.

rasz_pl2• 11 years ago

face is just a particular pattern, pattern matching and replacing was all done since the sixties.

god_bless_texas• 11 years ago

What you said was a little strong, but I do fundamentally agree with you that patent clerks do not perhaps have the benefit of 5-year technology trajectory to make informed decisions.

When you look at the patents that Newegg fights (and wins), clearly something has gone wrong at USPTO.

smcl• 11 years ago

I'm curious, did you put together an algorithm and\or program which actually did this automated actor replacement or did you just patent the idea? No judgement either way, "don't hate the player..." and all that :)

edit: looking at your bio it seems pretty likely that you had properly implemented it!

bsenftner16• 11 years ago

Fully implemented, but expensive to use. It's modern visual effects technology, automated. Also, not a single algorithm or program, but a media pipeline of over a dozen custom programs, plus integration with the major VFX software packages.

sabalaba• 11 years ago

It's not clear that the linked patent applies to the technique in the original post. https://www.google.com/patents/US7787664 and https://www.google.com/patents/US8712189 both seem more relevant.

Just because something looks like it could be patent encumbered doesn't mean you shouldn't pursue it. Patents need to be fought over in court. It costs millions of dollars to bring a patent lawsuit. In the event that your little side project or startup becomes a target for a patent lawsuit, frivolous or otherwise, you've likely already won.

Even with a semi-broken patent system, the world simply favors active, high-quality execution over passive "intellectual property" generation. Focus 364 days a year on a building a great product that people love and then spend a day to talk to your attorney about patents, IP, and how to protect the valuable company that you've built by focusing on your customers.

Jeff Schox has a great talk on patents and startups that is very relevant to this discussion: https://www.youtube.com/watch?v=hyv75d4JG7c

bsenftner16• 11 years ago

Be aware that a single technique can infringe on multiple patents, and a patent can extend other patent(s), triggering a situation where a single technique requirements licensing from multiple entities.

It is exactly the issue of fighting patents costing millions of dollars that makes technology development financially risky. When accused of infringing, one has to respond, and that response costs both significant time and significant money.

I agree a side project need not worry. A startup does. I also agree that the world favors active execution over passive intellectual property generation. Our world also forces the active executing organization to protect itself if they need any financing to realize their goals.

nkw• 11 years ago

... and people should also be aware those patents might be invalidated by re-examination or an action in federal court. I don't know why you aren't allowed to disclose who now 'owns' the patent you refer to. It is trivial to locate this information with the link you provide and some googling.

Also your statement "the collective IP is already owned by a major media conglomerate" if taken to refer to totality of technology which makes it "possible to replace people, objects, environments, & audio elements automatically through a series of techniques" is false.

bsenftner16• 11 years ago

I beg to differ, there most definitely is a series of patented technologies that in total enable replacement of people, objects, environments and audio elements in traditional media. That key is traditional, linear media. New forms of media, interactive forms, VR/AR and hybrids are wide open fields. That's a big reason new fields of technology are so exiting - it's not just interesting to us geeks, but valuable to the businesspeople.

pyre2• 11 years ago

It is owned by a major media company I'm not allowed to disclose.

I don't understand this. If the patents are public, why shouldn't the owners be as well?

fizixer• 11 years ago

Noob question: What if the developer who "might" have infringed on the patent, open sources his work, and it's made available on github with multiple forks, and now there's a whole community behind it, and so on?

zeckalpha• 11 years ago

Certain specific processes may be patented, but "this type of thing" cannot be patented outright. Ideas are not patentable, processes are.

mahouse3• 11 years ago

And it had to be fucking Miliband?!

Well that said: https://www.reddit.com/r/gentlemangabers

(Before you go dig up the actual source code: it just uses a web service to get the coordinates of the faces)

fnordfnordfnord5• 11 years ago

And it had to be fucking Miliband?!

Did you see what happened here the last time someone used Lena in their example?

mahouse3• 11 years ago

No, I did not.

By the way, I was shocked the first time I saw what the full picture looked like... 8-)

spot• 11 years ago

https://vimeo.com/29348533

piquadrat• 11 years ago

Wow! That is creepy and impressive at the same time.

tim333• 11 years ago

See also https://www.youtube.com/watch?v=i2VvvOWWquo using dlib and opencv.

henok• 11 years ago

Oh wow. This will make a great nightmare.

jdiez17• 11 years ago

The amount of mathematics that goes into automating faceswaps is awesome. The fact that anyone can do this at home with the sole purpose of generating a funny image is great!

saysfaceoff• 11 years ago

FACE/OFF: http://www.imdb.com/title/tt0119094/

murbard2• 11 years ago

Matching the low frequency image information is a good simple approach to do the blending, but better results can be achieved with Poisson image editing: http://www.cs.jhu.edu/~misha/Fall07/Papers/Perez03.pdf

jabooth• 11 years ago

This is exactly the kind of thing that Menpo makes easy: https://gist.github.com/jabooth/a52019f6e7c09f2a3e01

(Forgive the rushed nature of this, literally just whipped this up in the last 20 mins. Will provide a link to the menpofit model in a later revision)

aswanson• 11 years ago

This is awesome. How long did it take to develop?

ChuckMcM• 11 years ago

Isn't this like totally giving away the Enquirer's[1] secrets? I enjoyed the discussion too. Next up, deriving a set of makeup (cosmetics) steps that need to happen to have your face recognized as some other persons face.

[1] a gossip newspaper which often (always?) has dubious photographic evidence backing up their claims.

bmir-alum-007• 11 years ago

OpenCV is an awesome Swiss-army steam-shovel (thank you Willow Garage!)

In the 90's, there was Kai's Power Goo (now owned by Corel) and probably something similar used by Conan O'Brien's art dept. to produce the "If They Mated" segments. It's such a shame tools owned by big companies are effectively locked away from ever being marketed again by asserting IP rights.

I really think intellectual property needs a principle similar to adverse possession [0]; that is, if it's not being actively used in a meaningful, sustained venture, another shop can reboot it. (Giving capitalism the right to fork.) Btw, I'm not the first, nor the last, to conceive of this. [1]

0. https://en.wikipedia.org/wiki/Adverse_possession

1. http://jolt.law.harvard.edu/articles/pdf/v16/16HarvJLTech327...

eigenbom• 11 years ago

Neil deGrasse Tyson with Peter Serafinowicz face: https://youtu.be/gr4gIK7Bv3E

Had fun over the weekend playing with this system. :)

cstudyhelp• 11 years ago

This was a great and interesting article to read. I have really enjoyed all of this very cool information. <a href="http://www.thecasesolutions.com/hozho-a-16335">Hozho Case Solution</a>

Face swapping with Python, dlib, and OpenCV

Related Stories