Reflected hidden faces in photographs revealed in pupil

http://www.kurzweilai.net/reflected-hidden-faces-in-photographs-revealed-in-pupil

by ca98am79 • 13 years ago

138 112 13 years ago

kurzweilai.net

japhyr• 13 years ago

Wow! I had no idea this could be done. The implications of this technique for forensics work are intriguing.

Is this a new idea, or have people been working on this for a while?

walid• 13 years ago

Well it is in many crime movies except that our megapixels were not enough for such uses. Now with high resolution images it is relatively easier to do. I'm not surprised by it. It just is another use of tech.

jkubicek• 13 years ago

The idea of examining a subject's pupil has been around for years. I've used a similar technique to reverse engineer lighting setups for portraits. See my friend Mike's portrait of one of his children:

http://www.flickr.com/photos/oatmeal2000/8545400550/sizes/o/

You can clearly see that he uses one umbrella to the right of the camera. With higher resolution images and better focus you can sometimes determine the brand of lighting they use, but that's only if it has a very distinctive shape.

GrantS• 13 years ago

This 2004 paper got a lot of attention in the computer vision community doing essentially the same thing about a decade ago:

http://www.cs.columbia.edu/CAVE/projects/world_eye/

"The World in an Eye," K. Nishino and S.K. Nayar, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.I, pp.444-451, Jun, 2004.

apu• 13 years ago

There was a related paper on using the eyes as a means of capturing a full environment map which could be used directly for relighting a scene (e.g., for 3d rendered scenes or for compositing digital objects into scenes):

http://www.cs.columbia.edu/CAVE/projects/eyes_relight/

edent• 13 years ago

The cameras used have 39 megapixels. I wonder how well it works with Nokia's PureView cameras - or the lower quality lenses on iPhones & Androids?

jbrooksuk3• 13 years ago

I always thought that CSI was having a laugh with scenes like this: http://www.youtube.com/watch?v=53ilsuJOTG8

renekooi• 13 years ago

Well, there is a tiny bit of a difference in image detail between CCTV and the 39 MP camera used here.

So CSI has no excuses still.

krapp2• 13 years ago

Maybe the whole CSI franchise is secretly near-future science fiction?

thejosh• 13 years ago

Heh, or https://www.youtube.com/watch?v=sp77AjBdlEc.

jbrooksuk3• 13 years ago

Haha! "Uncrop" amazing! I should really watch Red Dwarf.

kristofferR• 13 years ago

This is probably the best/most relevant CSI example: http://www.youtube.com/watch?v=3uoM5kfZIQ0

jeffbr13• 13 years ago

Zoom... Enhance!

DanBC• 13 years ago

The relevant mocking image - "number plate enhance" http://imgur.com/uhvWKS8

InclinedPlane3• 13 years ago

In maybe 20 years or so this sort of shit is going to be scary.

Imagine what happens when we have several orders of magnitude more computing power available. A lot of things aren't possible today because they are too computationally intensive, but computing power is just getting cheaper and more abundant. Imagine what happens when it's feasible to process basically all data on the entire internet. Every photo and video. Every tweet, vine, and wall post. Every porn video and macro image.

It'll be possible to find the identity of anyone in any picture. Moreover, every picture or video that someone has been in will be easy to correlate. Now scale that up to every activity. Anyone with enough computing power will basically be able to create a dossier on anyone, or on everyone.

The implications are concerning.

6d0debc0712• 13 years ago

There are hard limits in terms of energy/thermal efficiency to computation. I don't think you will be able to process all the information on the internet - even assuming that people use ridiculously high resolution images regularly.

InclinedPlane3• 13 years ago

Consider that all of the video on the internet has been processed multiple times already, in order to encode it and decode it upon viewing. The hardware to do so is incredibly distributed though, but nevertheless that hardware has already been manufactured and put to use. The current youtube infrastructure already processes encoding/uploading video at more than 6,000x real-time. With significantly more powerful computers it will be possible to use a similar investment in hardware to not just encode but also analyze and correlate video at thousands of times real-time. What becomes possible 10, 20, or 50 years into the future?

If you can analyze video at, say, 10,000x real-time you can crunch a billion hours of video in a decade. Imagine the value of the meta-data you could get from such an analysis. You would have incredibly fine-grained details on people's lives, especially as video use becomes more common. Being able to see hidden details in reflections is immaterial compared to what will be possible. With enough computational resources it'll be possible to create dossiers on everyone everywhere. You'd know where they live, who their friends are, what they do for fun, what they own, what they wear, what they read. Using writing analysis you could find out how politically active they are, what their opinions are, and so on. Given that abundant individual liberty is still a fairly rare, and constantly endangered, condition on Earth it's frightening to imagine tools that could give such information in the hands of unscrupulous folks.

Gravityloss4• 13 years ago

Why 20 years?

I'd assume that most people have been facetagged in enough photos on Facebook so that they can be found in every photo by every random person who uploaded there, or to a few other cloud services, which any reasonably determined organization can scrape. People photograph a lot.

And then the CC cameras as well.

Though there are easier ways to spy on people, you can get all relevant info from their phone apps and social media services anyway.

InclinedPlane3• 13 years ago

20 years is just a SWAG for when computational resources will be so abundant that the scariest level of analysis will be routine. Today it's simply not practical to determine automatically everyone appearing in a given random photo or video. Facebook vastly simplifies the problem (by narrowing the search scope tremendously), but there's a lot of content elsewhere.

For example, consider identifying people by the way they walk, the clothes they tend to wear, their body build, etc. That sort of thing makes witness protection efforts practically useless, for example. The sheer amount and detail of information that could become available will enable things that we can't even remotely imagine today, but ultimately the biggest change will be loss of anonymity at every level and loss of privacy.

jere5• 13 years ago

In addition, “in accordance with Hendy’s Law (a derivative of Moore’s Law), pixel count per dollar for digital cameras has been doubling approximately every twelve months. This trajectory implies that mobile phones could soon carry >39 megapixel cameras routinely.”

Ah Moore's Law. It explains everything. Double every twelve months, eh? Hmm, quick math. Facebook has been around for 9 years... oh yea, that explains why Facebook allows you to upload full resolution 60MP photos. /s

What do your Instagram and Facebook photos reveal?

Nothing. They're about 100 times too small for this technique. And they probably always will be.

jamesaguilar4• 13 years ago

And they probably always will be.

Famous last words when predicting non-advance of technology.

jere5• 13 years ago

Yup, you're right. I'm lacking imagination.

Please help me understand a future where social networking sites will maintain and display ~ 6200px X 6200px photos of your selfies. Outside of genetically engineering new eyeballs, I'm struggling here.

redwood• 13 years ago

One thing I've always thought lost in the discussion of increased digicam pixel count is that at a point you gain no real useful information in those pixels, because you've crossed the threshold of whether vibration allows resolution at that level at all.

Sure more pixels usually mean slightly more information, and there's obviously techniques: stability-enhancement, faster shots etc, that change this critical point. But for the average person carrying a camera: we crossed the critical point long ago and now just waste a lot of SSD space.

hahainternet7• 13 years ago

The vibration is not as significant an issue as diffraction is. Modern 18mp APS-C sensors already hit their diffraction limits at around f/11. Increased sensor density will push this down to 'reasonable' f-stops, f/8 being the real cut-off point.

rtpg• 13 years ago

well with a bit more exposure and a motion sensor can't you get rid of the vibration effect to some extent? I have a camera with a 24x zoom that can take relatively clear pics despite my hands shaking like crazy (when enhanced 24x at least)

tanzam752• 13 years ago

Ah Moore's Law. It explains everything.

He never said that Moore's Law explains everything. He simply said that Hendy's Law explains why phones will soon has 39-megapixel cameras. Since the Nokia Lumia 1020 is available today with a 41-megapixel sensor, I do not find this to be farfetched.

Facebook and Instagram will depend not on Moore's Law, but on availability of bandwidth and cost of storage. To argue against Moore's Law for Facebook is to argue against a strawman. The article did not make that connection -- you did.

In 2005, would you have predicted that Youtube would never carry 1080p videos? I mean, back then, Blu-ray and HD-DVD were still battling it out for dominance of the next-generation disc market.

jere5• 13 years ago

He never said that Moore's Law explains everything.

I was referring to the fact this is on kurzweilai.net and Kurzweil tends to think it does explain everything: how we're going to beat the turing test in 16 years, how he's going to resurrect his dead father, etc.

From the subtitle to the last paragraph, the article clearly implies that this technique will soon be applied to social networking sites. And it seems to use Moore's law to handwave the details.

jere5• 13 years ago

Another thought:

A few people are pointing out use cases where a 39MP photo might make sense down the line:

-Group photos

-Head to toe shots

-Panoramas

Let me just point out that this research is done on "high-resolution passport-style photographs." That is a close up head shot with a normal FOV. You fit your whole body, your friends, and everything else in the photo... you're going to need hundreds to thousands of mega pixels for this to work. At the end of the day, it looks like you still need > 39MP just for the face.

aestra• 13 years ago

It isn't like there aren't photo sharing sites where you can upload the full res version, like Flickr. Flickr offers a free TB of space.

krapp2• 13 years ago

And now i'm wondering whether you could create an app which detected faces reflected in images and scrambled them... though with eyes that might destroy the way the original face looks in some creepy way. Of course you'd have to strip all the EXIF data as well including any internal thumbnails which would be of terrible quality anyway...

pg• 13 years ago

http://www.youtube.com/watch?v=qHepKd38pr0

pvnick• 13 years ago

Combine with content aware fill and you get http://www.youtube.com/watch?v=6i3NWKbBaaU

RokStdy• 13 years ago

Am I the only one who thought, "so what?", after reading this? A picture of sufficient detail taken under optimal conditions can yield reflected images off of the pupil. Big deal. It has always been the case that in a high resolution image you had more of a chance to see reflected images.

Not that this isn't interesting, but their hypothetical (hostage situation) would have been a much more intriguing test case. But, since those pictures/videos are rarely taken with super high resolution cameras with good lighting, it's unlikely this technique would be helpful.

Edit: Just to be clear, I think anything new in the form of research and development is generally good. Maybe this work will be built upon to create something great. I just don't like the implied promise that pieces like this make. It oversells what technology can actually do and is, I think, misleading.

a_olt• 13 years ago

This work is not entirely new. Here is another recent article, where it is explained how input to a smartphone/tablet can be reconstructed by analysing reflections of the screen in various nearby surfaces, or the user's pupil.

http://dl.acm.org/citation.cfm?id=2516709

devindotcom• 13 years ago

How cool. Careful, though, Neal Stephenson is probably lurking here collecting items like this for his next book.

ojbyrne• 13 years ago

As someone who has taken a lot of pictures, the key to photography is not pixels, but light. And so I had to read the paper:

"The room was flash illuminated by two Bowens DX1000 lamps with dish reflectors, positioned side by side approximately 80 cm behind the camera, and directed upwards to exclude catch light"

Yes, that's the kind of lighting conditions that are routinely present at crime scenes.

vacri• 13 years ago

Then there are what seems to be the standard-issue security cameras, where it's hard to even tell what the main subject is wearing.

im3w1l• 13 years ago

EXIF-data is a much more realistic threat.

EDIT: Oh and camera-charecterising noise signatures.

aeturnum• 13 years ago

Do you have any resources on fingerprinting cameras based on noise? Seems hard to do outside a controlled environment (as noise is so heavily influenced by environmental factors).

shmageggy• 13 years ago

That picture of Obama is 18x25, not 16x20.

ecocentrik• 13 years ago

and the reflection is off the cornea not the pupil but who's paying attention to details...

jbattle2• 13 years ago

I expect big things in the coming decades coming out of computer vision & image processing. I wouldn't be surprised if algorithms to 'zoom & enhance' images reach the levels we laugh about in movies today.

I'm not well read on the topic so I might be talking out of my ear, but this is the sort of thing I'm anticipating getting more and more powerful over time: http://users.soe.ucsc.edu/~milanfar/talks/milanfar.pdf (jump to page 23 if you want to see demos of the algorithm)

http://www.scriptol.com/programming/graphic-algorithms.php

edit: grammar

kayoone• 13 years ago

I wouldn't be surprised if algorithms to 'zoom & enhance' images don't reach the levels we laugh about in movies today.

wouldn't or would ? don't or do ?

kaoD• 13 years ago

Noise and lack of resolution are different beasts. That paper has nothing to do with "zoom & enhance", which will never reach the level we see in movies, since the information is just not there.

jbattle2• 13 years ago

Yet both noise-reduction and resolution-enhancement are both fundamentally about taking whatever information you do have and using that to interpolate/infer information that has been lost (or was never captured).

Especially in certain domains such as written letters (e.g. license plates) or faces that have a well-known structure with variations in known dimensions, I wouldn't be surprised to see algorithms that can probabilistically infer what information would have originally been there. Of course you'd never reconstruct an entire face from a single pixel, but how far can we go? I dunno - but my hunch is pretty far

Houshalter2• 13 years ago

That's not entirely true. Images are highly redundant and have a lot of structure to them. it might be possible to reconstruct the image a decent amount from that. Obviously a lot of the information and all small details would be missing, but you could make it more recognizable.

pasbesoin• 13 years ago

Shades of "Blade Runner".

And, I'm now waiting for someone to release a "Privacy Eye" Photoshop filter (a la the ubiquitous "red eye" filter).

oakwhiz• 13 years ago

This works on things like Christmas ornaments as well...

dhughes• 13 years ago

Although Jenkins did the study with a high-resolution (39 megapixels) Hasselblad camera,...

Medium format cameras have a much greater area sensor and probably overall better quality ($2,000 vs $20,000 or even $50,000 medium format camera) compared to a regular digital DSLR camera. Even a quick Google seemed to indicate photographers prefer the larger sensor size even if it has fewer pixels.

The article mentions "face images retrieved from eye reflections need not be of high quality in order to be identifiable" but even so there must be a massive difference in picture quality compared to a Hasselblad medium format camera sensor.

Still interesting though it's as if the goofy CSI or Blade Runner "Enhance!" has come true.

PavlovsCat• 13 years ago

It would be interesting to see what hidden information is buried in law-enforcement (and other) photo archives — some of which could even help exculpate innocent persons.

Yeah, it would be interesting. But would you hold your breath for it? In an age where kids "shoot themselves" while being handcuffed and whatnot, I for one wouldn't.

evanb• 13 years ago

This reminds me of dual photography [1], in which you trade many projectors and one camera for one projector and many cameras using time reversal symmetry.

The video [2] gives a good explanation, and the revelation of the playing card is what strikes me as similar to revealing reflected hidden faces.

[1] http://graphics.stanford.edu/papers/dual_photography/

[2] https://www.youtube.com/watch?v=p5_tpq5ejFQ

jrockway• 13 years ago

This is a long-time favorite effect of anime studios: http://tvtropes.org/pmwiki/pmwiki.php/Main/ReflectiveEyes

renownedmedia• 13 years ago

It's especially effective when the dreamy music is cued and the light colors get all blurry

siler• 13 years ago

Wouldn't it be interesting if we were able to capture so much information in a photograph that you could examine the pupils of the people that appear in pupils, etc.?

aaron695• 13 years ago

Given the Internet has millions/billions of photos on it yet the authors chose not to do a simple random sample, at all, I'm guessing this is more setting up everything exactly correct to create the answer they want.

This is not how one normally does science.

Houshalter2• 13 years ago

The hard part would be identifying the person in the photo, if you didn't already have a list of suspects. Facial recognition isn't that great, but perhaps we could make only partially automated (i.e. the human points out where the nose is and stuff like that.) Then use some kind of bayesian probability to narrow it down (20% of the population has this facial feature, calculating the probability distribution of different faces/facial features producing the image being analyzed, etc.)

DougN7• 13 years ago

This is hardly new. I've been looking at the background of people's reflected pupils in posters (think of the big posters at hair salons and stores) for many years.

crystaln• 13 years ago

Remember when they called it "exponential growth" instead of "a derivative of Moore’s Law"?

Brajeshwar• 13 years ago

[Video] Let's Enhance. http://www.youtube.com/watch?v=Vxq9yj2pVWk

Reflected hidden faces in photographs revealed in pupil

Related Stories