Eulerian Video Magnification for Revealing Subtle Changes in the World

http://people.csail.mit.edu/mrub/vidmag/

by clockwork_189 • 14 years ago

566 129 14 years ago

people.csail.mit.edu

samirahmed• 14 years ago

i wonder if this could be used for lie detection

tokenadult• 14 years ago

i wonder if this could be used for lie detection

I was tempted to downvote, but this is a legitimate question, and I guess you haven't had opportunity to learn about how bogus "lie detection" in general is. I happened to attend university where some of the earliest rigorous researchers on lie detection were active,

http://www.fas.org/blog/secrecy/2006/09/david_lykken_and_the...

http://images.salon.com/april97/news/news970410.html

http://antipolygraph.org/articles/article-018.shtml

http://www.psychologicalscience.org/observer/getArticle.cfm?...

so people in my town have known for a long time that purported lie detection tests don't work, and indeed our state has long banned the use of polygraphs or other purported lie detection devices for any courtroom purpose. Law enforcement investigators in our state use actual interviews of witnesses and suspects, and by state law all police interrogations must be videotaped. That helps immensely in reducing mistaken conclusions from improperly conducted interrogations.

bane• 14 years ago

I'm actually thinking if it might be a good tool to detect distress in a subject. Not lying exactly, but might provide insight into a subject's emotional state non-invasively during questioning or interrogation. People might become distressed for many reasons that may not just be simple lying.

Perhaps even a Voight-Kampff-like test a la Blade Runner/Do Androids Dream of Electric Sheep?

Symmetry• 14 years ago

Its quite true that polygraphs and similar means of telling whether a person is distressed are almost useless. However, there have recently been some advances in using fMRI to do direct lie detection by looking at what parts of a person's brain are being used at a given time. This doesn't seem to be reliable enough for courtroom use yet, but its getting there.

tsumnia3• 14 years ago

I said it above, but to carry on with what tokenadult said; Lie detection is a little more complicated than detecting facial expressions. If you have a defensive, introverted person looking downward, they might be lying or that could be how he responds to whatever interrogation they are under. Conversely, a mentally distressed individual (or someone who has done a heavy amount of NLP) could lie straight to your face because they personally believe they are telling the truth.

This type of technology will no doubt be used to attempt lie detection, but it won't work. The only thing we can do is use the context clues of the face to infer what we think is going on.

fferen• 14 years ago

I like how this kind of turns the "main idea" of computer vision on its head. Namely, instead of trying to emulate and catch up to human abilities, it augments them the way only a computer can; it plays to computers' strengths instead of their weaknesses.

Also, who would have thought you can determine someone's pulse just by looking at the color change of the face due to blood rushing in and out? If asked beforehand, I wouldn't have even said it was theoretically (physically) possible.

wxs2• 14 years ago

I also found it very surprising that cameras could pick these things up, there's actually an iPhone app from Philips that claims to be able to be able to read heart-rate, I assumed it was snake-oil at first :

http://itunes.apple.com/us/app/vital-signs-camera-philips/id...

gmurphy• 14 years ago

I was skeptical too, but when I tried this app a few months ago, I compared it to my Garmin heart rate monitor at rest and after exercise - they were usually within 1bpm of each other. I was very impressed.

jasonwatkinspdx• 14 years ago

There is so much faint and indirect information in the world it's dumbfounding. We're crossing a threshold where computation can make this visible. It's really exciting.

zher3• 14 years ago

Yes, it's hard to believe, but it's really possible to sense the small color changes due to blood flow in the face using cameras. My earlier work described how you just need a simple webcam :)(http://web.mit.edu/newsoffice/2010/pulse-camera-1004.html). As a result, I'm working on Cardiio (http://cardiio.com) to bring this technology to both the web and mobile very soon. If you're interested in computer vision based biosensing, please sign up!

Sidnicious• 14 years ago

That’s awesome!

Hmm, when I tried to sign up I got a message that I was already on the waiting list.

EDIT: It looks like the page is making a request to cardiio.com, specifically, and if you’re at www.cardiio.com (the one you linked to) the response gets blocked by Same Origin Policy, and the page treats it the same as a you’re-already-on-the-list response. You might want to avoid hardcoding the hostname in the JS, and/or force everyone over to the www or non-www domain.

taliesinb10• 14 years ago

That sounds awesome.

But from a scientific point of view, you don't really need spatial resolution for the heartbeat example, though, do you? If you just spent the same amount of money on getting high temporal and spectral resolution, surely one could do very cool things with 'just one pixel', so to speak. Especially if you had say 16 different wavelengths, instead of just 3.

FrojoS• 14 years ago

Congratulations for, what seems to me, opening up the door [1] to a new world of CV applications! And of course for having the courage to commercialize it while your academic career is taking off.

[1] famous illustration from the enlightenment: http://www.lehrer.uni-karlsruhe.de/~za874/homepage/aufklaeru...

defdac• 14 years ago

Do you think it would be possible to see shifts in photosynthetic activity in plants due to low/high nutrients/humidity? Especially when IR-filter removed?

aangjie• 14 years ago

Just curious have you tried on people with darker skin colour/tan. I am trying to imagine, what adjustments you'll need to make to detect on them?

lurker143• 14 years ago

What do you think of the many apps already on the market, possibly derived from your earlier work?

https://play.google.com/store/apps/details?id=com.macropinch...

thisrod• 14 years ago

Pulse oximeters work by looking at the fingernails, and they're the usual way to measure a pulse. There's a lot of blood flow in the face - try bumping into something sharp - so this isn't too surprising.

pygy_• 14 years ago

A Pulse oximeter doesn't measure the spontaneous signal.

There's a red/infrared led on one side of the finger, a detector on the other side, and it measures the amount of light absorbed by the blood.

wxs2• 14 years ago

If people find this compelling, they may also like this earlier work from the same group:

http://people.csail.mit.edu/celiu/motionmag/motionmag.html

This other algorithm amplifies only motion, and the algorithm is less elegantly simple, however its results are very interesting and fun to watch as well. Make sure to watch the video at the bottom of the page

http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2...

pbhjpbhj2• 14 years ago

I watched that second video http://people.csail.mit.edu/celiu/motionmag/VIDEO_siggraph_2... - that must be incredibly useful in engineering analyses.

zemaj• 14 years ago

Wow, reading someone's heart rate from a video.

Imagine that plugged into Google Glass. Overlay someone's heart rate next to their face while you're talking to them!

DannoHung• 14 years ago

How about amplifying micro-motor motions in the eyes or something like that?

jtchang• 14 years ago

Excuse me while I book a trip to Vegas.

reginaldo• 14 years ago

I was thinking about the implications of using this technique to analyze e.g. political speeches and try to catch people lying on the act. Your application (winning on card games) seems very interesting too.

jorgem• 14 years ago

Something tells me Casinos aren't going to let you wear Google glass... :)

aptwebapps• 14 years ago

Imagine when everyone is an amateur polygraphist ... some conversations are going to become unnecessarily fraught.

taliesinb10• 14 years ago

The use I thought of was figuring out when your date was trying to perk up the courage to kiss you.

And personally, I don't think any of the "modern-day lies" that people tell would cause any kind of increase in heart rate.

aw3c2• 14 years ago

and then it turns out your date was extremely uncomfortable and stressed out, trying to think of ways to get away.

it is not as simple as you wish. and that is good!

carterschonwald• 14 years ago

This is pretty darn amazing! I'm so going to try to implement this set of algorithms later this week! :)

theschwa• 14 years ago

The Is He/She flirting with me App is just around the corner. I wonder what new privacy issues will crop up with this kind of technology.

btipling• 14 years ago

Amazing for many reasons, but what I really find impressive is that messing with video encoding requires like a PHD in everything because of how complex things like H.264 are. At least that's my perspective after trying to grok the H.264 spec.

There are all kinds of optimizations and algorithms for network latency and video sizes, etc. I don't consider myself a layman but I certainly do not have the knowledge and skill to do anything at all with video.

Do you know what a Inverse sub-macroblock partition scanning process is? I don't, not even after reading the part about it in the spec.

justinsb• 14 years ago

No need to mess with the encoding format - just use an existing utility (or library) to grab the raw frame data. e.g. ffmpeg

ZenPsycho• 14 years ago

a macroblock is a chunk of an image which is identified to have some constancy between frames. a sub-macroblock would be some subdivision of this for finer motion details. sub-Macroblock partitian scanning would be applying an algorithm to identify the sub-macroblocks in a set of images. Add the word "Inverse" though and now I have no idea what the hell that means.

taliesinb10• 14 years ago

This really doesn't have much to do with what makes computer vision difficult (or rather, interesting). I doubt very many computer vision researchers are conversent with the H.264 spec.

Typically toolkits like OpenCV take care of codecs for you (which make them quite annoyingly difficult to compile on different platforms).

(I speak as someone who had done commercial computer vision algorithm development.)

joeconway2• 14 years ago

Unfortunately when they do (My dissertation is on motion analysis using H.264 compression artifacts) things get pretty complicated very quickly

anigbrowl3• 14 years ago

I'm not seeing what the big deal is here. Extraction of differential information from video frames has been standard in video editing and compositing software for years. Grab a demo copy of After effects, folks.

taliesinb10• 14 years ago

This is kind of the stock-negative comment that I don't like about HN. It would be redeemed if it had a link, but instead its just a look-how-cool-I-am-that-I'm-not-impressed-by-this actually-pretty-awesome-technology-demo, dammit.

tptacek• 14 years ago

The commenter you are replying to (and downvoting) has done (IIRC) professional digital video work, and written many detailed and helpful comments on HN about digital media. The comment you're replying to isn't snarky. Assume good faith. Maybe he's just mistaken? Or maybe he's right, and the thread just doesn't like that?

Here are some best-of's to peruse before concluding that 'anigbrowl is part of the problem with HN:

* http://news.ycombinator.com/item?id=1875412 (MIDI)

* http://news.ycombinator.com/item?id=995096 (3D film)

* http://news.ycombinator.com/item?id=4011770 (Google Moog doodle)

* http://news.ycombinator.com/item?id=1263289 (DV on DSLRs)

A couple suggestions:

* If a downvoted comment with no replies upsets you so much that you feel a need to pile on with a pointed negative comment, go to the bottom of the page, type the commenter's name in, click 'comments', then sort by 'score', and read the top 10 or so first. You may find the urge to pile on dissipates quickly.

* While it certainly isn't going to cause drama here ('anigbrowl being one of the more levelheaded contributors to HN, way more so than me), I think you'll find that comments like yours tend to read as personalizing disputes, and lead to embarrassing spats you'd be better off not attaching yourself to.

anigbrowl3• 14 years ago

I referenced a particular piece of software, which I thought was well-known enough not to need a link. I am not trying to look cool, but rather point to existing cool technology, and stand by my comment.

JackC• 14 years ago

I'll bite. I've used After Effects at the grab-a-demo-copy level for motion tracking and stabilization and rotoscoping. I don't remember anything that would have allowed me to detect and amplify color changes in someone's face that way, or amplify movement of a given period like the shots of someone's pulse or of a baby breathing -- especially not with a general-purpose temporal amplification filter instead of really specific code. I'd love to know how to do that stuff with After Effects, though. Can you be a little more specific?

My guess is you might end up with something along the lines of, say, take a specific clip of a guy's face, mask the face, average the color, detect the subtle changes with a script, and use the output of the script to control a color filter. Likewise if I had an existing clip of a particular sleeping baby, I could target the chest, motion track, and feed the tracking data into a filter. But that's a laborious way to fake particular outputs of what this software does in realtime on arbitrary clips of things moving or changing -- it's not nearly equivalent.

emmett• 14 years ago

Is there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.

taliesinb10• 14 years ago

According to the paper, it is really quite a simple idea.

Imagine a filter that one might use in an audio context that boosts the amplitude of frequencies falling in a certain range. But now imagine applying that filter to a timeseries that actually comes from the intensity of a single pixel over time.

This has the effect of emphasizing motions that occur over timescales that correspond to the selected band of frequencies.

The reason this is true actually comes back to the Fourier transform -- you can imagine the motion of part of a complex image to be composed out of the linear combination of many simpler 'sinusoidal motions', each of which is being amplified by the per-pixel filter.

tomp2• 14 years ago

What about motion amplification? For example, the baby breathing in the bed... Aren't multiple pixels involved then?

riemannzeta• 14 years ago

These guys are going to make millions on baby monitors alone.

pbhjpbhj2• 14 years ago

Maybe, does it work well in a darkened room?

Baby monitors should probably be IR based; then this would appear to be a great addition allowing a pulse to be determined from the video feed.

Current baby monitors use a pad underneath to detect movement including breathing.

Personally, we used the "you can hear them screaming half-a-mile away" method of baby monitoring.

RockofStrength3• 14 years ago

This type of technology would be a nice novelty item to implement into a futuristic zombie story. For example, there could be a large battle between humans and zombies, and the humans are equipped with pulse-magnifying glasses, allowing them to identify friend vs. foe.

vibragiel• 14 years ago

Instant Heart Rate for Android uses something similar to check the pulse using the phone's camera.

https://play.google.com/store/apps/details?id=si.modula.andr...

stefantalpalaru• 14 years ago

Even more similar: http://www.whatsmyheartrate.com/

joelthelion• 14 years ago

Does anyone have a link to the paper?

apu2• 14 years ago

It's on the page: http://people.csail.mit.edu/mrub/papers/vidmag.pdf

twp• 14 years ago

Finally, you can "enchance" videos, just like they do in the movies: http://www.youtube.com/watch?v=Vxq9yj2pVWk

eridius• 14 years ago

That was fairly ridiculous, but where on earth did that clip of rotating the video come from?

newbie12• 14 years ago

This technology is badly needed today for skin cancer screenings. We could save tens of thousands of lives a year with better detection of cancers and pre-cancerous growths. It is crazy that most screenings in 2012 are still 3 minutes a year with a harried doctor.

JacobIrwin2• 14 years ago

If anybody could help clarify a concept (a specific filtering method), from video:

"we filter the sequence temporally using a "???" ...and boost the temporal signal to generate a video..." (at 2min. 45sec in video)

What type of filter is being described (missing from quotes; "???") ? Bonus upvotes for providing a URL to definition (detailed description of the missing term).

noisebeuter• 14 years ago

He talks about a "Temporal Bandpass Filter". Not sure about the "temporal" part, but in DSP a bandpass filter is a filter (think Audio-EQ) that passes frequencies within a specific range and rejects the rest. See https://en.wikipedia.org/wiki/Bandpass_filter

JacobIrwin2• 14 years ago

Thanks much noisebeuter!

0x092• 14 years ago

It would seem to me from perusing the spec that they are simply taking a video, doing a 1D DCT on the time axis and amplifying a "frame" in the result corresponding to the desired period. Hopefully someone can correct me if there's more to it.

I'd like to get at one of their sample videos before filtering because it would be pretty easy to toss together a workalike based on the above.

jasonkester• 14 years ago

There's a whole business to be made off that five second clip where they amplify the motion of the sleeping baby's chest.

Anybody who's ever been a parent can remember going up to check on the baby at night, standing over the crib for a full minute thinking "move! Move! MOVE!" at the lifeless body of their baby, running all the horror stories they've heard about cot death over and over in their head until finally the kid's chest moves enough to breathe a sigh of relief and go back to bed.

The first company that adds this feature to a video baby monitor will make a mint.

wcoenen• 14 years ago

That business already exists, though the existing sensors typically come as a pressure-sensitive mat (placed under the mattress) instead of a camera.

JoeAltmaier5• 14 years ago

A business preying off the fears of new parents, marvelous.

elemeno2• 14 years ago

There's a bit difference between building a business that preys on the fears of a new parent and a business that acknowledges that new parents already have fears and tries to reduce them. This I would say falls firmly into the later category.

It's the difference between a tool which helps you make sure that your baby is ok and one which tries to persuade you that if you don't use it you're creating unconscionable risks to your baby which this product could save you from. The former is a tool which is good, the later is one which preys on parents.

ctdonath3• 14 years ago

Haven't been a new parent, have you?

$5 for a suitable app is awfully cheap assurance. $50 for a dedicated device, ditto. We're not talking "taking advantage" here, we're talking something every parent in history wanted.

psb4• 14 years ago

If someone wants to do a kickstarter to try and use this technology to build a glucose meter, I will be the first contributor

th0ma5• 14 years ago

:( unfortunately might not be entirely applicable to this problem... unless some measure of blood density and hence probably glucose mix... but at least at this layer of filtering and reasoning, it appears to be about minute amounts of motion... is there something about motion that could denote glucose? trembling maybe? maybe a score could be derived using something that borrows from this technique, but you'd have to put more into it than it seems this technique specifically shows.

psb4• 14 years ago

I was thinking more skin color or possibly changes in the eye (if I remember right eyes get swollen when blood sugar is high) Honestly, I don't know if there are enough outwardly visible changes for this to even be possible. But there's a lot of money out there waiting on the person who can figure out a way to use a cell phone do something like this.

mistercow2• 14 years ago

The simplicity and elegance of this algorithm is really breathtaking. I'm mostly thinking out loud here, but I think you could use a related process to do inter-frame interpolation for smooth time-scaling of video.

0x092• 14 years ago

You can! http://ompldr.org/vYWh5eA/test2.mp4

This was made by resampling in time with a DCT. Problems with this: 1. practically, it needs the entire movie in memory. 2. gibbs phenomenon in time is really weird. These could both be addressed by using short overlapping windows, and then the process becomes just like typical sinc/lanczos interpolation in time.

mistercow2• 14 years ago

Well, yes, you can interpolate in the frequency domain like that, which works pretty well for resampling 2D data as well. However, it does not transition shapes from location to location; instead, it looks like one frame is fading into another. What's fascinating about this multiscale algorithm is that it appears to actually exaggerate the motions rather than merely increasing the contrast of the changes.

siavosh• 14 years ago

From my experience with CV, one of the biggest unsolved problems and rarely publicized aspect of the field is how much variable tuning is often required to get good results. I'm curious how sensitive these demos are to different environmental conditions and the robustness of the algorithm.

Hitchhiker• 14 years ago

The team's initial work on topic:

http://people.csail.mit.edu/celiu/motionmag/motionmag.html

RockofStrength3• 14 years ago

I have a tangential question: What's the state of technology that would allow one to see through the eyes of another? This would be useful for piano tutorials, etc. I think it could be done with a separate eye-tracking device and a camera located between the eyes.

bergerj• 14 years ago

Wouldn't a keyboard with keys that light up be better and easier?

regularfry2• 14 years ago

You don't get to play with proprioception that way :-)

groby_b• 14 years ago

Uh, what? If you play piano, you're looking at the score, not the keys.

RockofStrength3• 14 years ago

Not necessarily; I usually don't use a score when I play. It would also be insightful to see the 1st person perspective of a jazz musician playing from a lead sheet, to see how his eyes jump between page and piano.

apu2• 14 years ago

From 8 years ago:

http://www.cs.columbia.edu/CAVE/projects/world_eye/

http://www.cs.columbia.edu/CAVE/projects/eyes_relight/

MPSimmons• 14 years ago

Excellent. Integrate this into the Tricorder projct: http://tricorderproject.org/

lurker143• 14 years ago

I just downloaded an app[1] that measures heartbeat rate by taking video of a fingertip.

I was stunned by the brilliance of the application, how ubiquitous video sensors can replace other senses.

There's something else to discuss here: We have a recently published paper from MIT, including mathematical details of the process; and at the same time a stack of closed-source commercial applications already in the market with slick interfaces. Are they using the same technique? Is one app-maker trawling preprints for app ideas? Are many app-makers copying one pioneering app-maker?

[1] https://play.google.com/store/apps/details?id=com.macropinch... but there are many similar apps.

lurker143• 14 years ago

This is a similar idea to the temporal arterty thermometer, which uses an infrared camera: https://en.wikipedia.org/wiki/Medical_thermometer#Temporal_a...

The mathematical model for the sensor is probably different, though.

Eulerian Video Magnification for Revealing Subtle Changes in the World

Related Stories