Fusion of Stereo Vision for Pedestrian Recognition using CNNs [pdf]

https://hal.inria.fr/hal-01501735/document

by Katydid • 9 years ago

79 42 9 years ago

hal.inria.fr

donatj3• 9 years ago

Automated jaywalking tickets, here we come.

jlebrech• 9 years ago

burqas for everyone, here we come

thedevil• 9 years ago

Shh.... don't give them ideas.

gotchange3• 9 years ago

Or better stopping vehicle-ramming terrorist attacks like the one in Stockholm last week. Just imagine a system setup on the vehicle where it detects above-average acceleration in a downtown area (GPS supplied) in addition to abnormal swerving coupled with pedestrian data from the sensor network, the auto-pilot system kicks in and takes over the vehicle bringing it into grinding halt immediately.

lawless123• 9 years ago

or we could just take the steering wheels away.

donatj3• 9 years ago

I feel like that would be easily circumventable given time and resources.

I imagine most attacks aren't people being like "You know what, I'm sick of life. I'm just going to ram my truck into people"

Florin_Andrei• 9 years ago

One could also think of the opposite - human-targeting AI vehicles.

I'm sure this is 100% doable right now with existing technology.

bArray• 9 years ago

Please update the fact it's a download in the title.

gridit• 9 years ago

Here is a more proper link, if anyone wants to avoid going straight to pdf:

https://hal.inria.fr/hal-01501735/

itchyjunk• 9 years ago

Pedestrian Recognition system with the ability to retrain for Gait Recognition [0] [1] would allow systems to track any individuals uniquely. Add to this sentiment analysis[2] and we end up in a very sci-fi realm. This make me curious about how an adversarial system [3] would defeat a system like this. Maybe early on, putting a quarter dollar in my shoes and thinking happy thought is all it will take.

On a side note, I am reminded of an anime called Psycho Pass [4] which uses psychological rating given by a super intelligence to decide who needs to be taken off the streets to be treated.

-------------------------

[0] https://www.newscientist.com/article/mg21528835-600-cameras-...

[1] https://en.wikipedia.org/wiki/Gait_analysis

[2] https://en.wikipedia.org/wiki/Sentiment_analysis

[3] http://www.popsci.com/byzantine-science-deceiving-artificial...

[4] https://en.wikipedia.org/wiki/Psycho-Pass

mabbo• 9 years ago

Then add in some of the face2vec algorithms coming out (takes in an image of a person, outputs a vector that stays roughly stable for the same person in different images).

Imagine the airport of the future where they won't bother with "Mr Smith, please report to Gate 6 for final boarding" while Mr Smith sits oblivious at the bar. Instead it will be "Mr Smith? We brought you your bill early because your flight is leaving in 10 minutes and you should get going" "But I didn't tell you my name or flight..."

kirubakaran• 9 years ago

Or more realistically 'The probability of Mr Smith making his flight has dropped below x%, based on his body fat estimate, number of drinks, and distance from gate. Sell his seat.'

"Would you like another drink sir?"

Why alert him when they can sell his seat, sell him a few more drinks, a massage, and some entertainment?

monk_e_boy2• 9 years ago

With strong(ish) AI in your pocket (phone) your personal digital assistant would be pinged by the airport AI, then your digital assistant would remind you get get going.

I imagine this sort of transaction going on a lot in the future. It kinda happens now via email and in browsers. My browser sort of organises and filters all sorts of messages from websites. Also some primitive geo-location also happens... this can only get better as the services mature.

emcq• 9 years ago

So an ensemble beats a monolith? Would be cool to see if the ensemble with distillation could fit to a more computationally friendly model!

w_t_payne• 9 years ago

Within the next 3-8 years every new passenger car will be fitted with camera sensors. (Mandated by public safety standards). Most new cars will also be networked. The ECUs on the vehicles may not be powerful enough for real-time biometrics, and the available bandwidth may not be sufficient for this particular application either .. but it wouldn't take much to make it technically feasible. Now., I feel uneasy about the societal risks involved with us going down this route, but I know that there are others who would jump at the chance to exploit this opportunity.

7952• 9 years ago

Safety technology can take a long time to be adopted. The vast majority of cars still emit toxic gas for example.

dagenleg• 9 years ago

Wait, who ate most of the paper?

roymurdock2• 9 years ago

Problem: Improve the classification component of an ADAS system to be able to discriminate between the obstacle type (pedestrian, cyclist, child, old person) in order to adapt the car driver system behavior according to the estimated level of risk.

Approach: Train a CNN-based pedestrian classifier using a combination of intensity, depth and flow modalities on the Daimler stereo vision data set. The resulting CNN classifies objects in images as "Pedestrian" or "Non-Pedestrian" (rather than the more granular goal above).

Result: Outcomes are measured by the False Positive Rate (FPR) at the 90% True Positive, or detection rate. Lower FPR=better.

The CNN created in this experiment ("Late Fusion") achieves an FPR of 0.0125%, which is almost 50% better than the 2010 "Early Fusion" CNN's 0.02% FPR over the same dataset.

Still, a 2011 model ("HOG+LBP/MLP, Monolithic HOG classifier") that uses "explicit features" vastly outperforms both, achieving an FPR of 0.00026% over the same dataset.

Conclusion: "We showed that the late-fusion classifier outperforms the early-fusion one but does not go beyond the state-of-the-art."

thecopy• 9 years ago

This should be the abstract.

selestify• 9 years ago

Wow, thanks for doing this summarization. I wish papers were written more in this clear manner.

yitterational• 9 years ago

No pedestrian recognition or accident avoidance system is going to be 100% effective. But now we have the opportunity to apply to motoring what we have done to aviation: every rare fatality and the situation that led up to it can be analysed in depth to prevent recurrence.

joshontheweb3• 9 years ago

Frankly, I'm very terrified at the prospect of being tracked in my every move and it seems entirely unavoidable. License plate scanners are already in wide deployment. There was an article on HN last week about the FBI's 100 million strong facial recognition database using peoples drivers license photos without their consent or knowledge. Not to mention the wholesale tracking of our online activities by the NSA which is already a done deal. I'm seriously thinking about leaving the US to buy myself some time. I'm not hopeful that anywhere will be safe in the end as other countries catch up with this technology. What can be done if this isn't the world you want to live in?

strictnein• 9 years ago

"Not to mention the wholesale tracking of our online activities by the NSA which is already a done deal. I'm seriously thinking about leaving the US to buy myself some time"

Leaving the USA will actually ensure the NSA collects a lot more about you. Staying will actually ensure they collect very little, mainly as a byproduct of the methods they use and not because of any nefarious plot.

joshontheweb3• 9 years ago

I was more meaning to buy myself some time from the physical real-world tracking. We have no way of knowing what the NSA is actually storing on citizens. I have to assume it is everything and that doesn't seem outlandish in light of the Snowden leaks.

BurningFrog• 9 years ago

"Unavoidable" is the key thing to realize.

Even if you manage to get your government to never do this, it is (or soon will be) easily within reach for private hobbyists.

My window faces a major freeway. I could probably rig up a license plate reader that registers all cars going by fairly cheaply. If not now, then surely in 5 years.

Everyone has a quite nice camera in their pocket. The car I bought last year comes with 3 cameras. And so on. I think it's an unavoidable fact that in the very near future, you have to assume you are being filmed whenever you're in public.

So I think the real question is: How do we adapt to this new world? How can we limit the bad aspects and enjoy the good aspects? Are we really sure what is bad and good?

randcraw• 9 years ago

Two thoughts:

1) It's a lot easier to recognize 100M distinct plates than faces. I don't see anyone having the resources and patience to do the latter any time soon. Even the FBI isn't claiming they can recognize 1 face/100M with low error today. At best, they have such a database of pictures. I'm have big doubts that usefully matching against so many faces is even technically possible.

2) I think you're right, given the rapid advancement of tech, our emphasis shouldn't be whether we should outlaw surveillance practices and methods. Instead we should focus on controlling access to such databases. Who gets to gather or use such data and when should it be authorized? Should we require that each ID match request be explicitly granted (like a warrant) or should we at least require that all such requests be logged, and perhaps publicly reported?

Once access to sensitive data is regulated, it no longer matters how the data is gathered. Abuse of such a law should poison any fruits of the data, at least on the part of lawful agencies.

joshontheweb3• 9 years ago

If it truly is entirely unavoidable, then I suppose one way to adapt is to democratize access to these tools and data. The power to abuse is too great if it is concentrated in a small number of hands. Still this seems scary to me. The one solace I take in this is that the gov't has the most dirty laundry to hide and this loss of privacy should help to expose a lot of the corruption going on.

dclowd9901• 9 years ago

It's great that this can recognize pedestrians, but I've still seen no theory around decision-making. This last week, I happened upon a tree that fell into the road after a bad wind storm. I _know_ a tree when I see it, and I know it is large and heavy and difficult to move and would take a great deal of time for that to happen, so I immediately cut to a route around it. But what if it was a bus or truck turning around? Well I know that a bus turning around is large and heavy and difficult to move, but it will be moving out of the way in a moment, so I can wait for that to happen.

When are we going to start seeing methods for measuring intent or theory around guessing the future nature of an object based on contextual understanding of that object? How does a car recognizing a pedestrian help if the pedestrian starts running toward the car?

It just feels like there's a lot of patting of backs that happens around this stuff, when we really aren't even close until we have a system that has as learning and understanding approach that is as abstract as a human's.

roymurdock2• 9 years ago

As we move from relying on humans to relying on machines, we'll need to develop better infrastructure to guide those machines.

Our current roads are OK for human drivers, as they are designed heavily around visual signals, we can process and synthesize a lot of visual/audio/learned data well.

Machines can process visual data faster than us, but can't synthesize it with other data nearly as well as we can. Adding sensors that passively communicate information, such as the location of lanes, other cars, and humans, will reduce a lot of the issues and edge cases of visual-based ADAS systems.

I could imagine a future where cars are always listening for smartphones (or other wearable/implanted sensors) in "pedestrian" mode, which could be used to override regular function and stop the vehicle if detected within a certain distance, or intersecting at a certain trajectory.

hacker_9• 9 years ago

Agreed. Reminds me of this article: "The state of Computer Vision and AI: we are really, really far away." (2012) [1]

[1] http://karpathy.github.io/2012/10/22/state-of-computer-visio...

monk_e_boy2• 9 years ago

And when you see a bus driver come across a tree or some other impassable object in the road, the level of decision making as the driver leans out the window, barks orders to other divers to back up. Does a 3 point turn, bumping up the pavement, reversing down someones driveway, scooting over the grass verge while giving a cheery wave to everyone he pissed off while doing it. AI is not there yet.

waynecochran• 9 years ago

Next we develop camouflage to avoid detection...

saosebastiao• 9 years ago

Stuff like this is why I believe that level 4 autonomy within the next decade is quite the lofty goal and level 5 is completely out of the question.

For example, in WA State, you can cross at any unmarked crosswalk which is defined as any intersection (plus a few non-intersections), unless there is a sign explicitly disallowing crossing. I wouldn't know how to accurately estimate this, but there are probably over 100k unmarked crosswalks in Seattle alone (most of which are ignored completely by humans). The pedestrian's responsibility is to not enter the roadway without enough time for a car to stop. The car's responsibility is to stop once the pedestrian signals intent to cross. Signaling intent does not require waiting at the edge of the roadway...merely walking towards the roadway while on the sidewalk is considered signaling intent to cross.

What this effectively means is that the car must do far more than understand pedestrian behavior in crosswalks. It must understand the periphery of the road...not just the pedestrians it can see, but the pedestrians it can't. Most speed limits are actually too high to take into account legal pedestrian behavior. The speed limit is merely an upper limit; driving too fast for conditions can always override the speed limit as a factor in an accident. This means that trees, buildings, etc., that obscure view of and behavior of pedestrians on the sidewalk implicitly lowers the speed at which cars can drive on that road.

Now today we essentially get away with favoring the driver. The driver can assume that they were acting reasonably by driving the speed limit, and we can say that the pedestrian was out of line by walking too fast out into the street, and most situations are resolved blindly based on testimony with no evidence. But the risk model of autonomous cars is different. The manufacturer is gonna be on the hook, not the driver, and the car needs to be able to obey the letter of the law with a higher burden of proof and with dozens of sensors recording the situation.

As we can see from this paper, pedestrian recognition is still a hard problem. It's hard in the engineering sense, and the mathematical sense. But cars don't need to just recognize pedestrians...they need to recognize and understand the context in which a currently invisible pedestrian could appear in the near future. And perhaps more importantly, they still need to solve the problem of instantaneous path planning in a 2d space with a dynamically changing 3d obstruction model consisting of far more than just pedestrians. It's easy to look at the past and say we've made rapid progress in the past so we'll make rapid progress in the future...but in the process of maturing a technology we regularly see exponential progress in the beginning and asymptotic progress near the end. This is going to be very difficult.

Fusion of Stereo Vision for Pedestrian Recognition using CNNs [pdf]

Related Stories