Will your driverless car kill you so others may live?

http://www.latimes.com/ct-driverless-cars-safety-google-perspec-20151209-story.html

by walterbell • 11 years ago

34 71 11 years ago

jasonjei2• 11 years ago

How about a simpler solution? Designate certain roads as autonomous only to assure safe operation among other vehicles (for example highways do not allow pedestrians, bicycles, etc; we could require certain roads to be autonomous only as self-driving vehicles become more commonplace), and maps should be updated to identify all possible pedestrian crossings to drive pessimistically?

CM303• 11 years ago

That's a decent solution for some roads, though it still doesn't solve the issue of how these cars will work in towns and other highly populated areas, where pedestrians and cyclists are everywhere and they can't reasonably be expected to only be on the pavement or only cross by pedestrian crossings.

It also doesn't really work outside the US, since a lot of places are quite fine with 'jaywalking'. For example, in the UK, you can cross the road just about anywhere. These autonomous vehicles will have to programmed with this in mind.

jasonjei2• 11 years ago

How about a lower speed limit where jaywalking is permitted or where the risk of pedestrians running into the road is high? For example, you could almost drive at 65mph with certainty on a freeway, but you might have to drive at 25mph to give you enough time to react to pedestrians?

erdojo• 11 years ago

Or, like with other automated equipment, people learn to not do stupid behavior that will get them run over.

For example, we all just know not to step in front of a train. We have NO EXPECTATION it can stop. Do accidents happen? Yes, but only at the fault of the pedestrian (intentional or not).

If we all know that every car won't stop for us in between crosswalks, we may still jaywalk but we'll be danmed sure a self-driving car isn't coming. Right now a lot of people jaywalk and are overconfident if a car is coming it'll stop.

CM303• 11 years ago

How about the car isn't designed with utilitarian ethics in mind and it instead automatically values anyone in it more than other people?

Okay, that's probably a very unpopular opinion, at least when expressed in that way. But to be perfectly honest, it's probably what a lot of people riding in said cars want to happen, and perhaps a bit closer to how cars work in real life when not driven by a computer.

Good luck trying to sell selfish AI to the public though...

bsder2• 11 years ago

How about the car isn't designed with utilitarian ethics in mind and it instead automatically values anyone in it more than other people?

I don't consider that an unpopular opinion, at all, thank you very much.

In reality, these questions don't take into account the fact that humans don't actually even make decisions in these situations. People don't choose whether to kill that child or swerve into oncoming traffic--they don't even react in time for there to be an option or the reflexively jerk away without thought. Truckers don't get time to consider whether they're going to kill other people on the road with a jackknife or roll when they have to do a sudden avoidance maneuver.

Autonomous cars are going to be so much better at avoiding situations preemptively that you will have to be pathologically stupid to be able to put an autonomous car into such a situation. And, if you are that stupid, YOU deserve to die.

CM303• 11 years ago

But you're never going to avoid situations like this completely, regardless of how good the car itself is. Imagine one breaks down. Or gets hit by something (whether that be due to a natural disaster, the local landscape, human actions, animals, etc). Or it gets hacked. Or it just plain run out of power.

Now imagine the other autonomous cars following it have to make an urgent decision; go straight or swerve out of the way. Imagine the only somewhat clear path has some innocent people on it, and the alternative is basically to let the car end up as a giant fireball after it hits whatever's in front.

You can try and avoid similar situations as much as possible, by having the car figure out what's going on ahead and do things to avoid it (change lane, slow down, etc). But the situation isn't avoidable all the time. If something happens close to you and you're going down a motorway at 70mph, then physics alone says the car cannot stop in time. It will have to decide then to either take the risk or get obliterated.

It's not pathological stupidity, it's often bad luck. And in cases like that, the driver is right to assume the vehicle won't sacrifice them for others.

quicklyfrozen2• 11 years ago

And good luck selling the unselfish AI to individual buyers :-).

People buy SUVs because they perceive them as "safer" -- but clearly that safety comes at the expense of passengers in other smaller cars.

chrisfosterelli2• 11 years ago

Every time I read one of these articles I feel like the author doesn't have a very good grasp of exactly how these work. The car doesn't have to "choose". Nobody at Google is programming these things to calculate the life value of humans, the percent chance of injury to occupants, or the benefits of sacrificing pedestrians.

They're programming them to try and avoid accidents in general. There will be times when the computer fails to do that, but that doesn't mean it decided to kill someone by making a moral choice about human life priority. Maybe in the future this will be a relevant argument but the sort of algorithms mentioned that decide which crash scenario is 'best' simply don't exist.

The best way to deal with these moral issues is to just... not program that behaviour.

m0nty• 11 years ago

I think people are excited about this because it seems to mesh so closely to the Trolley Problem, which has been an important thought experiment in the study of ethics.

https://en.wikipedia.org/wiki/Trolley_problem (They even have an "Implications for autonomous vehicles" section now.)

So, who knew? Suddenly it's not so theoretical, it apparently has a real-world application. Except, as you seem to be getting at, probably the only thing engineers will be considering in any depth is how and when to brake, not trying to swerve the car elaborately in one direction or the other. The number of occasions when a serious accident occurs because a driver reflexively swerved to avoid an animal or other obstruction (sometimes literally dozens dead in one accident) emphasises that swerving is not so good a strategy as braking. Or, as my driving instructor once told me, "There is only one cause of accidents on our roads: failing to stop in time."

Already__Taken• 11 years ago

A car has a fixed contact patch with the road, 100% of this is used to accelerate, brake, or turn. Any combination of those must stay within that 100%.

Why would a computer choose to swerve and not slam the brakes. With reactions that are as fast as a computer could be what kind of a situation even is a swerve vs a course correction.

adventured• 11 years ago

Even in the near future, I'm skeptical a car's AI is going to be able to know how many babies are in the other car, such that it decides to kill me instead of putting said babies at risk. It's such a reach as to be silly.

Then if you run the outlandish scenario, and assume extraordinary leaps in AI comprehension in the next 30 or 40 years, it still remains silly. How many teenagers are equal to one baby? How many puppies does it take to equal one 80 year old grinch? There is no scenario under which it will work. Ultimately the AI will not be able to make accurate judgments about morality / character, it can never choose correctly even if you try to program it with what would be strictly subjective value systems - that one flaw guarantees it can never work properly.

TeMPOraL2• 11 years ago

How many teenagers are equal to one baby? How many puppies does it take to equal one 80 year old grinch? There is no scenario under which it will work.

Well, this is something we don't expect from humans. No human, except maybe high-ranking businessmen, leaders and armchair ethicists ever does that kind of calculation, because in the usual case we meet them, there is no time. When you're a second before a car crash, you don't count the passengers of the other vehicle and evaluate their relative worth compared to yourself. You swerve, or brake, or generally do whatever you can to not hit them. And so will the self-driving cars, except better. In fact, I think we can avoid the moral issue altogether by just focusing on making it nigh-impossible to surprise an autonomous vehicle in such a way that it won't be able to reach a safe state[0] in time. After we have that, then we may consider optimizing the algorithms to take into account that e.g. the other vehicle is empty, so its suitable to be used as a braking device.

[0] - Either avoidance trajectory or a speed that would not be likely to kill a pedestrian if they were hit directly.

archgoon• 11 years ago

Nobody at Google is programming these things to calculate the life value of humans.

Well, not everyone at Google agrees with you. Yonatan Zunger seems to think that yes, you do have to work out what the car has to do in certain worst case scenarios.

'A self-driving car, on the other hand, would require that these decisions be specified in advance.'

https://plus.google.com/+YonatanZunger/posts/FMdAD6ckH38

And well, he's right. Once you have programmed the car, it will respond deterministically to inputs. You can close your eyes and ignore the consequences of the decisions you made for the Trolley analogs, but that's willful ignorance of your decisions at that point (and bad QA).

chrisfosterelli2• 11 years ago

And well, he's right. Once you have programmed the car, it will respond deterministically to inputs.

This is true, but that doesn't mean that the response is based on the car making moral decisions about the value of the human lives involved.

As the commenters below mentioned, the technology doesn't even exist at this point to do anything other than avoid the object in front of you and try not to crash while doing it. Sometimes this may result in a dead pedestrian and sometimes this may result in a dead occupant. That isn't the car making a moral judgement, it's just collision avoidance.

wysiwylwysiwyl• 11 years ago

The algorithm faces a choice when a drunk driver is on collision course and evasive action is required, what if there is no safe area to steer towards? It is a choice by design whether the algorithm deals with this situation by inactivity, failure or selecting the scenario with the lowest collision speed.

coldtea2• 11 years ago

Every time I read one of these articles I feel like the author doesn't have a very good grasp of exactly how these work. The car doesn't have to "choose". Nobody at Google is programming these things to calculate the life value of humans, the percent chance of injury to occupants, or the benefits of sacrificing pedestrians.

You'd be surprised. Actually even if they don't, they can (and probably will) be regulated to do so.

alephu5• 11 years ago

Moral issues aside, don't you think that crashing well is an important design feature?

For example, to use ultrasound to map the hardness of nearby obstacles and go for the one that provides the quickest safe deceleration, or to orchestrate a decelerating glance with an oncoming vehicle?

unabridged• 11 years ago

I'm worried about more mundane situations. What happens at a 4-way stop if they all get there at the same time? Will the cars drive so passive/defensive that none of them have the "balls" to break symmetry?

bsder2• 11 years ago

Will the cars drive so passive/defensive that none of them have the "balls" to break symmetry?

Yes, that "behavior" had to be programmed in.

The autonomous vehicles had to learn to be "assertive" (ie. start rolling slowly into the intersection) in order to not confuse other drivers at the stop.

The vehicles also had to learn that a hipster standing on a fixie's pedals rocking back-and-forth was not an intersection hazard in spite of scoring high on the idiot detector.

AstralStorm• 11 years ago

Human drivers communicate in such a rare situation, e.g. by blinking lights to signal they won't go. Driverless cars should be able to do that as well.

jacalata• 11 years ago

I think the typical CS solution to such a deadlock would be a random exponential backoff, no?

tomp• 11 years ago

Exactly. This is an actual issue in robotics/AI - "best" in each scenario might in turn become "worse" overall.

A funny story I read about once:

I've got a good story about the mines using this tech. A mate works for them and was telling me after changing over to driverless trucks they started getting huge divots in the ramps in the mines.
They went to site and watched the trucks to see what was going on, and the trucks were (with centimetre precision) changing gears in the exact same spot causing huge divots in the ground. They ended up having to program in a random number generator in that algorithm to avoid them.

Source: https://www.reddit.com/r/Futurology/comments/3p7qg6/driverle...

MiddleEndian• 11 years ago

The cars could easily have some sort of universal function to figure this out. Off the top of my head: The northmost car could go first, and if two cars are both "equally equally north" then the eastmost car could go.

Sir_Substance2• 11 years ago

The autonomous vehicle rounds a corner and detects a crosswalk full of children. It brakes, but your lane is unexpectedly full of sand from a recent rock slide. It can't get traction.

Why did an autonomous car with a map that tells it there's a crosswalk around a blind corner round the blind corner so fast it couldn't stop if there was unexpected debris on the road?

What a stupid scenario, autonomous cars have infinite patience, and when the people inside them can watch netflix all the way to work they won't care much about a few extra seconds either.

The obvious solution is to program the car to never take any blind corner, mapped crosswalk or not, so fast it can't stop on a dime. These things are going to have to work in winter in Canada. Low traction is a regular problem they'll have to solve. That'll mean every auto that takes this corner will do it very slowly, but that's fine because it can report back to the citywide network that this corner is unsafe, meaning all other autos will avoid it if they can (so no congestion), and as a plus it can be registered for modification with the local council. Maybe a networkable sensor can be erected, allowing the cars to see around the corner.

Taking corners faster than you can stop if there's a problem is a very human thing to do. One of the key benefits of autos is they won't do that.

tunesmith• 11 years ago

That's not really the point - knocking down that example doesn't knock down every possible scenario where uncomfortable questions come up.

We were talking about the Trolley Problem at a recent dinner with friends, and one of the guests remarked about how it was like Star Trek; "The Needs Of The Many" versus "The Prime Directive".

So it's a general argument about where a car has the capability of making an uncomfortable utilitarian choice (The Needs Of The Many), or of simply shutting off any ability to affect matters and let nature take its course (The Prime Directive). The uncomfortable bit there is that with the latter, there's still that actual choice (by the designers or whoever) to follow the Prime Directive - it's arguably not "natural" anymore.

Animats• 11 years ago

"The obvious solution is to program the car to never take any blind corner, mapped crosswalk or not, so fast it can't stop on a dime."

In general, that's right. There is no reason for the vehicle to out-drive its unobstructed sensor range.

One effect, which the Google cars already demonstrate, is that their behavior at corners with obstructed visibility is markedly more conservative than when the LIDAR can see down the cross street. That's a good thing.

Having once coded this for a DARPA Grand Challenge vehicle, that's how you look at the problem. You're always plotting stopping distance against the map constructed from sensor data. You figure out how far it's safe to advance given sensor data, and compute a top speed from that. We did this every 100ms. If the map wasn't filled in with new safe space into which it could advance, the vehicle would slow down and stop at the end of the safe area. So it never out-drove its sensor range. Remember, this was for off-road operations, with obstacles and cliffs; we had to be very conservative.

Humans tend to get into a mindset of not wanting to slow down. Software doesn't have that problem. Constantly having a plan available for an emergency stop is not a problem for software.

giancarlostoro• 11 years ago

I'm wondering why in that scenario the car doesn't honk the horn at all? Are all those children death and blind? No adults are supervising those kids either? I just don't understand. I hope the safety standardization of autonomous cars is a joint agreement that involves the companies involved, and the public, especially if a real scenario occurs...

RamenJunkie_2• 11 years ago

They can be deaf and blind if it helps push anti self driving agenda.

Even if a self driving car runs over a pack of school children daily, it will still be less fatalities than humans cause.

crististm5• 11 years ago

The AI used in these cars is probability based. They can't be held accountable in the sense that they can't explain their decisions rationally. You can't actually predict what you'll get by slight modifications of probabilities when adding new test cases.

Gerry Sussman explained this recently way better than I can.

adrianN• 11 years ago

Your brain is also probability based and you don't have an audit trail for your thoughts written to a black box.

mtgx2• 11 years ago

I think the auto/software makers should take responsibility for it, though, as some already have said they will. Just like when you have bad brakes in a car, it's the auto maker that is responsible not the supplier of that component. Also, they can't just say "it happens, what are you going to do?"

mtgx2• 11 years ago

I'm much more worried about self-driving cars getting hacked than "whether it will decide to kill me" in a scenario that is only going to get less likely as more and more self-driving vehicles are on the road.

ubersync• 11 years ago

I think it was more of an example of an imaginary emergency situation. You can hard program the corners, but you can't successfully predict all kinds of emergency situations that may arrive. Another example in that article is better: A rogue motorcyclist just turning blindly in front of you. Maybe, the only two choices are either you run them over, or steer your car off the road.

Sir_Substance2• 11 years ago

Why are you so close to the motorbike that if he stops, you'll collide with him?

If he's coming out of a blind sidestreet, why are you approaching so fast and with so little lateral room that you can't dodge a surprise motorcyclist?

These are all human patience related problems. An auto should be perpetually correcting to make sure that it's current minimum collision avoidance distance is smaller than it's current visual clear zone.

RamenJunkie_2• 11 years ago

This is my entire problem with this bull shit question.

It applies human limitations and distractions to a computer.

The scenario will never come.because the autonomous car sees everything with way greater clarity than any human. And it will never miss a person crossing the street because its rubbernecking at am accident or gawking at a hot girl or answering a test or trying to turn up the radio because its favorite song is on.

xbmcuser• 11 years ago

   There won't be a blind corner as the thousands of other autonomous Google cars passing by would already be napping and sending live data.
 Probably even some of the children crossing the street will have Google devices aw ding the data to Google and through to the car.

Eerie• 11 years ago

Nonsense. The car should behave in a way the law requires a human driver to behave. That's it, end of moral dilemma.

tedunangst3• 11 years ago

That's a dodge. Does the law require running over children or driving off a cliff?

The law distinguishes between deliberate actions and accidents in which there isn't time to decide. But a car will not have the luxury of a jury understanding that it panicked.

GrumpyBen• 11 years ago

The law says you should only leave your lane if it's safe to do so, no law requires you to sacrifice yourself.

"I panicked" is not an excuse, you need to control your vehicle at all time. You should never be in a situation where there isn't time to decide, you need to adjust your speed and safety distance for that.

solidsnack90002• 11 years ago

If we replace autonomous cars with taxi drivers, we have the same agency problem...but it's not a question people consider troubling.

A taxi rounds a blind corner on to an inner city railway crossing. Half a car's length across the rails is a small child who's fallen off their tricycle. A steaming locomotive is barreling down the track. If the driver breaks in time, your vehicle will stop short of the child but be torn to bits as its shuttles down the tracks in front of the train. If they maintain speed, you'll just make it past the train -- but the child is not likely to survive.

rogeryu• 11 years ago

There is one flaw in your reasoning: the taxi driver probably wishes to survive himself. Even if the front of the taxi, with the driver in it has passed the tracks, it will probably end badly for him as well.

crististm5• 11 years ago

I was very surprised to find that at least Google car is programmed top _not enter_ a railway crossing unless it has clearance all the way to the other side.

a_e_k• 11 years ago

Isn't that just standard rules of the road/defensive driving practice, though? Certainly that's how I handled it back when I lived near a train crossing.

solidsnack90002• 11 years ago

Where can I read more about this aspect of the Google car's programming?

taoufix• 11 years ago

TED-ED video about the subject : https://www.youtube.com/watch?v=ixIoDYVfKA0

nmc• 11 years ago

The thesis is disturbingly similar to the one detailed in: http://www.iflscience.com/technology/should-self-driving-car...

Almost feels like plagiarism.

huuu• 11 years ago

It all depends on the motives of the owner.

I think in general the cars will be programmed to avoid any damage.

But what if a company starts providing rides that promise no injury will happen to the passenger?

That's why laws are needed to prevent this from happening in the future.

Side note: in The Netherlands we have an automated sea barrier [1]. The system is made automatic so nobody has to make the decision to close it. So yeah, some systems are programmed to save peoples lives even if this means the lives of others can't be saved.

[1] https://en.wikipedia.org/wiki/Maeslantkering

fleitz• 11 years ago

The driver is presumably the owner of the vehicle therefore it should act in the interests of whom it is serving

danr4• 11 years ago

Can anybody who understand the market give me a reason as to why aren't smart roads built simultaneously to these driverless cars? I mean, why focus on making a car able to 'see' rather than being able to communicate with it's infrastructure?

The reasons I thought of:

1. As mentioned - building/upgrading infrastructure is probably much more expensive and logistically elaborate.

2. Market penetration - current batch of driverless cars can drive on existing infrastructure - no need to wait.

3. If smart cars won't be usable dumb roads it will also be hard to sell

4. Smart, durable, cost-effective infrastructure hasn't been solved.

HeyLaughingBoy• 11 years ago

Businesses build cars; governments build roads.

It's hard enough to get a government to keep roads pothole-free, how much harder is it going to be to get them to build "smart roads" when they have little or no incentive to do so?

datashovel• 11 years ago

I think the answer is obvious, but not because I have some moral reason behind it. I think it really comes down to the actors involved and what their current responsibilities are.

I can't imagine today's pedestrians assume a driver will attempt to preserve their life over the lives of the people in the car. So as a pedestrian you take measures to avoid putting the driver of the car into those situations.

I think it's reasonable to assume a driver's objective is self preservation and preservation of his/her passengers before preservation of people outside of the car.

I can certainly imagine a scenario where a person driving by themselves might make different decisions because they can't imagine living through an event where they were responsible for killing others, and didn't do everything they could to prevent harming others. But then what if there are passengers? I have a dilemma of attempting to preserve others' lives in the car and outside of the car. Are we honestly going to ask our drivers to "rate" each passenger? ie. If Joe is in the car with me, I could care less if he survives a horrible crash because he slept with my wife, so you can feel free to kill us both in extreme circumstances. If Mary is in the car with me by all means do everything you can to preserve her life. She has kids and a husband and is happily married.

The objective of the driver (and thus driverless vehicle) is to transport the driver and passengers safely. They shouldn't attempt to actively reduce likelihood of achieving that objective in order to preserve non-passengers.

kubiiii• 11 years ago

There ils also an issue with making different cars from different car makers cooperate efficiently. When an accident can't be avoided, involved cars would need to share info and processing power to behave optimaly and act as master and slaves.

brianmcconnell• 11 years ago

While we are exploring hypothetical scenarios, here's one.

A schoolbus full of young children has lost its brakes and is careening down a hill toward an almost certainly lethal high speed impact with a building at the bottom of the hill.

Two Google cars are idling at an intersection halfway down the hill. Both determine that if one of them pulls in front of the bus just so, it will result in a survivable crash (for the kids on the school bus). But which car should go?

Both cars run a PANN (psychometric artificial neural network) analysis of their owner's email. Car A determines that its owner is an abusive sociopath. Car B determines that its owner is a nice person, newly wed and possibly pregnant. The each compare their owner's PANN scores.

Car A signals "CARMA program activated. Taking one for the team!"

The schoolbus plows into Car A, which absorbs most of the impact. The kids are fine, minor injuries only, but the owner of Car A expires before the fire department can extract him with the jaws of life.

thoughtpalette• 11 years ago

This is the most interesting answer I've heard to this question so far. Will be using this when in conversation with friends.

bewatson• 11 years ago

Very interesting hypo. "CARMA" made me laugh. Another element to consider is if there are fleets of empty and unmaned cars on the road (eg an Uber), an empty car could take the hit to skip the single unnecessary death.

matobago2• 11 years ago

The main problem with this is we are thinking in autonomous car as they where super humans, so a human with endless amount of time will have to rely on ethics to make a decision, but not a driverless car.

A computer will rely in an algorithm, and most important than that in multiple sensors, not just their own sensors but also other autonomous cars sensors.

And there is a huge opportunity here to create even more sensors around crossways or sense people cellphones or bikes, or static speed sensors to warn other cars, creating tons of information for the car to know where cars, bikes and people are.

I see this as a remote scenario if we start building more sensors on top of the current highways infrastructure.

matobago2• 11 years ago

The car has to have the power to avoid being in a bad case scenario, it has more power to calculate the odds before even gets there. That is why this should be a DMV call not programers.

rdancer• 11 years ago

tl;dr: No.

pvaldes• 11 years ago

The problem here is that we are using a philosophical and humanist focus for a technologic problem. Machines do not need to be human.

Will your driverless car kill you so others may live?

Related Stories