Self-driving vehicles against human drivers: Equal safety is far from enough

https://pubmed.ncbi.nlm.nih.gov/32202821/

by Bologo • 6 years ago

147 445 6 years ago

pubmed.ncbi.nlm.nih.gov

Bostonian• 6 years ago

I think this related to the "illusion of control". People feel safer when they are driving, rather than a machine, even when they are not safer. I hope government regulators do not impose 5x safety requirements on self-driving cars.

VHRanger• 6 years ago

It's inertia rather.

People can drive tomorrow because people can drive today

macintux• 6 years ago

I think we also want someone to blame for an accident, and it’s not at all obvious who to blame when a self-driving vehicle is at fault.

philipov• 6 years ago

If corporations are people, you should be able to bring criminal murder and manslaughter charges against them, with the top-level executives acting as proxies to serve the jail sentence.

spaetzleesser2• 6 years ago

You may have to sue the manufacturer and prove that their system is at fault. Which is pretty much impossible considering the legal resources these big corporations have versus the little guy. This would end up like tobacco or junk food where companies were able (and still are) able to deflect any kind of responsibility.

ghaff9• 6 years ago

Assuming the system is properly maintained and used, if anyone's responsible it has to be the manufacturer. Certainly the passenger isn't any more than if an Uber gets in an accident today.

And, with the possible exception of drug side effect (and even there there are lawsuits), we don't really see consumer-facing products that, even if used as directed, kill a fair number of people and we just go oops. Let's say autonomous vehicles kill 3,000/year in the US, i.e. 10% of the current rate. (In reality, human-driven cars will take a long time to be phased out even when self-driving is available but go with the thought experiment.) Can you imagine any other product we accept killing thousands of people a year and we're fine with that?

ADDED: As someone else noted, you could argue that tobacco etc. fall into that category but we're mostly not OK with that and is reasonably thought of as in another category. (And pretty much no one is smoking because they think it's good for them.)

anonuser123456• 6 years ago

It may be easier to find fault in an autonomous vehicle. Assuming it has a black box that records sensor data, you can replay the algorithm and see what went wrong.

Analemma_• 6 years ago

The illusion of control is a thing, but actual control is a thing as well. One possible reason to avoid self-driving cars is that there actually are safe and unsafe drivers, and fatal accidents in self-driving cars will presumably be a much flatter distribution among those drivers than the one we have now. Which means that even if they're safer overall, they could still be less safe if you're a good driver.

mlthoughts2018• 6 years ago

If they are setting an objective measurement, how is that an illusion of control? In fact it seems like exactly the opposite - they are putting hard numbers on the level of risk they consider tolerable. They are making that available to everyone so they can debate and dispute it.

If anything, this is removing the illusion of control. The illusion of control would be to say you would never trust self-driving cars. Saying you will trust them at a level of 5x measurable safety criteria above human drivers is totally different.

Now we can make actuarial arguments about whether it should be 5x vs 2.6x vs 0.9x and debate how to measure the safety criteria - that’s a completely different world from one where people “feel like” human control of the car is safer.

dash25• 6 years ago

For sure it is good to seek a measurable criterion. The question is whether laboratory subjects' views on the right level should have normative force. An alternative take is: these are just not-very-informed people, and unless they can give reasons for their views, we shouldn't take them seriously as inputs into the policy-making process.

spaetzleesser2• 6 years ago

This seems pretty reasonable and also very possible to achieve. It would be insane to allow a technology on the streets that makes as many mistakes as humans are making. I certainly wouldn’t use self driving cars if they killed 30000 people per year like humans are doing right now. How would you assign responsibility for crashes? Our current system is far from perfect but at least it’s something people understand and know how to navigate. And there are drivers that are better and more cautious than others. So it’s not just an illusion of control.

sreekotay2• 6 years ago

Engendering trust and reducing materially regressive liability/litigiousness is a good call - and something that SHOULD be set as a standard by an external body.

IMHO this is typically a good role for government regulation - setting a standard measurement of outcome for the public good, but not dictacting HOW that should be achieved.

Now, we're just haggling over the price...

( as not-churchill infamously didn't say...)

dash25• 6 years ago

On the face of it, the delay in accepting self-driving cars till they are 5x safer would cost thousands of lives in the interim (while they are only twice as safe, three times as safe etc.) Is there a reason ordinary people's views should have prescriptive force here? Maybe they're just flat wrong.

msandford• 6 years ago

Arguing that people should do what you want, irrespective of what they themselves want leads to all kinds of pain, on all sides.

"Why are people voting against their own self-interest?" is an analogous phrase. It seems awfully condescending to me.

Nobody's bound to your perspective of what's rational. Better to just accept that this is the kind of hurdle that self-driving will have to jump over and work on getting there ASAP.

Elon realized that the best way to get people to buy electric cars was to make electric cars that are better than gas cars, not to tell people they're wrong and stupid for not wanting to buy some inferior electric car. Once self-driving cars are obviously better than all but the best race drivers, people will accept them as a matter of course.

jrockway• 6 years ago

Indeed. Why would 1.00000001x safer not be a no-brainer?

ianhorn4• 6 years ago

Why don’t we mandate that people submit themselves to a mandatory medical experimentation lottery? We’ll do so much better if we go through as many people as we do lab rats, and it’ll save unimaginable lives in the long run.

Utilitarianism via taking current lives to save future lives is the wrong perspective here.

amanaplanacanal5• 6 years ago

Ordinary people vote for the politicians who make the rules. How do you propose we get around their wishes?

sreekotay2• 6 years ago

Agree. 5X is probably too high... butttt... let's set the terms.

We can adjust the price as we move forward?

brighton36• 6 years ago

Doesn't this 5x requirement hurt more people than (say) 1.00001x? What am I missing here...

nextaccountic• 6 years ago

This is a psychological experiment. People aren't rational.

toolz• 6 years ago

I think this high level of certainty is basically just the governments way of acknowledging they are terrible at gathering/defining useful metrics and so with a wide margin there's very little room for error on the politicians part. I'm unsure if this is overly cynical, but I don't expect politicians today became career politicians by worrying about safety more than protecting their political status. Further, I suspect media would look for any definition possible to blame politicians for deaths so politicians feel it necessary to be blameless before allowing interesting, progressive ideas to materialize.

zebrafish• 6 years ago

I would say that we tend to reduce human flourishing to exclusively being alive. I think the 5x multiplier maybe covers things like loss of liability in an accident, a sense of ownership of the vehicle, loss of privacy or obscurity, regulatory or operational infrastructure costs associated with a switch to self-driving, freedom of choice, etc. All of these have some ultimate impact on human flourishing beyond just a binary dead or alive definition. My opinion is, if these aren’t included in the 5x, they should be.

notatoad• 6 years ago

This is not the government saying "we the government require...". It's the results of a study of what people believe. people's risk tolerance is almost never rooted in a rational calculation. Risk tolerance is based on emotion, and self-driving cars currently trigger an emotional response.

As soon as self-driving cars become a regular part of people's lives and not an exciting new thing, the calculation will shift to a much more rational one

Slartie2• 6 years ago

This calculation is actually very rational. What you seem to ignore is that, with conventional cars, there is a relatively small amount of "known unknown" risks. There are of course significant risks, but almost all of them are known not only in kind, but also in quantity. Drunken drivers, dumb people, broken brakes, whatever. We have several decades of data regarding these risks. The amount of "unknown unknowns" can also be assumed to be relatively low, given that the concept of humans driving cars has quite a history now and largely stayed the same for a good number of decades.

With autonomous cars, even once you have a few years of safety data from a large enough number of cars to be able to make the call of them being 5x less dangerous than human-driven alternatives in that data, you will still end up having much more "unknown unknowns" (of which I can't tell you any, because they are by design unknown) in addition to also having much more "known unknowns" like the possibility for large-scale software bugs potentially causing thousands of casualties at the same time. These risks will only go down slowly with enough time, there's practically no way of fast tracking getting these down, hence you have to incorporate a large enough risk buffer in your assumptions for rationalizing to even start using that fancy new tech, and the only place where this risk buffer can come from is having a much bigger difference in the "known knowns" department of risks.

Tuna-Fish• 6 years ago

You are correct.

However, how many people get hurt is not the only thing that needs to be considered. It's very likely that when a self-driving car would be equally safe as a human driver, the people who die in accidents caused by the car would be different than those who would die in accidents caused by human drivers, and so you'd end up with situations where individual next-of-kin could make entirely legitimate claims after accidents that their loved ones would be alive if not for the hellspawn car.

Trying to convince juries that it's alright because for every person who die in the cars, two other people who would have otherwise died got to live would probably be tough. Especially as the accidents that the self-driving cars are most apt to prevent are ones that could at least partially be considered caused directly by bad choices made by the driver (DUI, distracted driving, falling asleep at the wheel).

Once the data gets good enough that you don't need to do statistics on it[0], it becomes a lot easier to sell the idea to the public.

[0]: Relevant xkcd: https://xkcd.com/2400/

opportune2• 6 years ago

1.0001x is definitely not acceptable because dangerous drivers bring the total-human-safety metrics down. The "average" (median, or even maybe 25%ile) driver is probably much less likely to be in an accident (or fatal accident)than the drivers who drive most dangerously, e.g. frequently texting while driving or driving while intoxicated. So for most drivers, 1.0001x the average human rate would actually be worse than driving themselves, although they may find the risk acceptable.

rstarast• 6 years ago

How far do we think human drivers' safety level ranges? Like most drivers I'm falsely convinced I'm a safer driver than most, but still I expect quite a large range (say a factor 10 between 10th and 90th percentile?). It seems reasonable for self-driving cars to be expected to improve safety over human driving for the large majority of drivers, not just half of them.

dougmwne• 6 years ago

But how false is that impression that we are safer than average? I don't drive drunk, drugged, tired or distracted. I avoid driving in bad weather. I make sure I have good tires and brakes. I don't intentionally speed. I bet most accidents are caused by the above. I'm not interested in a self-driving car that drives like it's checking its cellphone after 3 beers.

rootusrootus5• 6 years ago

Exactly this. Some people think that Tesla's autopilot is just great, better than a human driver much of the time. As a Tesla owner, I am flabbergasted by that. At best, AP drives like I do. That is, perfectly straight and between the lines down a straight road. With any curves, and sometimes just on straight roads, it drives like a high-functioning alcoholic. I can't imagine how some people drive normally where they think that qualifies as 'good driving.' Some of us are very attentive drivers -- I never look at my cell phone, I never drink and drive, I don't drive when I'm tired, I avoid driving in inclement weather or at night unless strictly necessary, I am a very defensive driver. I don't get tickets, I don't get in wrecks, and this is by design -- I take great pains to reduce my exposure to these risks.

Personally I think the old joke about how 90% of drivers think they're better than average is both true, and also just a funny joke. We see a lot of perfectly good drivers on the road, but we don't notice them ... because they're perfectly good. There's only a few lanes on any given road, though, so if 10% of drivers in near proximity are crappy drivers then it practically shuts down the road. We notice that, and assume that most drivers are crap. Wrong.

the847218• 6 years ago

It seems reasonable for self-driving cars to be expected to improve safety over human driving for the large majority of drivers, not just half of them.

Depends, do you want to save lives? Then self-driving cars only need to be a little safer than the drivers they replace. Which means if infrequent drivers with little experience that get replaced by robotaxis of average reliability could be a net-win in saved lives. Delaying their deployment until technology arrives that beats the most conservative drivers just means accepting a higher death toll.

stkdump3• 6 years ago

if infrequent drivers with little experience that get replaced by robotaxis of average reliability could be a net-win in saved lives.

I don't know if frequent drivers are inherently safer drivers than infrequent drivers. There might be the negative effect of reduced attention due to more 'routine'.

But I seriously doubt that frequent drivers drive so much safer that they negate the effect of being exposed to the risk so much more. Is a person that drives 10x more than the average driver more than 10x safer? Why replace cars first, that don't get on the street a lot? And how do you organize deploying SDCs to infrequent drivers first? Unless those people don't own the cars anymore, but rent them, in which case I agree. That would increase utilizations of these cars.

amanaplanacanal5• 6 years ago

If you never drive after drinking or while sleepy, you probably are a safer driver than most.

rozab• 6 years ago

Would this baseline include all the accidents from distracted drivers, drunk drivers, drug drivers etc.? Or is it referring to an average human driver who isn't intentionally breaking the law?

If the baseline includes all these sorts of human error, I see no issue with holding robots to a higher standard. Imagine if we rolled out robot policemen who only executed black people for no reason at the same rate as humans do.

maxerickson• 6 years ago

Without meaning to comment on how possible it would be to carry out on a policy level, replacing the worst human drivers with robot drivers that match average human drivers should be an improvement for everyone.

It also potentially opens up policy options, or at least makes them easier to choose.

lftl• 6 years ago

Interesting idea. I think there may be feasible political routes to accomplishing that. Tighten up the points system that basically every state uses for deciding when to suspend your license, and simply force those who would normally lose their license into robot driving.

ksk6• 6 years ago

What does "worst" mean? Did they get into an accident once or twice, but drive just fine on otherwise? Do they drive bad every single day? etc etc.

aaaxyz• 6 years ago

Imagine if we rolled out robot policemen who only executed black people for no reason at the same rate as humans do.

Human cops still have to do the killing for now, but that's called predictive policing

superkuh3• 6 years ago

Behaving as a human would is often more important than staying strictly in line with absolute and relative positioning with a road.

The semi-permanent snow cover on many roads in 1/3 of the USA that lasts weeks if not months in duration. Humans driving on these snow covered roads form emergent lanes having little to do with absolute positioning or even relative positioning of the curb. They form lanes based on what other humans do.

Self-driving cars that depend on knowing absolutely where they are and relatively where they are simply don't and won't function. We need self-driving cars that can behave as a human will for that. And that is a long way off.

No autonomous car has shown it can handle these common situations. Until then self-driving cars should not be approved nationally and probably be restricted to the arid and warm states that do not have winter.

randmeerkat• 6 years ago

Tesla just dropped a snow driving beta: https://www.teslaoracle.com/2020/12/29/fsd-beta-8-tested-for...

rootusrootus5• 6 years ago

I think this is a good point. The best lesson my dad ever gave me back when I was learning how to drive was to 'be predictable.' People don't get in wrecks when everyone behaves as expected. And the rules of the road are largely aimed at guiding that predictability. But in the end, regardless of the written rules, humans behave as humans and a robot driver should behave like other drivers. And it may change based on locality.

cmrdporcupine4• 6 years ago

Yes, exactly, this is always the same example I give (in my case the 401 here in Ontario, Canada) -- blizzard in the middle of February, lane markings covered, highly unpredictable road surface, spontaneous temporary lanes, cars working at a crawl, snow plows coming through that you have to move over for, and can't pass, cars or trucks jack-knifed or half in the ditch. This kind of thing happens to varying degrees at least once a year, and I honestly don't think that these scenarios are actually properly in the imagination of the primarily-California-based engineers who work on self-driving.

For context, the greater Toronto region is 6 million people, and Great Lakes region from here over to Chicago is multiples of that. Winter is 4-6 months. This is not an insignificant edge case for a small population, and if self-driving can't handle it, no thanks from this driver.

albntomat03• 6 years ago

Does self-driving have to handle all weather conditions right away? A sensible implementation needs to take the current conditions into account, such as the weather and the status of the road and car. If those are bad, it would refuse to active itself, similarly to how a responsible human would choose to not drive in bad conditions.

nelsonenzo• 6 years ago

Sadly, the news around facial recognition and AI seems to imply the government has been happy to roll out exactly that.

I guess when the government contractors can profit off self-driving murders we will be good to go. /s

ianhorn4• 6 years ago

I think I read that ninety something perfect of driving deaths or accidents are people being irresponsible in the manner you said, but can’t find a source again, so take it with a grain of salt.

I was able to find one that of the 37k driving deaths in 2016, 10-11k involved BAC over .08 and about the same involved speeding. Not knowing the overlap, 10-22k/30k is 27-59% of deaths involving drunk driving or speeding.

If it was on the high end of that, then to do better than speeding and/or drinking alone, you have to be at least 2.5x safer than human.

I wish there were better stats on the safety of the sort of driver you would let drive you around (e.g. you wouldn’t get in the car with your drunk friend behind the wheel).

kube-system• 6 years ago

Exactly. For the benefit of passengers, self driving cars should drive as well as a sober, attentive, well-behaving driver.

When I get into a car as a passenger today, I already get the benefit of riding with a better than average driver — I can identify and choose not to ride with people who are drunk, inattentive, reckless, etc. That’s the comparison that is a more reasonable baseline expectation.

vincentmarle• 6 years ago

I find it weird that nobody seems to talk about the national security implications of self-driving cars. Imagine the Russian cyber attack we just experienced happening on millions of self-driving cars...

xmodem• 6 years ago

To be fair with the amount of computer control and always-online features in current cars, this is already a huge risk.

boomboomsubban3• 6 years ago

Why would a state actor do this? It's a clear declaration of war, and probably wouldn't kill more people than suddenly launching missiles. Even if they could somewhat hide it, the US is prone to retaliate on fairly flimsy pretexts.

There may be some terrorism risk, but any truly terrifying scenario requires a multitude of incredibly stupid design choices. Like constant internet connection, remote updates, no manual override, and a single widespread system. Hacking stoplights is roughly as scary.

tgv6• 6 years ago

It will cripple a country and its economy.

errantspark• 6 years ago

This is already a risk with normal cars.

https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...

jedberg• 6 years ago

This title is awful (but it was copied from the site). What it should say is "Study finds that most people surveyed didn't trust self driving cars until they were five times safer".

m4632• 6 years ago

More like "people who have never owned or been in a self driving car..." faster horses.

That said, it will happen. I just wonder what will happen as self driving car safety exceeds human drivers. Will people be prohibited/disincentivized to drive?

rootusrootus5• 6 years ago

what will happen as self driving car safety exceeds human drivers

Average human drivers, or are you including the drunks and other bozos who cause the lion's share of fatalities?

happytoexplain2• 6 years ago

The comparison between all human drivers and all autonomous vehicles is far more complex than "whichever is statistically safer, as a whole". Belittling people who feel differently than you about it muddies the conversation for no reason.

dooglius• 6 years ago

Based on the abstract, it looks like this is an attempt to measure how safe self-driving cars need to be in order for people to prefer using them. It is not any sort of requirement from the NIH.

happytoexplain2• 6 years ago

I too was confused. I thought "we" referred to the NIH. It should say "people".

Jabbles2• 6 years ago

I wonder what range in safety we tolerate in human drivers? How much worse than the average is a newly-licenced 17 year-old (or whatever age) or an 80 year-old?

rootusrootus5• 6 years ago

Well, in terms of overall crashes and injury crashes, you don't get safer than the 80 year old until you're 30 [1]. Though the rate of fatal wrecks is about the same between 16-17 and 80+. I think that may be due in large part to the fact that 80 year old humans are far more fragile and likely to die in a wreck where a younger person would walk away from.

[1] https://aaafoundation.org/rates-motor-vehicle-crashes-injuri...

Jabbles2• 6 years ago

Very interesting, thank you for the data.

11thEarlOfMar• 6 years ago

If it's a subjective matter tied to perception of risk rather than actual, statistical risk, such perception can be swayed.

The challenge remains that people will be killed in accidents involving autonomous control. And we anticipate that the number of people killed will be fewer, hence 'saving lives'. However, the lives lost in autonomous accidents will be a different set of people than those that would have died in human driven accidents. There will be cases where a court determines that the autonomous system was the cause. Families of those killed will want justice, while those separately saved by autonomous systems may never be heard from in the same case.

I expect that in the end it will come down to a business decision, and that decision will be informed by an actuarial exercise: Will profits and insurance be able to cover the costs of defending and settling such cases. Who knows, maybe the threshold is crossed at 5x safer.

Justsignedup• 6 years ago

Sensationalism will win out quite a bit. And responsibility. Its just a tough problem.

- People developing Bell's Palsy at the same rate with vaccine than not. But suddenly only those with the vaccine show up in the social media feeds. Because nobody just posts "I just got bell's palsy" but now they do because people are paying attention. Same will happen with AVs. "I just got into an AV accident" will make headlines, while "I just dozed off and hit a kid" will barely circulate.

- People inherently trust humans over technology. Just because. So they will be quick to distrust autonomous vehicles. I already had convos about the fact that yes, teslas do kill, but on the whole self-driving teslas do kill far less than non.

- When a human drives, the liability is on the human. When a car self-drives, the liability might be on the manufacturer.

SkyBelow• 6 years ago

When a human drives, the liability is on the human. When a car self-drives, the liability might be on the manufacturer.

I think this is a point that needs more emphasis, especially on the word 'might'. Without both laws and a history of court cases giving evidence for how those laws are interpreted and enforced, it isn't possible to tell where liability may end up. I wonder if that is part of the reason people are hesitant. Liability is being removed from the driver, but it doesn't see have have found a place to settle back down at, so people are viewing it as if liability is just being removed. For initial court cases (and the amount of time and money it takes to fight them), this may not be an unrealistic expectation.

cj• 6 years ago

People inherently trust humans over technology. Just because.

From the perspective of someone who rides a motorcycle, the #1 thing you need to do to not crash is to anticipate what all vehicles around you might possibly do.

For example, I always avoid riding in another car’s blind spot for obvious reasons.

The problem (for motorcyclists) will now be trying to adapt to understand what a Tesla might do and where a Tesla’s blind spots might be - and once you add in the idiosyncrasies of other AVs I could see it being really difficult to ride safely around AVs.

It’s fairly easy to anticipate actions of another human, and not as easy to anticipate when actions are decided by an algorithm.

FWIW I think the above also applies to cyclists.

(I suppose this becomes a non-issue if the assumption is that AVs will be so superior to human judgement as to never strike another motorcycle or cyclist - 5x safer sounds like a starting point)

frenchy• 6 years ago

People inherently trust humans over technology

I don't think that's generally true, or at least, the contrary notion is also true some times. I'm pretty sure that if someone asked me the product of 13 x 6, they would trust me more if I punched some numbers in to a calculator and gave them a result versus if I just did it in my head. I don't know, but I think the likelyhood of me mistyping numbers is embarrasingly high, and probably about as likely as a mistake in easy mental math.

It's also closely linked to your third point though. Liability with self-driving cars is difficult. When people talk about self-driving cars, they sort of just hand-wave away the fact that there will be accidents, so as to avoid this difficult problem. This does not instill confidence.

otabdeveloper4• 6 years ago

Just because.

Maybe they actually read those EULAs and privacy policies and made an informed choice.

I mean, I wouldn't want Facebook making a self-driving car, and it's not because I doubt their machine learning chops.

wffurr5• 6 years ago

> The challenge remains that people will be killed in accidents involving autonomous control.

While that's almost certainly true (and depending on one's interpretation of autonomous is already true), some people already believe that zero deaths in transport is possible: https://visionzeronetwork.org/about/what-is-vision-zero/. If it's possible to hold human drivers to that standard, why not autonomous systems as well?

bryanlarsen7• 6 years ago

That was my original hope for self-driving cars. The easiest way to ensure that you don't kill anybody is to limit your speed to 20mph. At and below that speed a car-pedestrian collision is highly unlikely to result in a dead pedestrian. Also, at 20mph you can stop on a dime. So I imagined a large fleet of robot cars traveling at 20mph and normalizing that speed, forcing human drivers to slow down too.

But it turns out that self-driving vendors spend a lot of effort on "driving like a human", which includes driving faster than the speed limit and faster than vision zero would allow on shared streets & roads.

riversflow2• 6 years ago

Well that’s a silly vision. I have a vision of immortality too. This is a textbook example perfect being the enemy of good. I don’t want to hold any system to the standard of zero fatalities, and further I think this is why “shoot for the moon and even if you miss you’ll land amongst the stars” is faulty. People waste an exorbitant amount of time and fossil fuels commuting in passenger vehicles. Anything that significantly changes that balance should be considered. Progress is progress and letting everybody use the time they used to spend commuting is certainly progress.

porknubbins• 6 years ago

Somehow the idea of a computer error killing me seems way worse than at least having a chance to save myself, since I’m a very cautious driver (though unlikely safer than average by 5x) . Self driving cars need to get to airline level safety where crashes are a rare thing and most people don’t think twice about giving up control to the pilots/auto pilot. If that takes expensive Lidar thats what we should use. I can’t imagine ever feeling good about trusting my life to a computer vision algorithm.

sliken11• 6 years ago

Problem is, 95% [1] of drivers think they are better than average.

[1] citation needed.

matt-attack• 6 years ago

I feel the opposite. The one sensor that is guaranteed to have sufficient information to drive in all conditions is vision. That's obviously because humans drive exclusively with vision (and a single vantage point to boot - modulo mirrors).

mankyd• 6 years ago

While I don't disagree, I will note that most people don't think twice about giving up control to bus and taxi drivers.*

I think we trust humans to make a reasonable decision in a trolley-problem scenario (rightly or wrongly). Or rather, we trust the human we're in a vehicle with to value their own life, and thus our own, more than those outside of the vehicle in most scenarios.

I expect there is research to investigate this, though I wouldn't know where to begin looking.

*I've definitely had a few bad drivers, of course.

colechristensen• 6 years ago

A computer error is going to be scarier by far.

When a human kills someone with a car it is almost always in a way we can empathize. When you’re around cars you have pretty good mental models for the humans that drive them and how the car will behave.

When you’re around human piloted cars you can look at the driver and get a pretty good idea of intent. You can tell if the driver sees you, you can tell what their mental state is, if they’re paying attention, what they intend to do. You can sum up a person with a glance, this is the power of evolution, we’re really good at figuring things out about other living things.

Crossing the street in front of a car is a leap of faith, not troubling at all when there’s a human there, but a robot? There’s no body posture, no gestures, no facial expressions, nothing to go on. There’s a computer in control of a powerful heavy machine that you’re just expected to trust.

Robot cars don’t make human mistakes, they make alien mistakes like running down kids on the sidewalk in broad daylight, things which don’t make any sense at all that make people feel like they aren’t anywhere safe.

It won’t take but a couple of cute kids killed in a surprising matter to shut down the whole autonomous experiment.

thesuitonym• 6 years ago

As cautious as you are, you still get distracted, focus on the wrong thing sometimes, and have blind spots. Computers don't.

toper-centage• 6 years ago

Good point. But it doesn't matter how carefully you drive if the road is full of idiots and intoxicated drivers.

bombcar• 6 years ago

A similar calculus was what lead me to get a private pilot’s license instead of a motorbike - general aviation fatalities are something like 90% pilot error but 60% of motorbike fatalities are NOT the fault of the rider.

njarboe2• 6 years ago

I can't think of any product that has been developed since 1970 that can kill people. The exceptions are medical devices and pharmaceuticals. I sure hope self-driving cars can be an exception, but that will definitely take a federal law limiting the liability of manufactures. Similar to how small aircraft manufacturers were being pushed to extinction due to very high liability costs until the passage of the General Aviation Revitalization Act in 1994.

ghaff9• 6 years ago

Lots of products can (and do) kill people. But drug side effects aside, it's hard to think of modern consumer products that, used and maintained properly, might just go and kill you some day and people being OK with that.

drjasonharrison• 6 years ago

Boeing 737 Max? Ikea Malm dressers? Many products listed at https://www.cpsc.gov/Recalls

yadaeno2• 6 years ago

Cars?

notriddle• 6 years ago

Li-Ion batteries can explode.

tgv6• 6 years ago

Idk about risk: that is hard to establish. There are so many conditions in which an automatic pilot hasn't been tested. We don't even know the factors involved in estimating the risk: e.g., is it dependent on the human co-pilot? And self-driving cars may change the car usage patterns, exerting a contextual influence on the risk.

Then there's the question of responsibility. Who will be held responsible when the automatic pilot is driving? If it's the human, then a high risk of causing an accident will be unacceptable to many drivers.

foobarian• 6 years ago

However, the lives lost in autonomous accidents will be a different set of people than those that would have died in human driven accidents.

This is a really good point. I drive really conservatively and like to think I will never ever cause an accident let alone a fatal one. I think if this lever was taken away I would have a hard time accepting the automated driving a significant amount of time.

throwaway2245• 6 years ago

There will be cases where a court determines that the autonomous system was the cause.

At the moment, manufacturers almost totally escape blame for fatal accidents [involving human drivers] - it's understood societally and in the legal system that the human driver was the one at fault.

That isn't a totally accurate picture of the responsibility. The manufacturer provided a vehicle that included a risk of fatal accident. (Reducing this wrong and describing it as 'lives saved' feels uncomfortable to me)

With an autonomous system, blame for fatalities can no longer be placed on a human driver: and yet, there is still a failure of responsibility (maybe this will be a more accurate placement of blame)

CuriousPerson23• 6 years ago

Hasn't there been only 1 fatal accident? If not, I can't imagine there have been >5, so seems unfair and misleading to make bold claims like that. I think the courts will rule, and if the system misunderstood something, the manufacturer will be at fault.

SoSoRoCoCo2• 6 years ago

I expect that in the end it will come down to a business decision, and that decision will be informed by an actuarial exercise: Will profits and insurance be able to cover the costs of defending and settling such cases.

I'm seeing type of phrasing occur more and more. Once the defendant can be named in a legal action, we'll start seeing SDVs. IMHO, the worry isn't that they will kill, but that no one is to blame.

Although it will change the day to day narrative of a pedestrian. E.g., My thought process will change from this person might not see me, to, that car's AI might not see me. ... or even "Oh, its a Toyota, they kill more than Hyundai... stand back!" But now I'm just writing SciFi.

dvfjsdhgfv• 6 years ago

But now I'm just writing SciFi.

I don't think so. Normally whenever I want to cross the street (as a pedestrian, but even more as a cyclist) and a car approaches I (unconsciously) examine its speed, and if it's higher than acceptable I try to make eye contact with the driver to make sure they see me and it's safe for me to go. How do I make contact with the AI of the car? More importantly, how do I get the cue I've been noticed?

HALtheWise2• 6 years ago

It strikes me that a useful analogy here is the adoption of automatic elevators in buildings. In some ways, it's amazing that pretty much everyone in industrialized countries is OK with being locked in a windowless box controlled entirely by a computer, hanging over a hundreds of foot deep shaft, and in fact many people were terrified of that when Elevator operators were first replaced with computers. Some places even had operators employed to simply stand there and push the buttons to provide confidence that an trained expert was there, even though they didn't actually contribute to safety. Eventually, autonomous elevators got common enough that people will look at you really funny if you're not willing to ride in one, even though they are still responsible for ~20-30 deaths per year.

dekervin• 6 years ago

there is some kind of implied machine capability that a layman assign to a computer. If a task is culturally ( through movies , series, ... ) though to be within that implied capability people will be comfortable. ( cf automatic trains, ... )

ggreer2• 6 years ago

Can you provide more info about people being terrified of automatic elevators? I searched around and everything I found seems to cite one NPR article from 2015.[1] The interviewee's book is out of print and costs $100[2], so that's where that trail stops. If public sentiment against automatic elevators was as strong as described, it seems like there would be more historical evidence available. It's easier for me to find articles disparaging self-checkout systems than for automated elevators. I realize the change in elevators happened long ago, making articles harder to find, but you'd think at least one of them would have gotten digitized and indexed.

1. https://www.npr.org/2015/07/31/427990392/remembering-when-dr...

2. https://www.amazon.com/Ascending-Rooms-Express-Elevators-Pas...

RcouF1uZ4gsC2• 6 years ago

The difference with elevators is that the safety systems are actually in large parts independent of the control. Since the Otis safety elevator, you could go so far as to cut the elevator cord and it would still be ok.

With self-driving cars, you don't have those type of backup safety systems.

Guthur• 6 years ago

The analogy with elevators largely doesn't work. In the main the only individuals at risk from an elevator accident are those that chose to enter the elevator, this is vastly different for automotive accidents where the individuals affecting by the autonomous car are not those that chose to use one.

ntsplnkv28• 6 years ago

I don't think a self driving car that has to make rapid decisions in quickly changing circumstances with tons of moving parts outside of its control is comparable to a specific use case with much fewer variables like an elevator.

kaba012• 6 years ago

I know it’s an analogy, but elevators are dead-simple concepts with many fail-safety mechanisms independent of the controlling ‘program’. Self-driving cars are ridiculously complicated black-box program with at most software fail-safes that are itself more complicated than one can reason about so the only thing remaining is empirical testing in real-world situations. And every single bugfix have to be retested substantially as it can just as easily make something else worse - and I don’t feel confident about today’s “self-driving” cars

xenocyon• 6 years ago

However, the lives lost in autonomous accidents will be a different set of people than those that would have died in human driven accidents.

So far it seems that this is very much the case. Autonomous cars do relatively well in highway scenarios whereas they appear to do poorly recognizing bicycles, for instance. Reducing safety to one single metric would be a big mistake.

BurningFrog3• 6 years ago

Obviously the car manufacturer will be liable when a fault in their product kills people.

And that is as it should be. The costs will be baked into the price of the vehicle. This aligns incentives. The manufacturer wants to pay as little as possible, and the owner will want to be safe.

With the car recording video and other data, determining what actually happened should be simple and reliable.

Note that the car owner no longer pays any insurance in this system.

foobar19622• 6 years ago

Smart manufacturers will cover this risk with third-party insurers, for some period of time: probably the vehicle warranty period. After that, it will be on the owner to renew.

If I was one of these insurers, I'd be insisting the vehicles were maintained to an extremely high level of roadworthiness by mechanics that licensed or certified. Home DIY would be forbidden, as would any modification to the hardware or software. Think something like happens now for aircraft.

ysavir• 6 years ago

Obviously the car manufacturer will be liable when a fault in their product kills people

But how will that be proven? We can't exactly pop the lid off the driving algorithm and say "ahh, there's the problem. Time to take it to court".

paulryanrogers3• 6 years ago

My guess is it's much easier for the public to understand the risks and liability of human drivers than AI.

Humans often get distracted or misinterpret signals. And their errors stop with the driver. AI could react in more unexpected ways when it goes sideways. Humans too can get weird when chemically impaired but the liability is much easier to attribute.

jonplackett• 6 years ago

To add to the ‘different set of accidents’ point - the accidents caused by autonomous cars will most likely also be harder to accept. They will be accidents that a human driver (at least with the benefit of hindsight) would appear unlikely to have caused.

Things like that Tesla driving straight into a wall and killing its driver. Or not seeing a van right in front of it because the sun was too bright. Things where a bit of common sense (not something AI excels at) might have avoided them.

On the flip side the lives saved will be based on having super human reaction times / situational awareness and will be things no person would have been able to do.

So maybe there’ll be a battle of public opinion (and PR!) weighing these things against each other.

semi-extrinsic• 6 years ago

No, if there are lives saved by self-driving cars, they will primarily be in instances where the driver was distracted/drunk/high. Thus any competent lawyer workin a case against a self-driving car company in the case where their car killed someone, will argue (rightly!) that those sane lives could also have been saved by less high-tech measures, such as AI detecting undesired human behaviour and putting the car in limp-home-mode or similar.

In that case you are saving the same lives and not causing any new fatalities, but it's not a super cool tech that will attract lots of investment. So the choice to implement self driving was ultimately putting profits over human lives.

Retric• 6 years ago

I think you’re going to find human caused deaths to be just as bizarre if you ever look into them. You might think people are generally ok drivers but when you’re looking at each fatality as the worst mistake per 1.5+ Million hours driven people are doing some truly dumb things.

On that time scale you can expect people to have fallen asleep at the wheel several times. They spent literally months looking at their phones rather than paying attention to the road etc. Because here’s the thing most accidents aren’t fatal, for every death you’re going to see a host of accidents that should have been avoided and tens if not hundreds of thousands of mistakes that could have resulted in a serious accident but didn’t.

troupe3• 6 years ago

It isn't just a perception issue. It is going to depend on how we assign blame. Today if a human causes an accident that kills someone, the family can sue the human and get some from their insurance company, but most insurance policies have a relatively low limit on what they will pay. As far as the individual goes, the median net worth of someone in the US is less than $100,000. You can sue, but there really isn't that much money to get.

However, with self-driving vehicles, if you sue the manufacture of the vehicle, they have significantly deeper pockets. This is going to effectively keep us stuck in the place where no matter how good self driving cars become, the responsibility remains with the driver.

Robotbeat• 6 years ago

There's actually a potential safety advantage from the public's irrational reaction to autonomous driving deaths (irrational if compared to their response to human driving driving deaths):

Autonomous driving deaths will be much more scrutinized. In the press, by regulators, by competitors, by the companies involved. There's likely to be a lot more data involved. The scrutiny may ironically increase as it becomes safer, maybe becoming a lot like plane crashes, in fact, where each one is scrutinized by groups of experts and fixes are put in place to prevent that kind of thing from happening.

The safer it is, the rarer and more novel the fatalities are and therefore the more resources will be spent on fixing the safety issues.

That is, if autonomous driving is allowed. If it's not allowed because overall irrational fear of autonomous driving, then that feedback cycle can't happen.

So the paranoia isn't necessarily a bad thing as long as it doesn't cause paralysis.

schoen• 6 years ago

we anticipate that the number of people killed will be fewer, hence 'saving lives'. However, the lives lost in autonomous accidents will be a different set of people than those that would have died in human driven accidents. There will be cases where a court determines that the autonomous system was the cause. Families of those killed will want justice, while those separately saved by autonomous systems may never be heard from in the same case.

This reasoning was addressed with the VICP program for vaccines in the U.S., based on the idea that vaccines save many lives overall, but still injure a small number of people who will want to be compensated for that injury.

https://www.hrsa.gov/vaccine-compensation/index.html

Maybe there should be a self-driving-car injury compensation program along similar lines once particular cars are convincingly proven to be even moderately safer than human drivers. People might be mad about it, but maybe the precedent of the apparent success of the VICP would be persuasive (at least for courts and legislators, and maybe for some people actually injured by self-driving cars).

A counterargument is that vaccine injuries are generally unforeseeable and reflect absolutely no fault on the part of the injured person (or injured person's guardian), while some people injured by self-driving cars will bear some fault for their own injuries, which insurers or manufacturers might well want the opportunity to adjudicate. (Otherwise, there could be a perverse incentive to deliberately do dangerous things around self-driving cars to provoke an injury and receive a payment, something I've heard already sometimes happens with personal injury claims against human drivers in some parts of the world.)

throwawaygh• 6 years ago

Will profits and insurance be able to cover the costs of defending and settling such cases.

Risk will be transferred to individual consumers through "pay attention / hands on wheels / you're still the driver" warnings. Those warnings will never go away, even with vehicles that are in all other ways marketed as L5.

kaba012• 6 years ago

And frankly, it should not be legal. Either don’t sell it as self-driving / don’t even have that feature, or take full responsibility (within normal limits) for any accident.

ehnto• 6 years ago

However, the lives lost in autonomous accidents will be a different set of people than those that would have died in human driven accidents.

That's an interesting insight into part of the problem, thanks for pointing it out.

Indeed in America if you ride a motorcycle but don't ride drunk, don't ride unlicensed, don't ride an unregistered bike, and do wear a helmet, you are orders of magnitude less likely to die because you are in a statistcally safer demographic of riders. Yet, any one of the riders in that safer demographic could still be one of the ones hit by a red light runner and killed without their blame. We would definitely feel that death more tragic and senseless.

sriku• 6 years ago

I'm wondering whether the first steps ought to be to make current driving much safer through automation - like detecting drunk driving, falling asleep, and such and limiting speed, refusing to start, alerting police and/or nearby vehicles etc. The lessons from that might better inform the long tail of cases that autonomous vehicles will eventually face?

visarga• 6 years ago

Yeah, people won't buy 'crippled' car brands. Probably no manufacturer will dare be first.

jhpriestley• 6 years ago

this is quite an academic exercise since a decade of intensive research hasn't brought us close to working self-driving cars, much less 1x safe self-driving cars, much less 5x safe, nor is there any clear path to resolving this open research problem.

c1505• 6 years ago

That might be their current stated preference, but I don't think it will be most people's actual choices. Imagine if self driving was available on every car right now with the press of a button and it was as safe or twice as safe as a normal driver. How many people would press that button, start texting, and just continuing to progress to paying less and less attention ? People already don't pay the attention they should when driving or when using a driver assistant system.

franklampard• 6 years ago

On the contrary, you can argue that button saves lives by doing a better job than the reckless drivers who aren’t going to pay attention in the first place.

loeg• 6 years ago

Yeah, not to mention value of time. If I could hop in the car, at the same level of safety as my own driving, and spend the 1.5 hours to trailheads reading a book or even programming, I'd much rather be doing that than paying attention to the road.

darksaints• 6 years ago

What so many autonomous car advocates seem to miss is that it is nearly impossible to meaningfully compare relative safety with current self driving cars, because we don't have level 5 autonomy yet.

In order to compare them with current technology, you'd have to be able to answer the question: how safe would human drivers actually be if they didn't have to perform their most difficult tasks? Because that is what current autonomy does.

I'm willing to believe that current tech is capable of being safer than human drivers, simply because they do so many things way better than humans do, like stopping for pedestrians and safely navigating around cyclists. But to compare them in general, that is left to be proven. You can't just compare incidents per mile driven, because autonomous vehicles can conveniently opt out of driving whenever the task gets too hard.

Bedon292• 6 years ago

It definitely tends to be the already safer driving, like highways, that they do well on. I have a Model 3, and trust it pretty well on highways. However it does not do turns at all, and other 'city driving' type tasks well. It can now do stop signs and traffic lights, which seems to be good so far too.

However, not living in an area with many sidewalks, I do not trust it for one second to navigate around pedestrians or bicycles. I don't think it will actually try to go around a bike but, I have never given it the opportunity to either. I take full control back and give them a very wide berth myself.

jjk1664• 6 years ago

You can't just compare incidents per mile driven, because autonomous vehicles can conveniently opt out of driving whenever the task gets too hard.

But isn't that kind of the point? We use autonomous driving for the tasks where autonomy is objectively better, and we have the human do what the human is still better at. Best of both worlds.

bradlys2• 6 years ago

But that's not the goal of autonomous driving or the goal of most autonomous driving advocates. The goal is that you never need to know how to drive. It's essentially a taxi. Most people I know who are waiting for the autonomous driving future are waiting for one where they never have to drive.

I imagine now: "Sorry, can't drive through this thick falling snow. Must pull over and you must operate now, human. We are 100 miles from civilization in any direction and no cellular service is available, best of luck!" I cannot imagine that most future drivers who are minimally driving in their daily life will be able to suddenly handle conditions that the "autonomous" vehicle cannot.

anoyesnonymous• 6 years ago

This should be calibrated to the risk the top X% of cautious/safe drivers, and exclude reckless, inexperienced, or intoxicated drivers. As a safe driver, you shouldn't have to accept risk calibrated to "average" (i.e. drunk, reckless) driver.

Isinlor• 6 years ago

Nobody will be forcing you to buy self-driving car for quite a while. But as a safe driver, you should care about eliminating the most unsafe drivers from the roads.

falcolas• 6 years ago

As a self-identified safe driver, no action I am capable of taking will put an unsafe driver in a self-driving car.

I'll even go so far to say that many unsafe drivers can't afford a self-driving car. They're often unsafe because their car is on balding tires, the brakes don't work, and the tail lights are busted.

The rest, well, they simply enjoy driving unsafely and thus have no reason to get into a self-driving car.

tgv6• 6 years ago

Exactly. Because now we can punish those individual drivers, lock them up, take away their car and license, but are we going to pull the plug on all cars with auto-pilot X because X is causing accidents? Is a small change in the software enough to establish it as a new driver? It's "smoking is good for you" all over again.

the847218• 6 years ago

Why should it be? In the end the dead bodies count and it doesn't matter whether a cautious or inexperienced driver killed them. Inexperienced drivers are prerequisite of experienced drivers, there's no way to get rid of them. Excluding them from statistics is just discounting those deaths as... somehow less important?

If a self-driving vehicle is only 1.5x (instead of 5x) as safe as the average human driver then you're not trading between death by humans vs. death by machine, you're primarily trading between death by human and spared by machine and only secondly between the former.

degrews2• 6 years ago

In the end the dead bodies count and it doesn't matter whether a cautious or inexperienced driver killed them

It matters to the safe drivers. Bad drivers are mostly a danger to themselves. At only "1.5x as safe as average", it's a good deal for the bad drivers, but there are probably a lot of "2x as safe as average" drivers that are getting a bad deal. They are in more danger than before.

Edited for clarity.

jjj12322• 6 years ago

This person is saying that on an individual level, they are not willing to cede control to an “average” AI when they know (or believe) themselves to be above average.

You’re talking about it at a societal level, as if everyone switched over to robot cars at the same time.

ianhorn4• 6 years ago

Let’s say you hire a chauffeur to drive your kids around. You find out they’ve been drinking on the job and speeding recklessly. When you confront them, they pull out stats that they’ve been actually less drunk than average. Do you fire them and find a new chauffeur?

When it’s a robot chauffeur, you have to evaluate it like you would a human one.

908B64B197• 6 years ago

I've questioned the lack of driving experience as a risk factor since the pool of experienced driver excludes those who died becoming experienced.

Assuming someone has a certain (constant) probability of excluding himself from the driving pool every year, over time the average percentage will drop as folks most susceptible from excluding themselves will have already done so.

dunefox3• 6 years ago

No, on the contrary: deaths through those drivers can be eliminated. It only makes sense to look at total number of deaths, including by alcohol, drugs, inexperienced drivers, elderly drivers, distracted drivers (smartphone, etc.).

grecy• 6 years ago

As a safe driver, you shouldn't have to accept risk calibrated to "average" (i.e. drunk, reckless) driver.

But you already do. Every day you're near a road, there is chance the next vehicle around the bend is drunk or reckless or using their phone.

It doesn't even matter if you are in a vehicle or not - even as a pedestrian you already deal with them every day.

It sucks, but it's reality.

JohnHaugeland• 6 years ago

"We arbitrarily chose a number so we could feel like we were making improvements. Nothing justifies 5 over, say, 3, or 10. When cars are in fact 3x safer, all those saved lives won't be saved, because our arbitrary 5 has yet to be reached."

This is meaningless and bad.

bumby• 6 years ago

Algorithm aversion is real and shows we prefer humans even in the face of statistical evidence that humans are sub-optimal decision makers. [1]

I suspect it’s because we inherently dislike the idea of handing control over to a complex black box. Barring sociopaths, we can reasonably assume to interpret how a person thinks. This isn’t necessarily the case for algorithms, which leads to trust issues.

[1] https://repository.upenn.edu/cgi/viewcontent.cgi?article=139...

Scandiravian• 6 years ago

So to speed up the adoption of self-driving cars, we could simply make human-driven cars more prone to accidents :p

xibalba• 6 years ago

This is just the sort of outside-the-box thinking for which I read HN.

Causality12• 6 years ago

Is 5x safer a realistic goal? There are limits to how safe a car can be on a road full of human drivers, no matter what sensor suite it has and how fast its reactions are. A vehicle can only respond so quickly to control inputs. Making a computer that's five times as safe as a human might be a thousand times more difficult than making one twice as safe.

sliken11• 6 years ago

Depends how you count. Being is 5x less accidents might be unreasonably hard. But causing 5x less accidents seems reasonable, especially since most humans are that safe.

I don't have the stats, but I believe the worst 20% of drivers cause a large fraction of the accidents. That 20% often includes the uninsured, the unlicensed, the drunk, high, emotionally distressed, and the physically compromised (senile, low blood sugar, tired, etc)

Causality12• 6 years ago

That's true but beside the point. The moment autonomous cars become better than the median human driver, keeping them off the road starts costing lives every day.

davidmurdoch4• 6 years ago

I rented a 2019 Mercedes last week and drove it for over 1200 miles, most of which was driven with the cars driving assist technologies enabled.

My guess is that because this car drives so "carefully", such as automatically following at a safe distance (leaving maybe a 3 second gap between the car in front of it), human drivers will end up causing many more accidents. There must have been more than 50 drivers (with many annoyed stares into my window as they passed) that made unnecessary lane changes to go around me just to then closely follow the car in front of me.

This large gap may make it seem like the car is going slower than it is, as so many drivers tried to overtake me but failed as slower traffic in the other lanes blocked them.

Human drivers may just become worse over time as more law-abiding autonomous vehicles hit the road. "5x" might not be as much of an improvement in the future.

stronglikedan• 6 years ago

I like the adaptive cruise control, because it drives more carefully than me. I have the same experience as you when I use it, regarding other drivers, but then I realize they're going to drive that way whether I'm using it or not. Therefore, I'm of the opinion that human drivers other than myself will cause the same amount of accidents, but I may cause less using while using it, so in the end there will be less accidents caused by human drivers as more people use adaptive cruise.

renewiltord8• 6 years ago

I hope people aren't driving with cruise control in the left lane. I think I'd probably pass on the right to get clear of them.

davidmurdoch4• 6 years ago

This is much more than just your classic speed-only "cruise control".

You certainly couldn't drive with classic cruise control in a center or right hand lane (USA) for any extended period of time, at least not in normal highway-speed traffic where cars are merging in and out every few miles.

opportune2• 6 years ago

On the contrary, I think as AV and other semi-autonomous driving tech becomes more frequent on the road, people will be more easily able to recognize it and won't behave irrationally as you mentioned.

davidmurdoch4• 6 years ago

I don't share the same optimism. Many people already stare at their phones while driving on the highway (I especially enjoy jolting them back to attention with a friendly honk), I'm not so confident they'll pick up on the subtleties of autonomous driving.

sliken11• 6 years ago

My Tesla has a user settable distance, which I do change based on conditions to avoid becoming a hazard as people constantly try to fill the gap.

davidmurdoch4• 6 years ago

Interesting that a feature intending to increase safety ends up being a hazard.

AndrewKemendo• 6 years ago

This is 100% just an artifact of a system going from human control to non-human control. Nobody bats an eye at systems which were never human controlled - or transitioned so long ago that nobody recalls human control.

I've never seen anyone hesitate when getting on a fully automated train system at an airport or an elevator. Even more-so with amusement park rides that literally put people in extreme situations.

stkdump3• 6 years ago

In amusement parks you get a much smaller selection of the population than on the street or even at an airport. People who don't enjoy thrill just have no reason to visit them at all.

Aside from that, all the systems you mentioned are mechanically constrained way more than a car. Accidents happen, when these mechanical constraints physically fail, not when a computer makes a wrong choice because it failed to detect an obstacle or similar.

segmondy• 6 years ago

What's so magical about 5? Why not 4x or 6x. 2x safer will be 500,000 lives saved yearly. We can see that even 1.25x safer is very significant. Just weird seeing that magic number 5x...

Traster• 6 years ago

I don't think it's useful to talk about global traffic deaths in this context. Since obviously regulation will change by country, the difficulty of developing self driving will change by country, and road safety varies enormously by country. The US is likely to get self-driving first, but is already way safer than the average country, and the countries where deaths are higher are less likely to be able to afford the roll out of self-driving cars.

In the US there are ~36,000 deaths from motorvehicle accidents.

To give some context, America could improve their fatality right by 5x by bringing themselves into line with the safety standards observed in Western Europe whose fatality rate is already at around 2.7 per 100,000 people.

It's also important to remember that self-driving is likely to represent the safest journeys - highway commutes etc.

Tempest1981• 6 years ago

Something psychological I guess? Or maybe one group said 2x and another said 10x?

psychological mechanisms influencing the decision-making regarding acceptable risk

sliken11• 6 years ago

Not magical, but you have to pick something. I suspect the goal is to pick a number that most would think is better than an experienced, awake, attentive (not looking at a phone) human at the wheel. So even the safest drivers would be safer on the road as the number of autonomous driven cars increases.

At only 1.25x it might well be worse than you. Keep in mind that the average safe human driver includes people that are tired, high, drunk, unlicensed, mentally ill, physically compromised, etc.

While common, someone getting killed because someone is drunk or asleep is much more acceptable than having a computer make a mistake.

If we want society as a whole to accept autonomous cars it's best to show a clear benefit to society, not just better than 51% of drivers.

comeonseriously• 6 years ago

I want to know who is responsible when the AI makes the wrong decision and someone gets hurt. I want that to be fleshed out first.

Beyond that, if SDCs are even just as safe as HDCs, I'm good.

stkdump3• 6 years ago

if SDCs are even just as safe as HDCs, I'm good

You might be a safer driver than the average human driver, in which case a SDC increases risk for you personally (and overall, in case the less safe drivers keep using non-SDCs). In that regard, we should wait until SDCs are safer than almost all human drivers.

Most drivers believe to be above average drivers, which is of course impossible. But there might be interesting correlations, for example with social status. Lower social status does disadvantage people in many regards. I am sure that insurance companies have data on the question if they have more (fatal) crashes as well.

And it does stand to reason that people with higher social status drive newer and better cars as well, so we could end up with a situation where the better drivers are replaced by computers before the worse drivers.

Interestingly, I think an ethics committee said a few years ago that once SDCs are safer than human drivers, it becomes a moral imperative to outlaw non-SDCs. I am wondering, if they will explore the human driving safety 'distribution' before inacting such rules. Waiting for the 5x margin could solve that problem, because then you will probably have SDCs that are safer than almost all human drivers, and it could be an incentive for companies working on the technology to get to that level faster, than they would in case they start selling them en masse earlier.

kevin_thibedeau2• 6 years ago

There will always be a long tail where the machines fail in scenarios a human can handle. We're just going to write off those deaths as an act of nature?

ogre_codes• 6 years ago

The big problem with numbers like this is how do you measure it?

Tesla claims their system is vastly safer than human drivers, but currently it only engages in situations where it's already fairly safe to use. So should that system be 5 times safer than all human-driving, or safer than human-driving under the conditions the Tesla is able to engage?

kstrauser2• 6 years ago

I have an older friend whose driving terrifies me, but who lives in an area with effectively zero public transportation or reliable cab service. While I don't want to see this person on the roads, the alternative is literally moving into a senior community (which would probably be the death of this person).

Frankly, if self-driving cars became .75x as safe as the average human driver, it would still a net safety improvement if got this person out from behind the wheel.

LoSboccacc6• 6 years ago

I hate this mindset, lots people get killed by senior driver with impaired vision/reflexes and it's always the same bullshit "but it would kill him to take away his independence".

well, tell that to people that will eventually be under his wheels.

kstrauser2• 6 years ago

I didn’t say it was OK. I’m not in a position to do anything about it, and my friend lives many states away. But as a practical matter, it they’re going to keep running errands and such, I’d infinitely rather they be doing it in a self-driving vehicle.

I’d also suggest a little empathy for people in such a situation. “Taking away their independence” sounds so minor and practical, but often is basically a euphemism for “removing the sense of self-determination that allows them to stay alive”. It’s not as simple as saying “Old Joe shouldn’t drive anymore”, because the implications can be pretty severe. That doesn’t mean that it’s a great excuse for Joe to keep driving! But it does mean that there are some pretty big philosophical conversations to be had about it. And if/when self-driving cars come around, a lot of those drawbacks vanish in a puff of smoke. Then we can say, as a society, “we want you to keep just as much independence as you’ve always had! You just don’t need to be the one working the controls of the car anymore, but it’s still going to take you on every errand and friend visit you’d ever want to go on, and actually help you be even more capable.” That sounds pretty damn splendid to me.

AtlasBarfed• 6 years ago

I think the insurance companies will have a different and much more financially based standard.

More importantly, I doubt NIH will trump that conglomerate and its influence on NHTSA

Also, you could argue that restricting a technology that will result in 20% less deaths on the road is the opposite of the public health.

To underline that, that is potentially 10,000 people in terms of death or major disfigurement. PER YEAR.

And self-driving could be, in a targeted/situational manner FAR safer if it took drunk/drugged/tired drivers out of the equation, which are responsible for around 33% of deaths.

If someone is drunk, a technology 2x an alert driver will be 10x a drunk driver.

thedudeabides5• 6 years ago

Maybe people want self-driving cars to be 5 times safer because they don't trust the people saying the cars are X times safer to begin with.

Like you are testing how much they trust machines, and how much they trust the people telling them the machine is X% better.

Given 2020, think a little skepticism on both/either is reasonable.

manfredo• 6 years ago

Why, though? Even in the case that it's the same safety self driving cars would yield huge productivity gains as people can work or sleep while commuting and truckers can have 1 or 2 self driving trucks following them cross country. And transportation for the elderly or disabled who cannot drive themselves.

gremlinsinc2• 6 years ago

Costs aside, what about something like mag-lev tracks for cars, that can start/go on a dime, and on freeways go faster, even switch lanes to get around slower traffic. Maybe even do away w/ speed limits just go as fast as you 'feel safe' going with the only limit being the max. In cities you'd have sensors/grids everywhere to detect non-car traffic, and regular cars could even drive over the mag-lev, or it could be a separate track, and you can go in/out of mag-lev/drive modes. Maybe it parks you, til you're ready to take over control (say you're napping on the commute). Alarm goes up, you wake up. Stretch, even get out and stand up for a minute, get back in. Buckle up - drive the final block to where you want to park at your job, or if it's a country side location, up in the mountains, etc you might drive for longer then park where ever.

Essentially you could just cover cities and highways to nearest gas stations. Car's running out of gas/electricity it routes itself to nearest depot.

Going cross country and want to stop at lunch? Program the car, and it'll pull to nearest gas station in tim-buk-two and let you figure out where to go from there.

Point: A/I self-driving aren't the only way to get autonomous cars. 50/50 re-thinking infrastructure, sensors, car-to-car communications could get us a lot closer faster.

Zigurd• 6 years ago

An oft overlooked factor in acceptance of AVs is that the evolution of technology from driver-assist to autonomy will alter perceptions:

First, human drivers using driver assist will become safer "drivers" even though the added safety is properly the result of the technology that is evolving toward AVs. For example, it should become very difficult for a human driver to hit a pedestrian or cyclist. Not impossible. Just exceedingly unlikely to be the fault of the driver.

Secondly, driver assist will habituate drivers and other road users to the performance characteristics of AV technology. The upshot is that AV technology will not be benchmarked against the way drivers and other road users behave and perform today. In some ways the expectations for incrementally better safety will be higher. In other ways, the "flavor" or road risks will change in a way that converges on how AVs perform.

gamerDude• 6 years ago

A lot of comments are focusing on safety via driving better. But with self-driving vehicles, can't we make the layout of the car safer, and thus accidents cause less harm to the people inside?

For example, right now because we need to see the road, I assume there is significantly more danger from the windshield vs. a padded back on both sides of the car with passengers facing each other like in a train car.

It seems likely that we can make self-driving vehicles much safer, even with the same number of collisions, by just changing the layout.

vernie• 6 years ago

Maybe Tesla can revive Elon's "mini-sub" idea.

wlesieutre• 6 years ago

For handling rapid deceleration, I think you’d be best off with seats facing the rear. A cushioned seat can distribute force much more evenly than a seatbelt when the car stops suddenly and your inertia wants to keep you going, and the backs could be reinforced to block anything coming into the passenger compartment from the collision. Hard to do that with a windshield.

e9• 6 years ago

On top of facing the rear they can attach industrial spring to absorb the shock of the fast deceleration/collision

cogman10• 6 years ago

After the crash is where I'd be concerned. While the windshield has a lot of negatives, one positive is that it's relatively easy to break and pull someone out of the car in a crash.

IDK, probably still safer, but it's something that'd deserve some aggressive crash tests to make sure we aren't making a pinto mistake.

philwelch3• 6 years ago

The ole sitting in the bed of a pickup truck technique, in other words.

hypertele-Xii3• 6 years ago

I would be unable to sit in such a vehicle at all, then. I must face forward and clearly see the horizon or I throw up in a matter of minutes in any moving vehicle.

dheera• 6 years ago

On a related note, before we get to self-driving cars I really wish we had un-crashable cars.

They have systems like active lane keep which are supposed to increase safety but do the exact opposite of what's safe. For example suppose a driver falls asleep or gets a heart attack. The lane keep will beep a few times quietly and disable itself when the car reduces speed to under 40mph, causing the car to then go straight and crash, instead of aggressively lane keeping until the car comes to a complete stop while increasing the beep volume the whole way, and then putting on emergency flashers, which would be the safe thing to do.

spullara• 6 years ago

Ultimately all the even lower level self driving cars will just pull over and stop.

nxc18• 6 years ago

Lane keep gets it wrong often enough that this doesn’t really work. Probably ~80 of activations are false alarms in the two vehicles I’ve driven that had it.

Assists are nice, but I don’t want the car to be aggressively wrong.

bsder• 6 years ago

Just getting rid of the steering column would make cars significantly safer.

This doesn't even require an autonomous car--just "drive by wire" electric vehicles.

RcouF1uZ4gsC2• 6 years ago

One issue that is often overlooked is that humans are pretty robust to unforeseen situations vs AI. Take for example the recent fires in California and the smokey skies. Many of the cell phone pictures did a horrible job of capturing the photos, because their AI had been trained that the daytime sky is blue.

And with such a failure, all the cars with similar software would be affected at once.

Tade0• 6 years ago

Here's how I think autonomous cars could be introduced, once they reach human-level safety:

As a penalty for bad driving.

I believe people would be much more accepting of machines taking the wheel if it meant that at least "this guy there" isn't driving.

philwelch3• 6 years ago

Fair is fair. If a self-driving car is a safer driver than John Doe, but we don’t allow that self-driving car on the road, perhaps we should revoke John Doe’s drivers’ license as well.

meekrohprocess• 6 years ago

We absolutely should. But we should also have alternative ways for John Doe to get around without a car, and we don't.

Car culture in the US has started to evoke that old saw: "If you owe the bank a thousand dollars, you have a problem. If you owe the bank a billion dollars, the bank has a problem."

Our society would be a lot better off if we regularly revoked the licenses of unsafe drivers. But the entire nation is designed around car ownership, so doing that today would be an unconscionable human rights violation.

foreigner2• 6 years ago

I love it!

floatingatoll4• 6 years ago

Human beings need a target for vengeance and hatred. If someone kills your child, you can hate them. If an automated car kills your child, you don't have anyone to hate. Saying that this is about safety thresholds is a distraction from the true human problem exposed by automated cars:

"Which individual will held accountable and risk jailtime if their car kills someone you love, and how can this individual be identified from the appropriate government registries within 24 hours of a death?"

Until this is clearly defined in law, automated driving will continue to be resisted under any number of plausible justifications, and arguing with those justifications will have little effect.

chii2• 6 years ago

The owner of the autonomous car is/should be responsible.

SammyNameTaken• 6 years ago

suppose you get framed by having a vulnerability in your vehicle getting exploited

floatingatoll4• 6 years ago

It’s less interesting to me who is registered as responsible, or what process is used to select that person. But if no specific single named individual is registered as personally liable without possibility of corporate liability shield, then we won’t get public acceptance of self-driving cars for a much longer time than could be possible.

amanaplanacanal5• 6 years ago

This seems obviously wrong to me. What is your reasoning here?

anfilt3• 6 years ago

Problem is your the average for human drivers is brought down mostly by bad drivers. However, even the if we talking about averages 50% of people are better than average. I would not feel to confident in a self driving car that is only approximately as good as 1 out of 2 drivers. I would want the car to be better than high upper percentile. Especially knowing an automated system can have reaction times that can put any humans reaction times to shame.

So yes equal safety is far from enough. Especially considering 50% would be better drivers than a self driving car that was only as good as the average driver. Your asking 50% of people and some percentage of people who overestimate their abilities to trust a car that would perform worse than them.

newacct5833• 6 years ago

Most of your point is just the Lake Wobegon fallacy at work. Obviously you think you're not a below average driver. But everyone does, and lots of them are wrong.

The part that isn't is that you're imagining a non-linear distribution where a significant fraction of accidents are the fault of these identifiable "bad drivers" and that the MEDIAN human driver is actually better than the average/median SDV.

To which I ask: is there evidence for this? It's a new argument, and I haven't seen it made before.

anfilt3• 6 years ago

If a self driving car is only as good as the average driver then by definition 50% of people are a better. How is that a fallacy?

It also could be considerably higher depending on the skew of the distribution. Depending how much bad drivers lower the average.

yazaddaruvala• 6 years ago

Keep in mind we also have selection bias that keeps human driving safer.

Bad drivers eventually have their license revoked or their insurance increase. Either way they typically stop driving.

scoot_718• 6 years ago

Economically, people will not buy a car that sacrifices them in favour of others.

howlgarnish• 6 years ago

Misleading title. The article is not saying we need more than equal safety; it's saying that self-driven vehicles are perceived as more dangerous and people in the study wanted them to be 4-5x safer than human-driven vehicles to overcome this.

enchiridion• 6 years ago

How many people have the current SOTA systems killed?

rkagerer• 6 years ago

5x safer isn't an unreasonable threshold to ask for. All life-critical, engineered systems incorporate a safety margin.

To win buy-in from the population at large, you sure as heck better have overwhelming statistics. Marginal just won't cut it.

When you're talking about taking control away from the driver, all you need is one accident a human would have prevented to create political backlash and deteriorate trust in the system. Just because several other unvoiced lives may have been saved doesn't offer consolation to the victims or deem the loss acceptable. I think of it as analogous to the legal doctrine that convicting one innocent is worse than letting 10 guilty individuals go free.

I'm always skeptical when developers claim a computer can do a better job than a human, as I've encountered so many edge cases the programming just never adequately accounted for. It will take time and a great deal of experience running these platforms in the wild until they become truly resilient. I would be quite pissed off to find the severed limb I suffered in a crash was due to programming not sufficiently distinguishing e.g. road paint from water streaks.

And the metric you choose to define "safer" will never be perfect. Extra buffer helps offset any bias or gaps in your methodology and capture more of the long tail of "one-offs" never accounted for in version 1.

I'm a big believer that driverless vehicles will provide huge improvements to our quality of life, I just feel there's a lot of misrepresentation taking place out there today as to how far along these systems are.

foreigner2• 6 years ago

Even ”overwhelming statistics" aren't enough, look at vaccines.

morpheos137• 6 years ago

What most people seem to misunderstand about "driving" is that it is not a sensory or stimulus response problem. It is a cognition problem.

Computers or computer based "AI" are good at solving bounded problems that do not require open ended or on the fly model building and judgement exercising in real time. At this biological intelligence, honed by millions of years of selective evolution still excels.

Computers can "solve" GO or Chess because fundamentally the rules are simple and the models required to play these games are subject to only a few, static constraints.

Driving, in all conditions on all roads, on the other hand, requires a flexible model of the real world that approaches that built by sentient biological intelligences.

The problem is not sensor or perception latency.

Sure LIDAR can blow human perception out of the water.

But that does not matter.

What matters is making the correct decision based on sensory inputs using a high fidelity model of the real world.

Computer does not understand the difference between two people playing catch by the roadside, parallel to the road and a situation where a child might be chasing after a soccer ball. This is not just combinatorics and probability...it is theory of mind. Thus until AGI is invented FSD will be a misnomer.

Doesn't matter how much faster a computer can perceive if it does not know how to integrate the raw data it receives into a model of the world that yields correct decisions to the circumstances.

Self-driving vehicles against human drivers: Equal safety is far from enough

Related Stories