Waymo recalls software after two self-driving cars hit the same truck

https://www.cnn.com/2024/02/14/business/waymo-recalls-software-after-two-self-driving-cars-hit-the-same-truck/index.html

by reteltech • 2 years ago

212 79 2 years ago

cnn.com

Fricken• 2 years ago

A pick-up truck being towed crooked and backwards. Both vehicles failed to read the situation in the same manner.

Autonomous vehicles have various redundant systems built-in that can take priority and override false positives.

I was previously under the assumption that one of the really important reasons for Lidar is that it can get you closer to an absolute truth about whether something is a solid object, and where that hypothetically solid object is relative to the position of the vehicle, regardless of what the classifier thinks it is seeing.

So did the lidar fail to read the solid object, or was the lidar, was it de-prioritized? or was it simply not available as a fallback?

Presumably Radar and proximity sensors were also involved. What were they doing?

This is a fascinating edge case, and I hope to hear about the real reason for the 2 incidents.

Mageek• 2 years ago

The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.

pests• 2 years ago

How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.

PH95VuimJjqBqy4• 2 years ago

I don't find that acceptable in any way, no human driver is going to do that and by that I mean no human driver is going to drive into something just because it moved in a way they didn't expect. they're going to slam on the brakes and the only way that's going to happen is if momentum is too high.

I understand we have to have explanations or we can't fix them, but it's just as important to understand this should never have happened even WITH the described failure.

If I had to guess, there's code to avoid stopping at every little thing and that code took precedence (otherwise rides would not be enjoyable). And I get the competing interests here but there must be a comparison to humans when these incidents happen.

ralph84• 2 years ago

Link should go to the blog post not CNN.

https://waymo.com/blog/2024/02/voluntary-recall-of-our-previ...

im3w1l2• 2 years ago

This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin.

But if everything is fine, everything is fine, everything is fine, and then all hell breaks lose? We are not as good at dealing with that.

add-sub-mul-div• 2 years ago

Yeah, there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once.

We also know how to hold individuals accountable for independent accidents. We know we won't get justice when people will inevitably start to get killed by standard corporate greed, incompetence, enshittification.

gdestus• 2 years ago

I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire

1000100_1000101• 2 years ago

there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once.

All, no. Enough to make a difference, easy.

  Black Friday in February!  In-Store offer only!
  $99 Playstation 5 to first 100 customers at each Walmart location!

You can bet there will be a significant increase in people driving badly. edit: Make it Taylor Swift tickets, and you can increase the size of the frenzy.

hyperman1• 2 years ago

I notice, when e.g. it rains enough for long enough, a lot if people get annoyed and start driving badly. You get some kind of contagion where everybody gets annoyed and tired because all other drivers are annoyed, tired assholes. Sometimes it even persists after the bad weather, especially if there was gridlock and/or end of workday. Humans do have a (lighter) version of collective bad driving.

leoh• 2 years ago

This will have to be priced in to insurance.

brk• 2 years ago

My fear is similar, but more along the lines of adversarial attacks as various weaknesses are exposed. Imagine people taking advantage of zero-day exploits that cause self driving cars to veer off the road, stop suddenly, collide, etc. It is really not that far fetched. This technology is very far away from maturity, IMO.

Ukv• 2 years ago

Relevant XKCD: https://xkcd.com/1958/

nicbou• 2 years ago

On the other hand, the cars were recalled, a team will study what happened, the problem will get fixed, simulations and tests will be created.

When I make a dumb mistake, the other drivers rarely learn from it.

simion314• 2 years ago

So in future your self driving car might be recalled at any random time because a new corner case from an infinity was found. If the car can't be driven by a human all owners of thsi cars will be stuck wherever they were at that moment.

isthatafact• 2 years ago

As someone else hinted, there may be nothing to "fix", but rather this seems like a specific situation that it had just never encountered before. Adjusting the model to cause a safe response to that particular single rare situation (either manually or by accidental training) does not solve the apparent problem that the machine is not able to comprehend the world.

At least that is how I (as a non-expert) imagine these models work -- the model has an excellent chance of crashing at every new unique situation it encounters out of a nearly unlimited set of possible situations (which implies a high frequency of encountering new situations).

Klaus232• 2 years ago

Sounds like they were relying solely on their neural network path prediction, which failed when the truck was dragged at an odd angle.

A simple lidar moving object segmentation, which doesn't even know what it's looking at but can always spit out reasonable path predictions, would probably have saved them.

I think Mobileye is doing something like this, but they release so little data, which is always full of marketing bullshit, that it is hard to know what exactly they are working on.

digitallis42• 2 years ago

It's unlikely to be neutral network based. This sounds like a model prediction failure. You take a mathematical model of car motion: the rear wheels generally don't steer. The front steered wheels can cause the car to drive along an arc. If you want to predict the arc that will be driven, you take the initial starting heading of the vehicle and project forward in time based on your understanding of the vehicles steering angle. For most "driving in lane at velocity" cases, you generally would assume that the vehicle has very little steering angle input.

We're now getting to see where autonomy needs to develop "spider sense": the scene in front of me feels wrong because some element isn't following the expected behavior maybe in ways that can't really be rationalized about, so we'll become much more conservative/defensive when dealing with it.

rasz• 2 years ago

Its NN black box alright, for a hand crafted model you would have something like:

    if prediction != reality
     {mismatch+= whatever the time tick is}

    if mismatch > second (or two)
     {abort driver assist}

im3w1l2• 2 years ago

I'm thinking it might make sense to have a sort of hierarchy of models. The stupidest model predicts that everything will be stationary. The second model predicts that everything will travel in a straight line. The third model tries to predict a circular arc based on same fusion of path history and observed steering input. Fourth model uses a notion of action to predict whats going to happen, like "the car is changing lanes". Fifth model uses body language and common sense to predict intention "the pedestrian wants to cross".

Each model can potentially predict longer into the future but also has more complexity and things that can go wrong. So you keep track of how well each model is doing (on an object basis) and if one level is failing then you fall back on a stupider one. You might also want to increase caution if your models are not doing well (lower speed and increased safety distance).

Klaus232• 2 years ago

It is definitely based on neural networks: https://www.youtube.com/watch?v=RpiN3LyMLB8

pinkmuffinere• 2 years ago

I wish there was a picture of the strange towing configuration. I wonder if I would be confused as well, although my guess is that I’d read the situation correctly

midasuni• 2 years ago

I’ve been confused occasionally by road situations. Somehow I’ve never managed to drive into a multi-ton object.

ghodith• 2 years ago

Can't say the same for everyone.

Cacti• 2 years ago

Well that car has been going around saying at least it hasn’t been posting uninformative, useless anecdotes on HN.

carbonatedmilk• 2 years ago

One of the best things I've learnt recently is how to apply the zero blame, process improvement approach that (many) air safety regulators take to my own teams.

I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs.

By comparison, the way we're investigating and responding to self-driving safety incidents still seems very primitive. Why is that?

esafak• 2 years ago

Where did you learn about that approach, and how is it different?

sokoloff2• 2 years ago

Not GP, but:

There is an explicit culture and mechanism of blamelessness around safety concerns and minor violations/deviations, which is incredibly helpful. Read about the ASRS* program (admin'd by NASA, with anonymity for non-intentional issues, prohibition on use of submissions for enforcement purposes, and explicit "get out punishment" card from the FAA): https://asrs.arc.nasa.gov/overview/immunity.html

The FAA is also explicitly including "evidence of voluntary compliance" and "just culture" in its approach to aviation safety, and explicitly changed its goal from enforcement and proof to ensuring compliance [with enforcement being only one available tool]: https://www.faa.gov/about/initiatives/cp (PDF pres: https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compli... )

I'd also read a bunch of aviation reports: https://www.ntsb.gov/Pages/monthly.aspx (more detailed reports are available approximately 2 years after the occurrence date and more details are available for fatal or air carrier occurrences, so if you don't care which ones to read, filter for those to start).

If you're more video oriented, watch @blancolirio, @NTSBgov, @AirSafetyInstitute, or @pilot-debrief. (I'd skip @ProbableCause-DanGryder.)

For a short summary, there is an intense focus on determining the facts (who, what, when, where, maybe some guesses as to why) and drawing conclusions about primary and contributing causes from there.

* Aviation Safety Reporting System

AndrewDucker• 2 years ago

Fascinating article here with an example of it: https://asteriskmag.com/issues/05/why-you-ve-never-been-in-a...

extua• 2 years ago

Another comparison with air safety, the disaster risk threshold is high enough to ground vehicles with suspected faults or flaws.

In this case two self-driving cars crashed into another road vehicle because they failed to recognise (in time) which direction it was moving. Waymo should be commended for having voluntarily issued a software recall, but this problem is severe enough that the decision shouldn't really be up to Waymo's good judgement.

cameldrv• 2 years ago

My guess is that Waymo does do this internally.

One difference with this situation in terms of the public perception/discussion though is that, say in the 1960s, air safety wasn't very good compared to today, but still there was no question of eliminating air travel altogether due to safety issues. Today there is definitely an anti-self-driving contingent that would like to hype up every accident to get the self driving companies shut down entirely.

andreareina• 2 years ago

Do we have a picture of the truck? I'm having difficulty imagining it given that surely the tow truck would want the towed vehicle in-line to make driving go smoothly?

rainbowzootsuit• 2 years ago

The towed vehicle has its rear driven wheels up on a tow hook type of tow truck and It sounds like a locked steering wheel that has been turned. This would lead to the angled tracking of the front wheels.

This would be common for a debt recovery or when a city impounded the vehicle where it's taken without cooperation of the owner.

https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...

Terr_2• 2 years ago

[Recycled from a older submission] Well, I feel kinda vindicated by this news, after previously noting:

People worry that ways and times [self-driving cars] are unsafe (separate from overall rates) will be unusual, less-predictable, or involve a novel risk-profile.

In this example, having a secretly cursed vehicle configuration is something that we don't normally think of as a risk-factor from human drivers.

_______

As an exaggerated thought experiment, imagine that autonomous driving achieves a miraculous reduction in overall accident/injury rate down to just 10% of when humans were in charge... However of the accidents that still happen, half are spooky events where every car on the road targets the same victim for no discernible reason.

From the perspective of short-term utilitarianism, an unqualified success, but it's easy to see why it would be a cause of concern that could block adoption.

camel_gopher• 2 years ago

Human drivers just have secretly cursed driver configurations.

Terr_2• 2 years ago

But not systematically, unless someone invents a completely novel tier of outrage-inducing bumper-sticker.

adrians1• 2 years ago

This is what people don't appreciate when quoting those statistics about how self-driving cars are safer than humans: when a human driver causes an accident, it was because that particular person did something wrong. When a self-driving car handles a situation wrongly that's a big issue, because all the self-driving cars run the same software.

skybrian• 2 years ago

Yes, but when the bug was fixed, it was fixed everywhere.

skygazer• 2 years ago

That phrasing makes it easy to underestimate the astronomical dimensions of edgecase-space, and what necessarily remains unexplored after such a fix.

bsaul2• 2 years ago

how do we know for sure that it was fixed ? What are the conditions that are fixed ?

83• 2 years ago

On the other hand when a human driver causes an accident one driver learns a lesson (maybe). When a self driving car causes an accident all cars get to learn from it.

rkagerer• 2 years ago

That poor tow truck driver, must have felt like Skynet was really out to get him.

Waymo recalls software after two self-driving cars hit the same truck

Related Stories