After 6 months of working fine, Tesla software update drives at barriers again

https://np.reddit.com/r/teslamotors/comments/b36x27/its_back_after_6_months_of_working_fine_2019515/

by SamuelAdams • 7 years ago

339 323 7 years ago

np.reddit.com

x38iq84n5• 7 years ago

I have always wondered... Does the AP have some higher-level notion of object permanence, continuity (road behind a horizon or after a curve) and things like that? Does it track a pedestrian that is momentarily hidden behind an obstacle and will probably reemerge in a second or two on the other side? Does it expect that kids may run after that ball that just flew from behind a car? Does it continuously track and improve classification of all objects in the field of vision, with their trajectories and speeds if they are moving? Personally I don't think it does, otherwise it would not erratically slam into clearly visible and marked large objects in its way, or it would be aware of a truck moving in a perpendicular way and so on. I am of opinion that without such higher-level awareness it can never succeed, hope to learn about the state of the self-driving art.

dragontamer2• 7 years ago

I don't think neural networks are wired to "remember" things. In theory, they could be hooked up that way. But your typical convolutional neural network is looking at things frame-by-frame.

In theory, ANNs could have an output layer that passes data from one frame to another frame to assist things. But there's no real programming to "hardcode" something like object permanence into an ANN. You pretty much throw a bunch of data into the system and hope for the best.

hemogloben• 7 years ago

NNs are just the first step in the pipeline. Their outputs (detected objects, segmentation, etc) will be piped into other software that builds higher level models.

Considering the path-planning requirements I would be absolutely shocked if Autopilot wasn't build history models and estimated paths for objects around the vehicle (other cars etc).

robrenaud• 7 years ago

There are architectures that use CNN on image inputs, and LSTMs across frames of the video to keep memory.

https://arxiv.org/pdf/1609.06377.pdf

eclipxe2• 7 years ago

https://www.youtube.com/watch?v=rACZACXgreQ

automathematics2• 7 years ago

In these videos, "Autopilot" is mentioned as the culprit, which seems to be a subset of the features Tesla has.

It seems to me there are 3 layers:

(level 0: Just Adaptive Cruise Control (use radar to adjust speed up to a max). Human still steers.

- 1: Adaptive Cruise Control (use radar to adjust speed up to a max) + Autosteer (camera's watch the lane markers and follow them). This is referred to as "Autopilot"

NOTE: This is where the accidents happen. The car isn't driving towards a barrier, it's following the lane markers and hits an error state. This is also NOT self driving.

- 2: "Nav on Autopilot". This is an additional function you turn on where the car has more intelligent (use this word loosely) capabilities on highways. The car will still do everything on level 1 combined with lane changes (using cameras to detect objects and trajectories differentiating cars from trucks from pedestrians from bikes from motorcycles etc). It will still follow lane lines, but with a lot of additional information (is there an object? am I merging? am I exiting? etc)

- 3: "Full Self Driving". This is an additional package that doesn't exist to the public. Internally I'm sure they're testing the functionality but this is using all of the sensors and algorithms and likely neural networks to decide what to do. A cool point though is that all Tesla's are likely running this code in "shadow mode" where data can be collected and assumptions can be tested without endangering any actual drivers. (see here for some cool data on this: https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-...). "Hey, I think the car, if full self driving SHOULD take action X. *compare to what the driver actually does and log the data" over BILLIONS of miles

So when a Tesla hits the barrier or gets in an accident, we're actually running "#1" and people start freaking out. But when we get to the capabilities of #3 a lot of the "object permanence and continuity" stuff starts to come into play.

Full Disclosure: - I drive a Model 3 every day on Autopilot 75% of the time - I only 75% think self driving is capability under the current Tesla software suite but I bought the package anyways.

SamuelAdams• 7 years ago

Additional context: on March 23, 2018, a Tesla Model X, while using autopilot, drove into a barrier and killed the driver. This was fixed in a software update, but the issue seems to have resurfaced.

NTSB analysis of the March 23 2018 collision: https://www.ntsb.gov/investigations/AccidentReports/Pages/HW...

Tesla's statement of the March 23 2018 collision: https://www.tesla.com/blog/update-last-week%E2%80%99s-accide...

MagicPropmaker• 7 years ago

Well, it wasn't really "fixed" -- the driver is still dead.

craftyguy3• 7 years ago

Sure, if you construe the meaning of 'fixed' to mean 'roll back all events and time', which I don't think anyone expects. They probably fixed the software bug.

sschueller• 7 years ago

Eventually we need the NTSB to certify updates before they are pushed out over the air. Similar to what the FAA does.

These OTA updates are not ok for large machinery and endagers not only the Tesla driver but all others on the road.

x38iq84n5• 7 years ago

Indeed. I have never understood the cheering for OTA of something that can literally kill you. Were they limited to infotainment and the useless easter eggs I would not care, however AP OTA is downright scary, especially pushed by vendor instead of being initiated by an informed user.

0815test3• 7 years ago

The AP can only kill you if you're distracted/asleep at the wheel and not paying attention. You're always in charge of driving the car appropriately, the "AP" can only provide tentative assistance of a very rough sort. It will try to guess at the control inputs you might want to provide (which does make things a bit more comfortable in the best case) but the actual control always rests with the human driver.

DarmokJalad17013• 7 years ago

It is initiated by the user ... the update only happens after you give say it's okay to do so.

kitsunesoba• 7 years ago

I agree, but is there a way that could happen without slowing the process to a crawl? Depending on what’s involved it could easily push the gap between updates from months to years.

toomuchtodo3• 7 years ago

This is a feature, not a bug, and should be expected in life critical systems. Would you want Boeing to push updates out as frequently as Tesla does with the same sparse release notes Tesla provides (“bug fixes”) when safety system functionality is modified?

Disclaimer: I own a Model S

notfromhere2• 7 years ago

do you want faster updates or a machine that doesn't inadvertently kill you? because for life-critical systems you really can't do both

CharlesColeman• 7 years ago

I agree, but is there a way that could happen without slowing the process to a crawl?

I don't see why a more rigorous release process would slow progress down at all. All the iterations that lead to progress should be done on test vehicles, not customer vehicles.

"Move fast and break things" is a development model that should only be applied to low-importance, low-risk systems. Most software development work occurs on such systems, and I think that narrows the perspective of the software development community as a whole.

v_lisivka• 7 years ago

Autopilot can automatically record all incidents and then upload them to Tesla, with obfuscation and by user permit, to improve autopilot software. Tight feedback loop is much better for AI, IMHO.

Mizza• 7 years ago

Not eventually - now. This is putting everybody in danger, not just the owners of the cars.

quasse• 7 years ago

Yeah, it's a pretty crazy world where a game developer pushing an update to a game on Xbox live seems to have a harsher approval process to go through than an automaker pushing OTA updates that literally drive cars.

alistairSH• 7 years ago

OTA is fine. It's unregulated, unannounced OTA that's problematic.

I'd love for my VW to get updates OTA without taking it to the dealer. But, I don't want to receive those updates without knowledge that the update has been tested sufficiently (and given I don't trust the vendor, I'd like NTSB or similar government body to do this on my behalf).

dragonwriter• 7 years ago

Eventually we need the NTSB to certify updates before they are pushed out over the air. Similar to what the FAA does.

You mean, similar to what FAA might do if it didn't allow manufucturer self-certification instead, right?

Also, wouldn't NHSTA, which does safety regulation and standards for autos, not NTSB, that does accident investigation, be the natural agency?

smileysteve3• 7 years ago

Similar to what the FAA does.

The timing of this is precarious (given the recent 737 Max allegations)

Traster• 7 years ago

Having the NTSB certify updates isn't going to increase safety. The accepted approach to safety as set in ISO 26262 largely focuses on making sure that the processes in place via which the software and hardware is designed, created and modified. The reason you wouldn't get regression if you were interested in FuSa is because you'd have a process in place to ensure that software can't be distributed without being tested and a process to ensure that bugs are included in the test suite.

It's quite clear in this case that Tesla doesn't have an organisational structure that fulfills the requirements for functional safety.

sangnoir2• 7 years ago

Having the NTSB certify updates isn't going to increase safety.

It will if they run it through regression tests that Tesla doesn't seem to have the discipline to.

FreedomToCreate• 7 years ago

Tesla can only really gather depth data from there radars, while the cameras operate a DNN to detect features and make adjustments for steering and speed based on fusing those two pieces of data together. An error in the radar or the DNN detecting features incorrectly because of lighting, road color change or object on the road which the model was not trained on, can cause problems like this.

notfromhere2• 7 years ago

if it came back after an update, wouldn't it be the issue of the algorithm training itself incorrectly?

zmarty• 7 years ago

"Tesla can only really gather depth data from there radars" - they can also do it visually through depth from motion

SilasX3• 7 years ago

I would be a lot more forgiving of these screwups if Tesla didn't constantly swear up and down that they've solved self-driving cars.

As far back as 2016 they were claiming they had full SDC capability above human driver safety [1], and their recent Model Y announcement suggests that the only thing holding it up is regulatory approval, and not failure to achieve the desired spec.

Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot.

[1] https://www.tesla.com/blog/all-tesla-cars-being-produced-now...

[2] https://news.ycombinator.com/item?id=19397942 (linking HN because it's hard to find the text in the page with their UI)

Scoundreller10• 7 years ago

As far back as 2016 they were claiming they had full SDC capability above human driver safety

One crash/incident doesn’t mean it’s less safe than a human driver on average, even if it’s something a human driver might have avoided.

We should expect different failure modes from a machine, but adopt it anyway if it avoids enough human failure modes to make up for it.

Source: my thermostat thought 33C was an acceptable temp once, but I still didn’t switch to full-manual HVAC control.

tptacek• 7 years ago

Don't Teslas fare significantly worse than cars in similar class in terms of driver fatalities?

slg8• 7 years ago

I don't want to dismiss your whole point, because it is certainly valid, but that isn't really the issue here. It is entirely possible for bugs like this to exist in the self driving tech and for Tesla to be correct in their claims that Autopilot is on average safer than a human driver.

It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.

ketzo• 7 years ago

That’s one of those thorny AI-human problems, though — if I crash 1 in 10 drives, it’s my fault, and therefore much easier to rationalize to be, well, not my fault. If my Tesla crashes 1 in 100 drives, it’s a faulty machine and an obvious death trap.

skwb2• 7 years ago

No, it's still the issue. If there's a bug in the software code for a programmable pacemaker, it doesn't matter if the fundamental design is flawed. For governmental regulators, it's the actual implimentation and real world performance that matters.

Sure, a bug fix for a routine being called is maybe easier to fix, but the real world performance is still the important metric to track.

disiplus4• 7 years ago

i would not take those chances. if the self driving car is only slightly better then average driver in fatal accidents that's not good enough. most of accidents happen because of distractions (texting and so on) and if a self driving car is only slightly better then average driver I can get a lot better if I just don't use my smartphone while driving and don't sit behind a wheel tired.

so it's not enough to be better then average driver. It needs to be as good as "good" driver or better before I would trust it.

rayiner4• 7 years ago

I think you're missing the forest for the trees. We're not even at the point where we're comparing safety rates. Can a Model Y drive through DC? Can it ignore traffic signals in favor of following hand signals from a uniformed traffic officer, or follow the directions of a construction worker having traffic in turns through a road that they've decided to turn into one lane? Can it deal with an unannounced presidential motorcade rerouting traffic? Because that happens literally every day in DC. If Tesla can't handle that, it's inaccurate to say it has "full SDC," even before you get into an analysis of safety.

xkjkls• 7 years ago

It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans. You and I certainly don't have enough data to say one way or another on that. I would bet even Tesla doesn't have enough data to say definitively. However writing this tech off as unsafe just because it makes what seems like an obvious mistake is a great way to slow progress which will result in more human deaths long term.

And where is the research to show that? To prove that within any significance, they would have to have millions and millions of miles to compare against human drivers. This is new technology and making false claims about its safety is dangerous.

We as a society force drug companies to rigorously show that their drugs do what they say they do and all of their side effects are properly accounted for. This should be the case here too.

TheSpiceIsLife3• 7 years ago

Hang on.

Unless something has changed, the Telsa self driving tech intentionally ignores stationary objects.

And Telsa are claiming this is a feature not a bug.

Am I understanding this correctly?

I don't know anyone who isn't suicidal who intentionally ignores stationary objects while driving.

prestonh• 7 years ago

Is it, though? If every car that drove that route was a Tesla on autopilot, you'd probably have more traffic fatalities than the national total, and that doesn't even factor in every other similar traffic barrier this can occur at.

fmpwizard• 7 years ago

see self driving cars run into solid and stationary objects, but human drivers do that all the time too.

Human drivers who are distracted do that all the time, AP is supposed to avoid that, it is supposed to be alert all the time, but when it sees a stationary object, the result of their algo is, "it must be a sign we can somehow go through"

We need to make it very clear that self driving cars are better than driving drunk, or when you haven't slept in 24+ hours, but if you are a driver who pays attention, don't use this tech.

And it is not that I don't want the tech to take over the world, I wish I could just put my kids in a self driving car and have the car take them to the school that is 4 miles from home, but we are nowhere close to that, even with me in the driver seat, if I only have seconds to take over before I end up on a ditch or worse.

m4633• 7 years ago

I think this graph:

https://upload.wikimedia.org/wikipedia/commons/a/a5/Causes_o...

puts a lot of arguments (and young people) to rest.

thisisit• 7 years ago

It is obviously troubling to see self driving cars run into solid and stationary objects, but human drivers do that all the time too. The question shouldn't be whether this technology is perfect, it should be whether this technology is safer than humans

I fail to understand this line of reasoning. Are you saying because humans tend to run into solid objects it is okay if the self driving cars do it, only if it does less times than humans?

jrs95• 7 years ago

It may be better than humans on average, but is it safer than a sober person who is driving responsibly in a similarly priced vehicle? Anecdotally, most accidents I've seen have been caused by at least one driver doing some stupid shit. Based on my own personal experiences, I'm just not comfortable surrending control of my vehicle. Especially not when we could be comparing apples to oranges. Safety of a self driving system should be evaluated against currently available safety technologies like blind spot detection etc. If you're including older vehicles that don't have the best possible non-self driving technology in your data, you're making a comparison that is inherently biased in favor of self driving cars.

All that being said, I do think self driving is what we will inevitably arrive at...we just need to have a higher degree of confidence in it before it's widely used in my opinion.

Haga• 7 years ago

Nn tech debt.

Animats• 7 years ago

Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot.

We've heard that before, for the Model X.

pbreit• 7 years ago

The passage says “will” which implies the future.

bitL• 7 years ago

They solved it, they just need 100x faster GPUs in cars to be reliable outside datacenters ;-)

sabareesh2• 7 years ago

Since ModelX incident I have made sure not to drive on first lane. I always drive on the 2nd lane and it is much better.

bdcravens4• 7 years ago

For a $700 car I might consider crazy compromises, but $70k+? Absolutely not.

warp_factor• 7 years ago

That's what I find so interesting with Tesla owner. They spend a fortune on a car, then they minimize every single issue they have with the car. My explanation for this is that for a lot of owners, the car is a way to be part of a hyped group more than being a utilitarian object (what a car should be).

MockObject• 7 years ago

Folks might be even more likely to compromise with a $70k car than a $700 one. The psychology of sunk costs!

https://www.lesswrong.com/posts/tyMdPwd8x2RygcheE/sunk-cost-...

sabareesh2• 7 years ago

Autopilot is a beta feature that you are choosing to buy. And it costs 3k and no one is close to what Tesla has that you can buy now

kurtisc2• 7 years ago

Where I live, that's not legal.

anth_anm2• 7 years ago

Said it before, saying it again. OTA updates to safety critical things is not a feature. It's a bug.

bsaul• 7 years ago

I wonder how do unit tests work with NN (or if they're even a relevant concept at all).

You could replay some testing video frames and make sure the objects are correctly identified, but i suppose that's already what training is about...

If an issue like that resurface, does it mean that the original frames leading to the 2018 accident aren't part of the training (or at least frames from someone driving in this kind of scenario) ?

inertiatic• 7 years ago

Well, testing in the strict sense would be measuring accuracy on data not in the training set, and even accuracy on the training set data (which isn't guaranteed to be 100% I believe)

bluGill• 7 years ago

The frames from the 2018 should not be part of the training. They should be part of the test set to prove whatever training they do works.

neilalexander• 7 years ago

Yes, the problem is indeed that there's no real way to "look into" a neural network and understand how it has been trained. All you can do is observe that the given inputs generate the desired outputs.

Even if there was training based on the 2018 frames, that doesn't mean that you have verifiably fixed the problem. It's difficult to train a neural network selectively – every time you "train" the network with additional data, you are increasing the chance that you are also teaching it something you didn't intend which then can have a side-effect in some seemingly unrelated scenario.

You can see this in real life with image recognition networks. Teach them too much and they gradually become less effective at identifying anything.

jononor• 7 years ago

Unit tests are not sufficient for neural networks. Say the network takes 2D images of typical 224x224 pixels with 3 channels (RGB) of 8 bit values, this input space has 256^(224x244x3) = 3.5×10^362507 possibilities. Billions of years to test them all. This is before we consider stereo vision, 3D images and state over time. How to know which subset of these inputs are necessary to give reasonable coverage? Right now I don't think we have very good answers to these things. Of course one can always add a regression test (with some K mutations) when someone crashes. It is better than nothing, but hardly good assurance that something like this will never happen again.

The entire area of safety and quality assurance with neural networks is still in active research actively, from multiple angles. For some examples of how chaotic neural networks can be, look up 'adversarial examples'.

dv_dt• 7 years ago

My first thought was, well someone is in store for a revamp of their regression test suite.

ummonk2• 7 years ago

Should be using integration tests, not unit tests. I.e. SITL simulations + having the car drive around test circuits / scenarios.

skwb2• 7 years ago

You don't unit test a NN. You can unit test certain functions, but fundamentally this is an integration test.

This is why serious scientific training is needed to understand these complex systems when health and safety are on the line.

leggomylibro• 7 years ago

"Autonomous vehicle integration test track" could make for a great setting in a spy thriller. The villain could own the megacorporation which makes the cars, and the heroes could find evidence of their evil plot among the sprawling acres of labs and potemkin streets. But then, in the distance, the sound of revving engines...

Seriously though, I wonder if that sort of physical test track will become popular. You would load your build onto an idle car, queue it up, and make sure that it didn't hit any of the silhouettes which spring up, unusual traffic and weather conditions, etc. They must already do that in some capacity, right?

Scoundreller10• 7 years ago

That’s a very strange way to close a highway.

Where are the blinkenlights? The words? The flashing arrows? The plastic jersey barriers?

The buckets full of sand or water?

The repainted lines directing you to the right?

Can’t tell if this is temporary or long-term closure, but if it’s a multi month thing, I would expect more “Don’t you dare fork to the left” signalling.

Not everyone is a local.

blahyawnblah• 7 years ago

It's a lane of traffic that can go either way depending on demand. Common in larger cities in California and Washington.

jakeogh2• 7 years ago

Why would any of that be relevant?

Scoundreller10• 7 years ago

Because the same things that confuse humans can confuse computers that are largely trained with human data.

muzika• 7 years ago

Increasingly, cities will be expected to plan roads in a way that would be simpler for autonomous vehicles to bavigate.

stagger87• 7 years ago

I would think the several reflective construction signs with arrows would be enough. Also there is an impact attenuator (buckets of water). It's also cut too soon to the turn to know whether there was more signs leading up to it.

Scoundreller10• 7 years ago

The only reflective gate that swings into the second lane from the left is after the barrier.

I checked the street view linked above and followed it back: no signs.

sbierwagen• 7 years ago

The section of road shown in the video is the I-5 express lane where it rejoins the regular freeway in North Seattle: https://www.google.com/maps/@47.7021836,-122.3299637,3a,71.1... Note how it's configured for southbound traffic in the street view image. It switches direction twice a day.

Though you can't see it very well in the video, there is a crash arresting net after all the signs. https://www.google.com/maps/@47.7034263,-122.3302029,3a,19.3... You can't just drive through a couple of boards and run into oncoming traffic.

Scoundreller10• 7 years ago

I followed back the first link for the opposite direction of travel and didn’t find any signage before the swing-gates. You kinda truck along at 45mph+, sticking to the left to continue and with a small amount of notice time, you need to fork to the right.

I get that a local would know you are to always fork to the right, but someone unfamiliar or a machine could be confused by the initial half barriers, with only a 2 lane wide barrier after the demarcation point. Along with the oddly painted lanes.

But as I said in another post, I guess DOT would have the data on incidents (and hopefully act on it if there’s enough evidence in favour of more safeguards).

paul79862• 7 years ago

Scary... software developers at these robotic cars and their mistakes/bugs aren’t just going to bring down a business application(lose money) but kill their customers and innocent drivers.

Progress to where it’s safer is going to be a killer and we the drivers on the road are unwilling guinea pigs to billionaires’ dreams/goals.

nathanaldensr• 7 years ago

Unwilling? Don't buy a Tesla. Don't believe the hype. It's that simple. Granted, there is nothing stopping any ECU from killing you, but I'd trust a company like Honda way before Tesla.

Tomte2• 7 years ago

I generally have zero interest in cars and don't follow the new models, but my impression from articles I've seen in the last years is that Volvo is actually a top contender for driver assistant systems (when you don't fool yourself into thinking you have an autopilot, but you really want sensible safety augmentation features).

Is that impression accurate?

adrianN• 7 years ago

Tesla crashes can injure people who don't drive a Tesla themselves.

maxxxxx• 7 years ago

"Unwilling? Don't buy a Tesla."

That doesn't protect me from being killed by a Tesla. I am pretty neutral on the topic but I am getting the feeling that they are in danger of pushing out half baked stuff like we tend to do in software. For most things like software this is OK but maybe not for things that are moving at high speeds.

trhway• 7 years ago

we the drivers on the road are unwilling guinea pigs to billionaires’ dreams/goals.

we'll adapt, ie. adjust our behavior to account for that new factor. Police for example have already learnt how to pretty safely stop a Tesla on autopilot with a driver sleeping behind the wheel and not reacting to any signals (because of being deadly drunk for example).

benj1112• 7 years ago

I like comments where you can't tell if the author is defending something, or absolutely condemning it.

breakyerself2• 7 years ago

Willing guinea pig here. I'm going to do my best not to die, but I'm excited about this tech and willing to deal with the drawbacks of being an early adopter.

lambda_lover• 7 years ago

It's great that you're willing to die for Tesla's profit, but you realize that autonomous vehicle-induced crashes affect everyone on the road? Even someone who doesn't consent to Tesla's TOS is still sharing the road with potentially dangerous software.

Granted, individual drivers are awful enough that it probably doesn't make that huge a difference in danger. But would you still feel the way if a family member was killed in an accident where their human-controlled car was rear-ended by a Tesla?

jseliger• 7 years ago

Right now 30,000-40,000 people die in car crashes annually: https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in.... And those are just fatalities: hundreds of thousands more are injured. How do we deal with those presently?

We just accept that amateurs should be hauling around at high speeds in several thousand pounds of missile.

The most relevant question is whether Tesla AP is safer or less safe than typical amateur drivers per 1,000 vehicle miles driven. I don't know the answer to that question.

mslev• 7 years ago

Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split.

Does AP use other cars as reference points, or just the road? Ideally in this situation it would be both: "The line has disappeared, and there's a new one now, but that car went over it". Instead it seems to just be following whatever lines it can see. Does that make sense?

Note- not at all defending the AP behavior here. Just thinking out loud.

treis• 7 years ago

Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split.

It seems like it's failing in different ways:

Try 1- Toughest to tell, but it looks like it failed to recognize any lines. Kept going straight which was at the barrier. Hard to tell if the car would have recovered.

Try 2- Looks like the car tried to go left into the closed lane. Seems like an error in detecting the barriers closing the road. I'd guess that it would have avoided the concrete barrier and driven down the closed lane

Try 3 - This one looks like it picked the wrong lane marker to be the left side of the road. In that it thought the right lane marker of the closed lane was actually the left lane marker. This one probably ends up with a smashed car and dead driver.

Tomte2• 7 years ago

The comments are scary. People defending Tesla's misleading advertisement re: full self-driving hardware etc.

Because clearly "having full self-driving hardware" only means the hardware is there, not that the car can actually self-drive.

I bet those people find themselves misunderstood in pretty much every discussion with people outside their nerdy circle.

mikejb• 7 years ago

On the earnings call, Musk even claimed that the Navigate on Autopilot feature set is "full self-driving" [1]

[1] https://youtu.be/oB0sAGykWT8?t=2895

tbabb6• 7 years ago

I have been yelling about this for a long time: Tesla is not going to be able to deliver full self-driving as promised. They don't have the hardware, for one (ranging is terrible; they need stereo cameras), and second, their software strategy seems to be a dead end.

They need sensor fusion. The system needs to make maximum use of all the information available to it: Where is the road striping? Where are the other cars going? Where are the road signs and signals? (If there's one in your path, you certainly shouldn't drive into it!) Are there camera-visible obstructions? What were the interpretations and actions of previous Tesla trips along the same route?

In these problem cases, all data except the left and right lane striping seems to be completely ignored. There was even more information at the fatal offramp location (cross-striping over the lane separation zone), which the vehicle drove straight over. The system is not making maximum use of the information available to it, in fact it is using hardly any of it at all, and fixating on what it thinks is a single most salient piece of data.

Sensor fusion algorithms tend to behave the opposite way-- each additional piece of data informs the interpretation of all the other data. You can have very poor-quality data, but if it is even moderately over-constrained, your state estimate can be very good in spite of it. I think it would be completely reasonable to have a neural net in the loop of a sensor fusion algorithm, with fusion constraints informing the NN's interpretation, and the NN's estimates feeding back into the fusion algorithm as uncertain data.

IMO Tesla will do at least one of:

* Very expensively retract their promise of full self driving for delivered vehicles

* Completely overhaul/redesign their driving software and start again nearly from scratch

* Get into a regulatory/legal tangle with the NTSA/courts/DOJ over all the dead people their system is making.

Dumblydorr2• 7 years ago

Where is your evidence for these claims?

eclipxe2• 7 years ago

They do have stereo cameras and sensor fusing, as well as detecting more than just the lines in the road. Here is what the camera sees: https://www.youtube.com/watch?v=rACZACXgreQ

tbabb6• 7 years ago

What in that video suggests sensor and/or stereo fusion to you?

I notice that the temporal coherence is pretty bad-- Pedestrians pop out of recognition when they go behind trees; lane/exit boundaries wiggle all over the place and occasionally frame-pop into different configurations. A kalman filter, for example, is a state estimator which maintains temporal coherence, and makes heavy use of previous estimates/sensor inference when computing the most updated estimate. It doesn't look to me like that kind of strategy is being used to maintain the vehicle's world model. IMO a good estimator wouldn't treat "a pedestrian popping out of existence" as the most likely estimate for any circumstance, let alone one where they were clearly present in the previous 50 frames. I don't doubt they're using KF on the vehicle's inertial movement, but based on the failures and this video, it sure doesn't look like it's using a fusion technique for the world model.

There are left and right-looking cameras, but the FOV overlap between them is not very substantial, and there can't be stereopsis where there is no overlap. Per the Tesla website, there are three forward-looking cameras, and they each have a different FOV. The parallax baseline between them is only a few centimeters, too, so the depth sensitivity isn't going to be spectacular. It's certainly possible that there could be some narrow-baseline stereo fusion, but it could only really happen inside the narrowest field of view, where the coverage overlaps with more than one camera. That's the circumstance where having a narrow baseline would hurt the most. Based on that it doesn't really seem like the system is well set-up for stereopsis; if it's there it seems like an afterthought.

I could certainly be wrong, as I don't have access to the code. Are you going by some other secondary source/information?

gamblor9565• 7 years ago

Stereo cameras overlap. Tesla's cameras are intended for 360-degree coverage, not overlapping vision.

tim3335• 7 years ago

Musk is more optimistic than you - says late 2020 for sleep while the car drives https://www.youtube.com/watch?v=Y8dEYm8hzLo&t=10m25s

Also quite interestingly he says there will be a big jump forward in quality when they switch to their own computing hardware (18m40 or so)

FireBeyond5• 7 years ago

He's also said 2016. And 2018.

x38iq84n5• 7 years ago

late 2020 for sleep while the car drives

Musk is often overly optimistic and he keeps underestimating the problem at hand. I call BS on this, it won't be ready in 5 or even 10 years. And then there is a regulatory approval.

madeofpalk• 7 years ago

Musk is more optimistic than you

Indeed.

What Musk says and what Tesla delivers are two completely separate things.

ec1096855• 7 years ago

I think you are confusing what the software is capable of today with hardware limitations in the sensors deployed in the cars.

systemspeed• 7 years ago

With demand for self-driving vehicles as high as it is, yet with the lag in advancement of computer vision, I think it's about time for a serious discussion about smart roads. I realize this isn't entirely relevant, but I can't be the only one thinking that we're missing the forest for the trees by trying to solve a transportation problem while simultaneously solving a vision problem.

progfix• 7 years ago

Might as well put them on rails (Smart-Rails(TM)) and make a railway network for smart people. Transportation problems solved!

vkou• 7 years ago

So, trains?

ummonk2• 7 years ago

Maybe it's merely that I'm used to cones rather than barriers of that sort, but that stretch of road looks rather confusing to me as a human as well.

syntaxing2• 7 years ago

Tesla should just partner or acquire a LiDAR startup already like Baraja [1]...They can let the computer vision do all the magic they want. Just have one scanning LiDAR as a redundant system so that it doesn't run into anything in front of the car.

[1] https://www.baraja.com/

ip26• 7 years ago

I don't know why, but supposedly they've made "no LiDAR" their hill to die on.

syntaxing2• 7 years ago

Yeah, it's such a weird mentality. Their new patent on a "CNN" ASIC is pretty neat but still doesn't solve a lot of their problems. Are they banking on a magical depth CNN architecture to be released or something?!

edshiro• 7 years ago

Costs. Installing LIDAR will significantly hike up the price of their vehicles.

newnewpdro• 7 years ago

I find it absurd how the NHTSA doesn't treat Tesla similarly to how it treated Toyota ~10 years ago with the sudden unintended acceleration controversy. [1]

Tesla has demonstrably flawed and dangerous vehicles operating on our roads. These persistent autopilot bugs are killing people. If we treated the Tesla autopilot as a licensed driver, it'd have its license suspended. (Not that it could have passed a driving test in the first place)

[1] https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic...

DarmokJalad17013• 7 years ago

These persistent autopilot bugs are killing people.

Are there any numbers on this?

GoToRO• 7 years ago

That's one way to keep drivers alert I guess... Add a little bit of randomness into your boring drive.

lgleason• 7 years ago

Tongue in cheek......this is planned obsolescence. New model with upgraded features is announced so you start to brick the old ones by making them run into things. :) Of course there is that pesky issue of it potentially injuring/killing the driver and passengers....

In all seriousness, these things are getting better, but Skynet is not here yet. Just because a company, governmental agency etc. says something is safe you always need to evaluate things yourself..and it's probably not a good idea to rely on these auto pilots unless you want to win a Darwin award. I love the cars, but the hype is a bit annoying around stuff like this.

perfunctory• 7 years ago

It troubles me that we use the term "software bug" to describe this.

mcguire• 7 years ago

One might suspect, given the reintroduction of the bug, that Tesla doesn't understand their code.

erobbins2• 7 years ago

I think Tesla eventually has a catastrophic accident and is sued and/or criminally prosecuted into oblivion. I feel sorry for the people who are going to have to die for this to happen.

This trope that humans are bad drivers is, in general, crap. Humans are very good drivers. The US has 7.3 deaths per billion km driven. This means if you drive 50km a day, every day, you are (essentially) guaranteed to die... after 7500 YEARS. You have less than a 1 in 10 million chance of dying on any given trip you take. That is NOT risky, and is NOT dangerous.

Joe-Z• 7 years ago

Cars have gotten so safe though. There's all this measures put in place to ensure your survival _in case of a crash_. Wouldn't it be more relevant to calculate the chances of getting injured / end up in a crash at all?

I mean just being caught up in a traffic accident and have no bodily harm done to you can be a traumatic event.

mdorazio• 7 years ago

Yes, this is a very important distinction. Humans have gotten pretty good at not getting killed in car accidents, mostly due to increasing safety engineering and assistance features in cars. Humans are not very good at avoiding collisions entirely. In 2016 there were 6.3 million motor vehicle accidents reported to police. And NHTSA estimates about 10 million accidents go unreported each year (mostly minor fender benders and people damaging their own vehicles on fixed objects). To me, that's pretty clear evidence that humans are poor drivers.

erobbins2• 7 years ago

You're not wrong. There are plenty of accidents caused by inattention and poor driving that aren't fatal, but that do result in injuries... a quick search didn't give me much information but crashes with injuries appear to be about 100x more common than fatal crashes. This doesn't take into account how serious the injuries are, however.

So if we assume that serious injuries are 10x more common than fatalities, that raises the odds to one incident per 750 years, or 1 incident per 75 years of any accident at all. You're still talking about things that happen to people once or twice in their entire lives. Of course there will be outliers (I've been in 5 accidents myself, only 1 with injuries) but the odds of being in a crash don't seem to be worth the risk of trying to automate mousetraps... we should concentrate on replacing car-based transportation instead of trying to use magical black boxes to make it theoretically safer.

matz1• 7 years ago

I hope not, yes there will be death, nothing is perfect, but i will just consider it an acceptable loss.

derefr• 7 years ago

The US has 7.3 deaths per billion km driven.

This is a meaningless measurement. Segment those km by where they occur. Most of them are either:

1. "cruise-control compatible" kilometers—e.g. freeway straightaways—where you're surrounded by cars but they're all going the same speed in a straight line, and all you need to do to be safe is to go the same speed in a straight line as well.

2. "closed-course" kilometers—e.g. most rural roads, and most suburban roads any time other than rush-hour—where the road may curve and have intersections and such, but at any given time of day, it's probable that there aren't any other cars (or even pedestrians) on the road for you to collide with, no matter how bad your driving is. (Think "roads you'd have a teenager practice driving on." These roads are good for practice because there are effectively no accidents to get into.)

3. (a smaller segment, but still relevant because of the number of freight kilometers driven here:) "empty" kilometers—this has all the properties of segment 2, but also, the road is at grade, and there's nothing abutting the road (i.e. the road isn't a street), so even if you veer off the road, you're unlikely to hit anything. (Examples: the Nevada desert; Saskatchewan; most farmland.)

People point out that the safety-per-km stats for airplanes are a nonsense measurement, because what little crashing that airplanes do tends to mostly occur during the first and last 50km of the planned flight-path—so short flights are just as dangerous as long flights.

Well, the same goes for car accidents. Subtract all the "trivial" driving that humans and AIs can both do by just doing... nothing much at all, with no obstacles/hazards to evaluate, let alone react to. The kilometers that are left (freeway merging; city driving; suburban streets during rush-hour; parking in parking lots) are a lot more crash-y, and are the place where both human and AI competence is questionable.

kirse2• 7 years ago

I feel sorry for the people who are going to have to die for this to happen.

I try to explain this to friends who are far too optimistic on self-crashing car technology. Self-driving cars (SDC) ultimately trade one class of problems that result in death (human-attention deficit) for another class of growing issues (sensor malfunctions/incapabilities, software defects).

Ultimately SDC deaths end up as bugs/features on some random devteam's backlog, and I have no desire to have a JIRA ticket named in my honor.

In my opinion, by the time all the money and effort is spent making SDC's capable of successfully driving from point-A to point-B in the near infinite possible conditions they could encounter, it would have been 50x cheaper to simply build a fully modernized high-speed rail network over existing highways and roads.

1123581321• 7 years ago

You are saying that the cost to develop successful self-driving cars is 1-50 quadrillion dollars using an optimistic estimate of rail costs. That does not seem reasonable. Perhaps 200 billion across all companies have been invested in self-driving cars so far (I.e., Waymo is just a fraction of that.)

albntomat0• 7 years ago

Ultimately SDC deaths end up as bugs/features on some random devteam's backlog, and I have no desire to have a JIRA ticket named in my honor.

Instead you can be the victim in a vehicular manslaughter case due to DUI/texting/etc.

As stated elsewhere on this thread, self-driving needs to be measurably better than a human. My state displays how many people have been killed thus far this year on the automatic traffic signs (used for amber alert, traffic info, etc as well). A 20% reduction in the 3000+ folks killed in 2018 would mean a whole lot for those saved.

tzs• 7 years ago

Self-driving cars (SDC) ultimately trade one class of problems that result in death (human-attention deficit) for another class of growing issues (sensor malfunctions/incapabilities, software defects).

One big difference between those classes is what you can do after an accident. With both, you can investigate why it happened and then make recommended changes to prevent it or reduce its chances in the future.

But with driver attention problems, such as drunk driving or driving while texting, it is easy for those recommendations to be ignored. We've been telling people not to drive drunk, not to text while driving, and so on for probably a century in the case of drunk driving, and for as long as texting has existed for texting...and people still do them frequently.

With a software defect, it is a lot easier to make sure that the fix actually gets deployed. Make it part of the annual registration renewal for cars that all safety updates have been applied to their self-driving systems.

mr_toad• 7 years ago

it would have been 50x cheaper to simply build a fully modernized high-speed rail network

Where do you get that number from?

mandeepj• 7 years ago

This trope that humans are bad drivers

Humans are bad in general. At least, self-driving cars will not be texting or using their phone while driving or involving in road rage. This list is really long.

"Nearly 1.25 million people die in road crashes each year, on average 3,287 deaths a day. An additional 20-50 million are injured or disabled. More than half of all road traffic deaths occur among young adults ages 15-44"

https://www.asirt.org/safe-travel/road-safety-facts/

MSM• 7 years ago

You're looking at it strictly from a safety perspective which is a small part of the whole picture. If we looked at transportation from a safety perspective throughout history, we'd still be walking everywhere. What's the risk of crashing and dying while walking? Zero; it's fully optimized for safety. We rode horses and now drive cars to save time. Driving to work is unquestionably a better experience than riding a horse into work every day, but it could be better- we could be sleeping on our way into the office, or starting our work day as we leave our driveway, or eating a good breakfast and finishing our makeup (while not endangering everyone else).

Having commuting time available as what amounts to "free time" is an insane boost to daily life. A quick google says Americans spend 12.2 days per year in their cars. If 300 million people can save 12 days' time per year, you're freeing up ten million years of time every year.

I'm not advocating for throwing everyone in self driving cars untested and who cares how many people die, but if, on the journey to saving many millennia of time every year, a person is killed, why should the company be sued into oblivion? If we're going to sue everyone into oblivion whenever anything doesn't go quite right, why would any company ever try to take on difficult problems?

perl4ever4• 7 years ago

"What's the risk of crashing and dying while walking? Zero; it's fully optimized for safety."

Huh? Have you never seen or heard of anyone falling, hitting their head, and causing a concussion and/or death?

deevolution• 7 years ago

Certainly we can do better than this tho? Your bar for very good driver" is pretty low. I'm sure everyone has at least one acquaintance they know who has died in a car crash. I can count 3 who have died from car crashes and 3 others who have been seriously injured. Humans arent going to get any better at driving. Something has got to change.

m4633• 7 years ago

yet the leading cause of death in youths is automobile accidents:

https://upload.wikimedia.org/wikipedia/commons/a/a5/Causes_o...

perl4ever4• 7 years ago

Well, we don't need self driving cars to fix that, just a way to increase the fatalities from cancer, heart disease, and Alzheimers in the young.

ec1096855• 7 years ago

30k deaths a year in US, 50 years of driving. That is 1.5M deaths over your lifetime. Let’s say there are 690M drivers throughout your lifetime. You have a quarter of a percent chance of driving being your cause of death.

Also, 2.45% of deaths worldwide are from road accidents: https://ourworldindata.org/grapher/share-of-deaths-by-cause-...

jtaft• 7 years ago

Can they simulate their sensors input pretty well? Do they use car driving simulators (video-game like) to see what would happen in similar scenarios?

Of course, real world testing would still be necessary.

jaimex23• 7 years ago

When a customer disengages or corrects AP that periods sensor data is recorded and uploaded to Tesla. The uploads are curated and submitted into the neural network training data-set.

They basically have an infinite amount of real world data coming in.

alanh2• 7 years ago

My Model 3 suddenly changed lanes today for no discernible reason. I think it considered my lane to jump over into the next lane, for some reason. I should have hit 'Record' to save the footage from TeslaCam.

(Notably, AutoPilot is not supposed to change lanes without explicit confirmation from the driver, which is clearly illustrated on the dashboard/panel.)

It's a stretch of road on which I have previously used AutoPilot many times.

gpm• 7 years ago

Recently I accidentally changed lanes in an intersection. The road had 3 lanes in each direction (+ separate streetcar tracks down the center), including the right lane which was required to turn. I was in the center lane. After the intersection the road was still a 3 lane road - which didn't quite register on my brain. I moved from the center lane to the right lane (figuring it exited so I was supposed to be in the now-right lane) when I should have remained in the center lane.

While I am a novice driver, I've gone through that intersection before without blinking or doing the wrong thing. It's not a particularly complicated intersection.

Anyways, point is, driving is surprisingly hard. I think counting anecdotes on the internet probably gives you a sample heavily biased against Tesla, because most people don't go post "so I did this stupid thing" but they do post "so my car did this stupid thing".

alanh2• 7 years ago

My car and I are stupid in different ways. It has perfect attention and will always brake in time to avoid a fender-bender in traffic. It’s also a lot worse than I am about reading and communicating intent with other drivers or simply recognizing the objects around us.

cmurf• 7 years ago

So far Tesla automation seems to handle edge cases. I have never rear ended or side swiped anyone, getting into such an accident or avoiding one is an edge case for me. Useful but not really automation.

For it to even have a chance during lane keeping of doing something only a sleeping or suicidal driver would do? It's utter bullshit. It's less safe than a human.

Autonomous driving is overhyped vaporware. Autonomous airplanes would be easier than cars, and we still are no where near that.

gwbas1c2• 7 years ago

The video really doesn't convey what happened. It's not really clear when the driver took over.

My Model 3 is very odd when going through forks like this. It just has trouble picking one side of the lane or the other. Yesterday I got a little scared when it swerved back and forth while trying to figure out how to take an exit.

jaimex23• 7 years ago

Warning: Autosteer is intended for use only on highways and limited-access roads with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer on city streets, in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death.

https://www.tesla.com/content/dam/tesla/Ownership/Own/Model%...

kurtisc2• 7 years ago

And then... https://www.businessinsider.com/elon-musk-breaks-tesla-autop...

steelframe• 7 years ago

The day I took delivery of my 2016 Model X with AP 1.0, Tesla announced AP 2.0. A friend of mine immediately ordered a Model X with AP 2.0 and rubbed it in my face.

For the entire next year, my AP 1.0 (which is non-Tesla technology -- Mobileye rocks) had no trouble doing adaptive cruise control and lane assist. Meanwhile his AP 2.0 would brake suddenly and swerve all over the place. It took a full year of OTA updates before his AP 2.0 was finally on-par with the functionality that I had the whole time. Of course, by then Tesla pulled a "we're sorry, but the princess is in another castle" and came out with AP 2.5.

Now this kind of stuff doesn't matter to me. I got tired of that company's shit and have pulled out of the Teslasphere entirely. I'm now driving a non-Tesla EV, and I'll never look back. I'm also letting my government representatives know that they should support a common EV charging standard and keep Tesla so-called "self-driving" shit off public roads.

dreamcompiler• 7 years ago

I used to be a big supporter of Tesla, but I now have to apply Internet-of-Shit Rule #1 to them: No software updates ever without explicit owner approval.

And "owner" is me, not Elon Musk.

sandos• 7 years ago

What I dont understand is, that yellow sign is actually very visible compared to other things the neural nets can detect. Why hasn't it been done yet? Detecting things as visible as that sign should not be a huge problem.

After 6 months of working fine, Tesla software update drives at barriers again

Related Stories