Tesla Full Self Driving Beta Causes a Crash

https://twitter.com/mostlyharmlessz/status/1469421762555351042

by camjohnson26 • 5 years ago

121 57 5 years ago

SQueeeeeL• 5 years ago

My amateur interpretation of this is that the vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line in the road, which caused it to believe it was seriously out of lane. After verring, the algorithm reoriented, and realized an impact was imminent and diverted off the road.

I think what's interesting about this example is that in this case, an alert human driver would not be confused in this way. It shows issues with the particular machine learning approach to automated driving.

jarolinvargas• 5 years ago

The driver intervened and lost control. It looks like Autopilot was going all in on that incoming vehicle.

rightiousrob• 5 years ago

I don't think that a root cause analysis like this is good for anything else than improving the algorithm.

It's the same as with face detection. The algorithms are better far superior to humans, but they fail at occasions where a human would not even think twice

rasz• 5 years ago

vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line

not the lights, but the vertical lens flares captured by single front facing center mounted camera cluster - afaik Tesla still has no stereo vision.

the algorithm reoriented

more like driver jerked to the right while having a heart attack and seeing itself going into a full frontal collision thanks to move fast and break things mentality

catlikesshrimp3• 5 years ago

Wow, you just slapped me with this idea:

  - What should be the core tenet of the self driving system for a car?
  - Why, of course it should be "Move fast and break things"!

rvz• 5 years ago

Oh dear. FSD (Fools Self Driving) Beta once again causing havoc and putting the life of this driver at risk and deciding to hit another car after the driver regaining control of the vehicle very quickly and going off road, especially in the most dangerous time to drive, which is at night.

Had the driver not kept their eyes on the road or was sleeping in the car, this system would have crashed into that incoming vehicle. This video just proves that night-vision is non-existent in this 'safety-critical' system.

Outrageous.

cblconfederate• 5 years ago

Thankfully we have no teslas here yet, but i gotta watch out when they arrive. Is it possible to tell if a tesla has self-diving?

devortel• 5 years ago

Not from the exterior. When full self driving beta mode is active, there is a circular, blue steering wheel icon visible at the top of the car's control screen. So unless you're inside the Tesla, you won't be able to tell.

bdcravens2• 5 years ago

I don't think we are far from legislation being required to mandate some sort of indicator. (if not outlawing these features)

rsynnott2• 5 years ago

I would not bet on this continuing to be the case, mind you. If regulators allow this sort of thing to continue at all, I'd expect warning lights visible to pedestrians and other drivers will be required.

zardo5• 5 years ago

If you have to be at the ready to take over from the car trying to kill you at all times, wouldn't you rather just drive?

tshaddox2• 5 years ago

That depends on the probability (say, per mile) that you’ll need to take over.

faeyanpiraat• 5 years ago

A lower probability is even worse psychologically imho.

zaphirplane2• 5 years ago

Objectively only If you perform statistically better than the automatic system

zardo5• 5 years ago

So far so good. I suppose there are a few acute medical situations that might cause me to fuck up driving on par with what FSD can manage to do.

whateveracct2• 5 years ago

Most accidents are even due to human error (even if they are reported as such) - it's self-driving propaganda fodder.

https://amp.theatlantic.com/amp/article/620808/

IshKebab• 5 years ago

Which you almost certainly do today.

olyjohn• 5 years ago

The accident happened so fast. How would anybody be able to take over fast enough if your hands aren't hovering directly next to the wheel?

psyc• 5 years ago

It does seem like I'd have to maintain the same level of alertness and attention as when driving, in order to have the continuity of awareness needed to react. Including having my hands on the wheel.

zardo5• 5 years ago

You have to keep your hands on the wheel, but even then it's not really a motor control issue, the problem is attention and surprise.

kelnos• 5 years ago

Can we change this to link to the original video, at https://www.youtube.com/watch?v=7VhrG-7SBZg ?

Video description contains details about what happened, which might not be clear from the video snippet. FSD swerved into oncoming traffic, and the driver manually pulled back to the correct lane, but lost control and went off the road.

pja• 5 years ago

That video has been removed by the uploader according to YouTube.

AtlasBarfed• 5 years ago

Is there corroboration that this actually is self-driving? I suppose we'd need Tesla's cooperation. Is it possible to replay the "AI view" of the same thing? Can any FSD beta users chime in?

I will say that two lane highways (aka offset impact at 110mph+ IF/BIG IF they aren't speeding) are pretty damn dangerous and I'd only do self-driving on them with a LOT of repeats. I think I'm getting close to the point of trusting self-driving on divided highways under good conditions, the big caveat being stopped cars/obstacles.

It sure doesn't help that every Tesla post on HN has comments that seem like obvious astroturfs. It's like jalopnik in the comments.

Transparency is a must. The new NHTSA head doesn't help, we do need someone to hold Tesla's claim to the light, but appointing someone with publicly posted anti-Tesla screeds? That doesn't help anything.

Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year and be a legitimate boon to the economy. NHTSA needs a dedicated group of government officials with legit industry and software experience to test, regulate, and analyze incidents. Plus it would be apocalypse to one of my less favorite sectors in the world: the auto insurance industry.

NHTSA should have immediate delivery of Self-Driving accidents with all the footage and any recorder information immediately uploaded to government servers with an active software harness to replay the logs. And it's not like this would be occaisionally tested, even with high-quality theoretical self driving, there will be daily accidents to replay from every significant vendor.

catlikesshrimp3• 5 years ago

"Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year..."

Launching a self driving car which crashes head on into traffic is not a good start. If they used unmaned Teslas for the live traffic tests they would be risking less lives. /s

It is interesting that consumers kind of volunteer to test a potentially lethal product. I say "kind of volunteer" because they are actually paying for the car.

sidibe2• 5 years ago

Not just the car, they are dropping an additional 10k to be specifically part of the test.

avereveard2• 5 years ago

impact at 110mph

can we get the "frontal impact == impact at sum of speed" myth die already? it's been thoroughly debunked over and over. the only way it's relevant is if there's a large weight difference between vehicles.

siddarthd2919• 5 years ago

It looks like auto highbeam assist turned of and it might have caused a weird vision bug

bdcravens2• 5 years ago

There's just something I don't like about the word "bug". It suggests a condition of the software, not a condition of the developer. Maybe it's the full phrase "caused a ... bug". As if the bug wasn't present until the condition occurred. Unless that condition is actually altering the commands in the software, the bug was already there. I could even talk about what "weird" means in this context. It's not a poke at you - I heard phrases like "caused a weird bug" all the time in our industry. In generally, and especially in situations involving safety, it's important we not engage in minimization, but call it was it is: a flaw, where conditions in the operating environment weren't properly considered.

mirker• 5 years ago

Well, given the “flaw” is a machine learning model, who was the developer and how do they relate to the training data?

josephcsible• 5 years ago

I'm suspicious that this may not have occurred as described for 2 reasons: the recording doesn't show the part of the screen where the autosteer indicator is, and the original uploader took down the video as soon as it got significant attention.

camjohnson262• 5 years ago

The original poster gave more details here: https://teslamotorsclub.com/tmc/threads/fsd-beta-attempts-to...

a-dub• 5 years ago

that wasn't a crash! the car simply decided that the turn was dangerous and took it upon itself to disrupt the road construction industrial complex by forging a new and safer path.

sidibe2• 5 years ago

It doesn't surprise me that it's this particular scenario, I've seen a lot of Youtube videos of them going way over the center line on turns in this kind of road (2 lane rural with 55 mph) even in broad daylight

chapchaps• 5 years ago

Don’t forget with this sensational “headline” we don’t actually know the truth. We can’t see the driver in the video and we have only his word to go by. Statistically it’s much more likely he’s lying versus FSD causing the accident in this situation. FSD has a long way to go but it’s already demonstrably safer than humans.

Regardless, Hope this guy is okay and wasn’t injured in the accident.

ivraatiems• 5 years ago

Sorry, but that isn't true of Tesla's FSD. It's notoriously unsafe. Here's some evidence:

https://www.motorbiscuit.com/tesla-beta-testers-say-full-sel... https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-... https://www.roadandtrack.com/news/a35878363/teslas-full-self... https://electrek.co/2021/11/15/tesla-serious-phantom-braking...

https://www.businessinsider.com/tesla-full-self-driving-beta... (with video!)

Or just look on Youtube for videos of people using the FSD, and see how frequently they have to stop it from doing something dangerous.

jondwillis• 5 years ago

For what it is worth, I want to try FSD beta, and have a safety score of 99, and can’t.

You have to jump through so many hoops to even have FSD beta right now, that this is the devil’s deal (i.e. potential manslaughter) you accept by even trying it.

Tesla should investigate, and continue to keep FSD in beta with these strong obstacles while it improves. And eventually (arguably), it needs to put up or pay up for level five FSD which it has been selling as vaporware for something like 5 years now. Maybe this is where public and private law needs to step in.

But in my opinion, casual outrage over FSD is unwarranted until Tesla rolls out software that is less safe than humans to a lot of its users that will operate it as if it is level 5 regardless of actual performance.

dzhiurgis• 5 years ago

Spend 10k on FSD tip jar

Fail to buy real tyres

Or is Tesla’s stability control so bad that it underperforms 2 decade old economy cars?

jaimex2• 5 years ago

It was fake. Delete the post.

Borgz• 5 years ago

Evidence?

irthomasthomas• 5 years ago

There are loads of videos of Tesla self driving fails, like this one posted yesterday, where the Tesla tries to hit a pedestrian and prevents the driver from taking control, tries to drive on the train track, and stops for the Wells Fargo sign.

https://mobile.twitter.com/TaylorOgan/status/146940457943982...

rsynnott2• 5 years ago

Er... Yes, this definitely seems like it'll be ready to use any day now. Bloody hell, people _pay_ for this?

oxplot2• 5 years ago

Flagged this. Some random video with no official investigation and countless speculations.

Keep in mind that Tesla will sue your ass for defamation if you post BS video and lie about it. Just ask the Chinese dude who posted about Tesla accelerating by itself into a crash. Turns out he stepped on the pedal himself but of course when someone on twitter posts about a Tesla fuck-up, it’s taken as gospel.

AbrahamParangi• 5 years ago

The degree to which people are willing to jump to the defense of Tesla self-driving is truly incredible.

The experience of “your Tesla tries to do something INSANE” is shared by literally every Tesla driver I know.

oxplot2• 5 years ago

Yes, I’ll happily jump to Tesla’s defense (not that they need it) when baseless crap is thrown around, because for every 1 of me, there are 100 who can’t wait to see Tesla crash and burn.

Tesla Full Self Driving Beta Causes a Crash

Related Stories