Tesla FSD Beta 10 veers into pedestrians with illegal right turn

https://twitter.com/TaylorOgan/status/1438141148816609285

by jeffbee • 5 years ago

88 119 5 years ago

nodesocket2• 5 years ago

2021 Model Y Owner. Hard to tell, but it looks like that street is a one-way heading right which is legal. However, not yielding for pedestrians is no bueno though they were just barely off the sidewalk into the street when it turned.

jdminhbg2• 5 years ago

It's a one-way to the right but there's a no right turn sign on the pillar to the left of the street as you approach it. Not signed great, but still not a legal turn.

notJim4• 5 years ago

I think that sign applies to the left lane only. It's to stop you from turning right from the left lane. I think turning right from the lane the car is in is legal.

benjaminjackman• 5 years ago

There is sign on the wire near the traffic light that says no turns, not sure if it only applies to the other lane though.

gpm7• 5 years ago

There's also a sign on this half of the traffic wire that looks like it might be saying "yes turns" (low resolution, but I can't imagine what the sign says with an arrow pointing right other then that)

nodesocket2• 5 years ago

Ahhhhh I see the no right turn sign on the pillar. That is confusing as there are also one-way signs (with arrows pointing to the right) further along.

Moral of the story, do you think most humans would do a better job at this intersection than the Tesla did?

rocketraman• 5 years ago

There is a no right turn sign on the post to the left of the vehicle just before it begins the turn.

obilgic2• 5 years ago

Misleading/Exaggerated HN title. Not an illegal turn. No turn sign applies to the left lane, and it's a one way street

bagels7• 5 years ago

Agree, it didn't hit anything, even though it might've without intervention. "into" is the misleading part.

gpm7• 5 years ago

Right, and it didn't "veer" either, it performed a normal turn.

The pedestrians had just started crossing, so it was in the wrong, but barely. It's the sort of manoeuvre that you see human drivers do all the time.

BugsJustFindMe12• 5 years ago

Not an illegal turn.

[edit] I linked to the law from the wrong state. Here's the right one:

https://app.leg.wa.gov/rcw/default.aspx?cite=46.61.305

RCWs > Title 46 > Chapter 46.61 > Section 46.61.305

(1) No person shall turn a vehicle or move right or left upon a roadway unless and until such movement can be made with reasonable safety nor without giving an appropriate signal in the manner hereinafter provided.

(2) A signal of intention to turn or move right or left when required shall be given continuously during not less than the last one hundred feet traveled by the vehicle before turning.

Every other state has similar laws.

The Tesla was not signaling before or during that turn. That on its own makes the turn illegal.

obilgic2• 5 years ago

I don't think that is California though..

shepherdjerred2• 5 years ago

This is pedantic. Human drivers routinely don’t use their turn signals.

gruez• 5 years ago

Isn't that just "failing to signal a turn"? If you told me someone made an "illegal turn", I'd think they made right on a intersection with a non right turn sign.

loeg5• 5 years ago

California Vehicle Code

This is Seattle, under the monorail (5th Ave).

tzs2• 5 years ago

It looks like any entry of the car into the crosswalk would be illegal at that point. RCW 46.61.235, "Crosswalks", states:

(1) The operator of an approaching vehicle shall stop and remain stopped to allow a pedestrian, bicycle, or personal delivery device to cross the roadway within an unmarked or marked crosswalk when the pedestrian, bicycle, or personal delivery device is upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning. For purposes of this section "half of the roadway" means all traffic lanes carrying traffic in one direction of travel, and includes the entire width of a one-way roadway.

The street it was turning and/or veering into is a one-way street abd so the entire width counts as "half of the roadway" and thus any pedestrian in the crosswalk counts as being "upon or within one lane of the half of the roadway upon which the vehicle is traveling or onto which it is turning".

You can only drive into the crosswalk if it is a two-way road, no pedestrians are in the part of the crosswalk on the side that you are driving, there are at least two lanes going the other way, and none of the pedestrians on the other side are in the lane on that side that is adjacent to your side.

gallerdude• 5 years ago

I don’t know how the UX works, but it seems like it de-activated itself when it got closer to the pedestrians? Or was that a manual override?

bagels7• 5 years ago

If you hit brakes or steer, it disengages.

awsthro009452• 5 years ago

The video in the Tweet replies is even worse, IMO: https://twitter.com/TaylorOgan/status/1437743602579423233

The Tesla FSD would've caused a t-bone in the first two clips, and a head-on collision at decent speed in the third clip. Yikes.

bagels7• 5 years ago

How long can they get away with charging 10k for something that is trying to crash the car in to bits?

sidibe• 5 years ago

Everything about FSD is truly mindblowing. They sold the product but are not letting most users get the version that tries to be autonomous (and thank god for that). It won't be available to most of these people on the car they purchased it for, and can't be transferred to other cars. That has to end in a refund or class action law suit right? And now they have a subscription available? That also doesn't include these features? Who are the suckers paying monthly for this, especially after years of failed promises about this product (that's giving the benefit of the doubt, I think "lies" is more appropriate at this point).

BugsJustFindMe12• 5 years ago

Shitty sudden turn against the no-turn sign without using a turn signal aside, why was it turning right at all when the route showed it going straight for another several blocks and then turning left?

GravitasFailure5• 5 years ago

I've dealt with similar but less extreme behavior in lane keeping systems where it senses the right-side reference veering away so it "corrects" and follows, causing a veer to the right, so it might not have been trying to turn but instead drive next to the crosswalk at the same distance it was from the curb. Regardless of what happened, that's a really bad bug to have.

BugsJustFindMe12• 5 years ago

Interesting. This could explain why it doesn't signal. But honestly even veering like that should elicit a turn signal to notify other drivers of the move.

diggernet2• 5 years ago

Nope. Watch the dotted line ahead of the car on the left side of the screen. At the last moment, it suddenly removes the "continue on this road" line, and replaces it with a "go down the road to the right" line. It literally decided to make a sudden turn.

You might also notice that while pedestrians and other vehicles are displayed on-screen, not a single one of the solid concrete pillars are displayed. That car is completely blind to them, and at any moment could have made a sudden LEFT turn into one of them.

Aliabid942• 5 years ago

If you have to anxiously hover your hands over the wheel constantly like the driver is in this video, is it really helpful even if it doesn't crash? I don't own a Tesla, curious how other drivers are when on Autopilot.

notJim4• 5 years ago

This is the city-driving beta that's only available to a small number of people. I have read that at this point, it is more stressful than driving yourself, but it is not yet at the point where anything more is expected.

alkonaut• 5 years ago

Assuming it stays in limited beta until safer than humans (statistically causing 1/10th as many accidents or some acceptable level like that) and until people actually feel measurably more relaxed when using it before it’s released to the public.

Who then believes it will even be this decade?

shepherdjerred2• 5 years ago

Test driving a Tesla Model 3 with autopilot last year was one of the most stressful experiences I’ve ever had. It’s not at all intuitive, and I had no idea what autopilot would/wouldn’t handle.

aclatuts• 5 years ago

It might be worth it for Tesla to build a voice system to tell the driver what manoeuvres the car will be doing soon.

Although implementing might be just as hard as self driving in the first place.

yakkityyak• 5 years ago

The cognitive load of using FSD in a city honestly seems more than driving yourself. The car is no longer an extension of yourself, where you navigate the road carefully. Instead you are constantly trying to predict and determine if FSD is going to screw up, and how to quickly react to said screw up.

camjohnson262• 5 years ago

The way it fails is non intuitive too, reposting this list of videos from below:

https://twitter.com/28delayslater/status/1437866328950517768...

https://twitter.com/BS__Exposed/status/1437367617468129280?s...

https://twitter.com/TaylorOgan/status/1437385266046586890?s=...

https://twitter.com/bagguley/status/1437328015789367297?s=20

https://twitter.com/TaylorOgan/status/1437405124004089863?s=...

https://twitter.com/BS__Exposed/status/1437043592560791552?s...

https://twitter.com/bagguley/status/1437048381042475008?s=20

https://twitter.com/BS__Exposed/status/1436806907663159299?s...

npteljes• 5 years ago

This is what I don't get at all about autopilot. It's fantastic that it can do X, but if I have to supervise and also take full responsibility, then what's my benefit? That I get to babysit the latest shiny tech? It either takes off the mental load of me paying attention to the road, or I'll drive myself.

bfung• 5 years ago

Recently got a Tesla without Full Self Driving (FSD) for many reasons (it's $10k for a full install as one). Autopilot is now like a product name that has several features and FSD is a different beast. Within Autopilot, there's an Autosteer (Beta) feature.

The Autosteer seems to work decently well on highway without hover hands, but it does require no sharp turns and any traffic or stop and go isn't what it was built for. I can def. see it nice when doing longer road trips.

That said, I've turned Autosteer on several times and maybe 70% comfortable with it. It does like to hug the right side of a lane, and when I'm in the very right lane on the freeway with on-ramps and exits, there have been times I didn't want to take any chances and assumed full control of steering, or else it drifts into the exit lane and gets back into the right lane, depending on the road lines and how they're drawn. This is with me having knowledge of how computer vision works - I wonder how anyone w/o tech knowledge can put their full trust into these things or to manage the situations correctly.

So even w/o the full FSD feature, I think there's still some work to do on the Autopilot autosteer (it could be a subset of FSD, maybe different ML models all together underneath) - and city driving FSD probably needs even more work.

https://www.tesla.com/support/autopilot#capability-features

drooby• 5 years ago

Level 3+ requires AGI for unmapped non-highway roads. Change my mind.

bagels7• 5 years ago

What is agi?

Aliabid942• 5 years ago

Artificial general intelligence, the ability to generalize the way humans can.

ddlutz• 5 years ago

artificial general intelligence

gpm7• 5 years ago

AGI implies things like "motivations" and "feelings" not just understand of the world (i.e. beyond understanding other things "motivations" and "feelings"), it's a fundamentally different problem. Even if you think you need extremely good world understanding to make a self driving car, AGI is irrelevant (and would actually make it much scary, because an AGI is perfectly capable of deciding to intentionally kill you).

singularity2001• 5 years ago

Most insightful comment my week.

Maybe we need a new catchphrase CAI comprehensive AI as an intermediate goal to AGI.

But then: maybe a car needs to understand the motivations of pedestrians and human drivers in order to predict their movements. And we are almost back to AGI requirements

camjohnson262• 5 years ago

Remember this is the build that Elon Musk said was almost ready for public release and would “blow your mind”, https://twitter.com/elonmusk/status/1435967157662150675?s=20

But there have been plenty of failures, and remember that the only people who have the beta are employees who are presumably under an NDA, and devoted fans who have publicly said they don’t want to post the disengagements because critics “take them out of context”

Other failures:

https://twitter.com/28delayslater/status/1437866328950517768...

https://twitter.com/BS__Exposed/status/1437367617468129280?s...

https://twitter.com/TaylorOgan/status/1437385266046586890?s=...

https://twitter.com/bagguley/status/1437328015789367297?s=20

https://twitter.com/TaylorOgan/status/1437405124004089863?s=...

https://twitter.com/BS__Exposed/status/1437043592560791552?s...

https://twitter.com/bagguley/status/1437048381042475008?s=20

https://twitter.com/BS__Exposed/status/1436806907663159299?s...

bagels7• 5 years ago

Looks like we are pretty close to robotaxi.

mensetmanusman• 5 years ago

It would be super interesting for Tesla to release context after these events, e.g. this is the 628,392,292th right turn this year.

ogwh• 5 years ago

Why is everyone freaking out over a beta test with a driver at the wheel ready to take over immediately? It's a beta test, finding bugs is kind of the point.

Or have we truly reached the point now where we expect "stable" software to be of the same calibre as beta software so nobody makes a distinction between the two?

woofyman• 5 years ago

Because it’s a 5000 lb weapon and the pedestrians didn’t consent to the beta.

thatswrong0• 5 years ago

I see people making similar mistakes pretty frequently. Not trying to excuse beta testing this software with live pedestrians (like, why on earth did it not signal? What was it's planning software doing?), but humans make this mistake too

ahahahahah• 5 years ago

In addition, and unlike every other serious company in the space, Tesla is using random untrained drivers for testing this crap.

colechristensen• 5 years ago

Because a car is a life critical system and nobody should want an automated vehicle, even if controlled by a human, to "accidentally" try to run over some humans. I doubt the pilot's ability to stop all missteps before an injury.

The car should be much safer before it goes on the road, regardless of test status. This was a near miss and should be taken extremely seriously by highway safety regulators.

Or have we truly reached the point now where we expect "stable" software to be of the same calibre as beta software so nobody makes a distinction between the two?

I expect every vehicle on the road to meet a safety standard. I do not accept "beta software" tests on public roads if this is the kind of mistake they can make.

dham2• 5 years ago

I've been using Open Pilot for the last 3 months (Rav4 Hybrid). I've really enjoyed it a lot on the highways. It also works really well in stop and go traffic.

I'm starting to agree with George Hotz about making driving chill. Highway driving is chill. Stop and go traffic is chill. There's nothing chill about letting a car drive in these situations (pulling out intersections, right hand turn on red, congested streets with people everywhere) and I'm used to my car driving itself. There's nothing chill about your car having unlimited torque on the steering wheel where it can do something gnarly. Making your car do these maneuvers has the opposite effect, making you more stressed rather than taking the stress away.

I'll stick to open pilot for now. I'd rather just drive in these situations. With Tesla FSD you already have to have your hand on the wheel, might as well put your foot on the pedals.

ProAm• 5 years ago

Which version of Open Pilot?

dham2• 5 years ago

0.8.8 in laneless mode. Comma 2. Decided not to return and get Comma 3 because 2 does everything I need it to do right now.

xiphias2• 5 years ago

I started to agree with George Hotz that a half autonomous car should try to drive like a human because that's what humans can superwise efficiently.

md2020• 5 years ago

Agreed. Just waiting on a good BEV that has full openpilot compatibility, maybe Hyundai Ioniq 5. Openpilot has a great community as well.

ruph123• 5 years ago

So from Googling, Open Pilot is more a fancy HUD and camera system that assists the driver but leaves all control with him/her? Did not follow Hotz for a while and the last thing I remember was that he was also just working on another full self driving startup.

improv32• 5 years ago

This is in Seattle, on 5th ave. The concrete pillars are the monorail track. The street confused me driving as a human sometimes. There are more videos of the Tesla software being confused on this particular street floating around

loeg5• 5 years ago

This street slightly confuses me, too, but I've never decided to suddenly and without signaling dive into the crosswalk when pedestrians were in it.

GravitasFailure5• 5 years ago

I thought that street looked familiar. There was a twitter thread[0] discussed[1] about two months ago showing a failure along the same street.

[0]https://twitter.com/giacaglia/status/1414605317841702914

[1]https://news.ycombinator.com/item?id=27811853

modeless4• 5 years ago

I believe that it is not illegal to turn right there. Both streets are one way, and the "no right turn" sign is for the lanes to the left of the pillars.

For context, these pillars in the middle of the road are handled poorly by Tesla FSD and this guy is repeatedly stress testing the system by forcing it to drive there, for YouTube videos. The latest FSD build is the first one that actually recognizes the pillars as obstacles; earlier builds would simply plow into them. What I'm speculating happened here is the new obstacle detection failed and decided that the road ahead was blocked, so the path planner decided it had to turn right despite that not tracking the planned route.

As far as the pedestrians, it looks to me like there was plenty of space to turn into the near lane without hitting them, and the pedestrians shouldn't have been starting to cross while the red hand was flashing, but of course a law abiding driver should yield in this situation. I expect many human drivers would choose to turn anyway as long as the pedestrians were far enough away as they appeared to be. But FSD should have stopped.

BugsJustFindMe12• 5 years ago

It is illegal to turn right there without signaling though.

What I'm speculating happened here is the new obstacle detection failed and decided that the road ahead was blocked

Know what would be able to tell if the road ahead is blocked? Literally any kind of three dimensional sensing that isn't just single frame image object classification. It's a real shame that no one knows how to do that...

modeless4• 5 years ago

Not signaling in advance for what the car likely considers to be an emergency maneuver is a pretty nitpicky criticism. The problem is the unnecessary emergency maneuver, not the lack of signaling.

As for lidar, well lidar isn't perfect either. You might get the same behavior from a lidar car if a trash bag happened to be blown into the road ahead by the wind. All types of sensors rely on machine learning to classify their observations. Lidar isn't a shortcut that eliminates that machine learning dependency.

Finally, calling what Tesla is doing "single frame image object classification" really betrays your bias here. That hasn't been an accurate description of the system for a long time.

lmilcin3• 5 years ago

"Not perfect".

Yes, killing pedestrians is not perfect.

I mean, seriously. You are too fucking optimistic guys. Go back to the drawing board. If your program can do that you are way off from the target.

You can't write software on which innocent lives depend like you would write a startup AI showcase.

rndmind• 5 years ago

Just to clarify here. no pedestrians were hit, and no one died.

I am curious how many people died on the first airplanes and commercial airline travel ( from 1900 - 1950's )

Do you suggest that the government should have made commercial airline travel illegal until there were zero accidents?

AnimalMuppet• 5 years ago

No, lmilcin didn't suggest that.

And your analogy is imperfect. How many people died on the ground from airplane crashes from 1900-1950? Probably, not very many. (Why does the difference matter? The passengers got on the plane, knowing - at least approximately - what the risks were. The people on the ground didn't voluntarily accept any risk.)

lmilcin3• 5 years ago

Just to clarify here. no pedestrians were hit, and no one died.

That because it drove slowly and the supervisor took IMMEDIATE control of the car. But it looked like it had every intention to drive into these people.

Do you suggest that the government should have made commercial airline travel illegal until there were zero accidents?

No. I suggest that government should ground commercial airplanes when they make stupid software mistakes that kill people.

Oh, actually they already do: https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings

sumedh• 5 years ago

We dont live in 1900 - 1950's, the standard of safety has improved.

Would you have the same opinion if a self driving car killed your loved one?

tiahura• 5 years ago

Maybe I’m missing something but what I saw is common with human drivers (myself included) as well.

I.e. pedestrian at corner sees car and stops, driver sees pedestrian and proceeds into turn because pedestrian is stopped at corner, pedestrian doesn’t wait and starts crossing while driver is still turning because they’re in a hurry, driver panics and slams brakes. Just seems like life at the corner of an intersection in a busy city to me?

88j88• 5 years ago

Just wondering how much of an increase do insurance companies charge for cars with autopilot?

jazzyjackson2• 5 years ago

I was thinking of this too when watching one of the videos testing whether their car would weave through concrete pillars. if you record yourself colliding with a concrete wall because you were testing beta software, does insurance pay out?

tzs2• 5 years ago

I wonder how well it does with stopping for school buses?

Here in Washington I see people getting that wrong all the time, usually in the direction of stopping when they do not have to.

The rule is that when a school bus has its flashers on and/or its stop sign out you stop unless:

1. You are going in the opposite direction, and

2. The road is either a divided road (separated by a median or a barrier) or has at least 3 lanes.

I see people going the opposite direction often stop on three or four lane roads.

(The reason you do not have to stop on a 3+ lane road or divided road is that the law requires that if school has students from both sides of such a road the school must have separate bus stops on each side, so that no child has to cross such a road. School bus stops can only serve both sides of a road if the road is 2 or fewer lanes).

(I've not been able to find a definitive answer on whether or not a road with one lane in each direction and a center turn lane counts as a 3 lane road).

bagels7• 5 years ago

It's low likelihood that it handles that. There are so many videos of it not handling even more mundane things, like this one, or the ones showing it trying to run in to a road closed sign.

dude1234• 5 years ago

The comments here are funny. Having lived in Seattle, owning 2 teslas and actually watching more than this single clip of a beta software: this is deliberate testing in a crazy corner case of a road. The driver is nervous because of the pillers. I use navigate on autopilot (FSD for the masses) literally every day and it navigates multiple highways, every day. It's awesome. Please continue skepticism and never partake, more room for me at the superchargers.

Woodi• 5 years ago

Attempts to prohibit repairing things is stupid and it is stupid on regulators part to not seeing it progresses.

But allowing live-testing auto-drive technology on public roads is beyound that. Police should just pull over anyone who turn this on and is on the move and revoke driver driving license. Simple as that.

"Oh, but it works for me..." is not how public security standards are created.

Arnt2• 5 years ago

I think you're holding Tesla to a perhaps unreachable standard here.

Before I go on, I'd like to digress and come out as a Waymo fanboi. I've been that ever since I read about a demonstration involving the software developers walking in front of the car, blindfolded. I admire that attitude, I really do. They put their feet where it matters.

But back to Tesla. What other than public roads is similar to public roads, in the senses necessary for testing driving capability? I know Waymo tests its software on recordings of public roads, but those recordings were made on public roads too. Maybe Tesla does the same, I don't know one way or the other. It's not an answer, because this approach doesn't avoid public roads, it only extracts more value from each test drive.

It would be possible to set up a test area. Maybe rent one of those "towns" the film studios have, and use stunt pedestrians, also from the film studios. I worry that those stunt pedestrians wouldn't be really like pedestrians, though, so this too doesn't replace street tests, with the real variety of real people.

These things can supplant public tests. It would be possible to regulate this and say, "software may be used for tests on public tests only when the same version of the same software has been run on one million km of recorded test drives" or such. But as far as I can see, not replace.

loeg5• 5 years ago

I know Waymo tests its software on recordings of public roads, but those recordings were made on public roads too.

Recording a human driver on public roads is very different from testing a novel software product on public roads.

simion314• 5 years ago

IMO each update should be considered a completely different new software, you would need to get again approval to put the shit on the roads. This is not 2020s video game industry where you ship buggy software as done and do monthly patches that fix some shit and break other shit.

cblconfederate• 5 years ago

Reads more and more like a video game

Tesla FSD Beta 10 veers into pedestrians with illegal right turn

Related Stories