Researchers fool self-driving cars with stickers on street signs (2017)

https://www.thedrive.com/tech/13258/researchers-fool-self-driving-cars-with-stickers-on-street-signs

by bookofjoe • 3 years ago

24 53 3 years ago

Like the issues we have seen with Bing and ChatGPT, it seems that self-driving cars are a long way from being resilient to real world uses. The issue is the complexity of the domain space, which which might not lend itself well to the mathematical model. You can get impressive results, in some cases exceeding human performance, but the problem is when they fall short in spectacular ways. The human brain is much more flexible in recovering from certain types of errors. Bing might suddenly profess its undying love for a user, while self driving might become inexplicably convinced that a cliff is the correct path because of some odd arrangement of pixels. The problem is especially keen when there are adversarial influences dedicated to creating problems.

ly3xqhl8g9• 3 years ago

Also Go: "Amateur exploited weakness in systems that have otherwise dominated grandmasters." [1]

[1] Man beats machine at Go, https://arstechnica.com/information-technology/2023/02/man-b...

WithinReason4• 3 years ago

How does putting specially engineered stickers on a sign say something about real world use?

rimliu3• 3 years ago

Directly. It once again shows that so called AI has no world model. For it all the possiblities are more or less equal why the human will spot what's wrong at once.

unlikelymordant• 3 years ago

Putting stickers on road signs can trick humans too...

In fact, road signs are just stickers on a bit of metal themselves.

dwighttk• 3 years ago

Sure but these look like the sort of thing that would be a low probability to fix where as something that’d fool humans would be at least a high priority (not gonna say every town would fix it quickly)

Also something that fools humans about street signs would probably also fool AI

woodruffw2• 3 years ago

I think the important distinction is that humans aren’t confused by the stickers that confuse “self-driving” technology. The stickers in the article, for example, are unlikely to confuse a human.

pards• 3 years ago

Although the article didn't say which stickers were used - they __implied__ that the signs shown fooled the vehicles but they didn't state it explicitly. Who knows, maybe they stuck 65 mile/hour speed limit sign over a stop sign.

themitigating2• 3 years ago

That increases the danger of self driving cars over people but there are other situations where self driving is safer.

When discussing this I don't hear the positives

TomK322• 3 years ago

On almost all yes, but the stop sign has a unique shape.

Verdex• 3 years ago

This was my thought as well. And I think it is an important fact to keep in mind. You can fool an AI by putting a sticker on a stop sign, but then again you can fool a human by stealing the entire stop sign.

Either way you're defacing infrastructure in a way that's potentially dangerous. It's just that the particular failure cases for an AI is different (and somewhat incomprehensible) to humans.

Of course, a self driving system could also be linked to a network that has redundant information about the road and could also broadcast its position and immediate intent to all other nearby self driving systems.

Now, on the other hand. If a police officer sees a missing stop sign, then it's pretty obvious that there is something that needs to be reported and addressed. However, if there's a small sticker on a stop sign (possibly something even confusable as just dirt or bird poop) then it might not be obvious that this represents a road hazard.

csunbird• 3 years ago

as always there is a relevant xkcd: https://xkcd.com/1958/

drowsspa• 3 years ago

When I read about self-driving cars being so reliant on street signs, I can only imagine the total shitstorm they would cause in a country like my own (Brazil).

detrites2• 3 years ago

When seeing this, are we meant to assume that not one company in the self-driving space, in the 5-6 years of research and development since this study, has moved to training their AI on anything other than stock, pristine, textbook roadsigns?

passwordoops• 3 years ago

But that's the point, and main argument people use against deep-learning based AI for real-world applications like driving. The moment you go outside the training set, failure generally follows

detrites2• 3 years ago

Cool. And the moment you get an incompetent, emotional or impaired human driving a car, failure generally follows. All you need to do is make sure your training set is complete enough to do slightly better than the average human.

TomK322• 3 years ago

Stop signs are octagonal all round the globe (except Japan), how could any self-respecting, self-driving car misread that shape and colour scheme?

eimrine• 3 years ago

If to color this sign with a texture of elephant skin, there are high chances that NN will give an output like it is seing an elephant.

bookofjoe• 3 years ago

The bar is SO much higher for self-driving cars.

I believe that if car crash deaths were reduced to 4,500/year in the U.S. with only self-driving cars on the road — a 90% drop from last year's actual 45,000 — an overwhelming majority of Americans would still reject self-driving cars because of the loss of agency involved.

wnkrshm2• 3 years ago

That's one factor contributing to fear of flying, though with a car you can stop and get out at least.

Edit: Also, I think that many of the casualties resulting from autonomous vehicles will be situations that humans could have easily avoided (vs. lots of saved situations where humans couldn't have done anything).

Cthulhu_• 3 years ago

I'm sure there's numbers, but a quick listicle says that most accidents are caused by distracted driving (e.g. using phones), reckless / aggressive driving, and driving drunk. Those aren't solved by automatic driving per se, but mitigated.

But then, aggressive drivers will be the last ones to put the autopilot on I'm sure, they're also the ones that will insist on the agency aspect of driving.

marricks• 3 years ago

What is the latest data for car crash deaths involving teslas? I know there’s a case against them for a few deaths involving a breaking bug.

People often pitch this view but don’t provide real numbers.

tonightstoast• 3 years ago

I also think people who don’t speed, and don’t drink (ever, or especially before driving) want to put themselves at risk to a dumb software bug. Like it’s great if we could reduce deaths but man I’d feel horrible about my kid dying because some computer vision engineer misidentifies a truck for a road.

quonn• 3 years ago

Teslas have consistently ranked at the top of most crash test benchmarks. They are extremely safe. Additionally their driver assistance systems enforce more checks that the driver is paying attention (more than Volkswagen at least).

Can you please link to the lawsuit? It could be one related to unnecessary emergency breaking which would likely be annoying but not cause death.

bdcravens3• 3 years ago

How many of those life savings don't require FSD? Speed limiters would make significant impact. If we want to be really extreme, ignition interlocks could eliminate many of the 10k+ that die from alcohol-related accidents. I'm not saying we should go that far, but I'm not certain that's any further than FSD.

jonhohle• 3 years ago

I was just thinking yesterday that it would be nice to have the feeling of a tactile detent or similar when the car reaches the posted speed limit that requires pushing through. Most of the time my car knows the speed limit through its own maps or CarPlay. The cruise control speed already changes its color if the limit is exceeded.

Just that subtle hint that the speed limit has been reached would likely keep me near the limit most of the time. I don’t even want it to prevent me from exceeding the speed limit.

This doesn’t help with the 95% of other cars that don’t know the speed limit, but it has to start somewhere. It seems odd that my car has redundant features and alerts for avoiding speed traps and traffic cameras, but little to actively help me stay within posted traffic rules.

vasco• 3 years ago

I'll be honest, I prefer to die in an accident "of today" where for example someone crashes into me which maybe an AI could stop, than to die in an accident where if I had the AI disabled I would never crash as a human driver.

Basically I prefer to be exposed to risks that a human "can't" avoid than from something like this which a human wouldn't do: https://www.sfgate.com/bayarea/article/tesla-firetruck-car-c...

bko2• 3 years ago

Do you feel the same way about auto break technology?

pigsty• 3 years ago

Personally, I’d simply prefer not to die. If something can reduce the risk of that happening by 90%, that sounds great.

People were worried about dying from electricity—a power source out of their control—to turn on their lights, and it certainly did and still does kill people, but more people died by starting fires in their homes to read at night or stay warm. But the number of people wanting to replace their lightbulbs with candles for a little extra control of their energy source is quite low (although they do exist).

TacticalCoder• 3 years ago

I believe that if car crash deaths were reduced to 4,500/year in the U.S. with only self-driving cars on the road — a 90% drop from last year's actual 45,000 — an overwhelming majority of Americans would still reject self-driving cars because of the loss of agency involved.

And very rightly so. Someone who never drinks under influence and nearly always drives during the day (say to/from school) and is driving since 30 years has a probability so much lower to be involved in a deadly crash than a 18 years old on alcohol or drug driving on a Saturday night that it'd require much more than a 90% drop for that person to be safer by moving to a self-driving car if they were to only create 90% less accidents.

Here's something totally bonkers: in several statistics the car brand the less involved in car crashes (per miles or per 1000 cars of that brand or per anything you want) is...

Porsche.

You read that correctly. And it's not because Porsche repeatedly gets amazing short braking distances compared to other car brands: it's because young inexperienced people don't have the means to buy a Porsche (and Porsche are still pretty much regular cars, not crazy nervous stuff like Ferrari or Lamborghini).

If self-driving cars could be implicated in 90% less accident, the winning strategy would very likely be something like this: "put anyone under 25 years (and over a certain age too) in a self-driving car".

And maybe, maybe, once self-driving cars are implicated in 99% less accidents, experienced drivers never driving under influence would accept to move to self-driving cars.

There's absolutely no reason that those the less likely to have crashes should be moving to cars that are less likely to create crashes for those the most likely to be implicated in crashes.

"Moving everybody to self-driving cars" is the wrong way to think about the issue.

It's the segment at risks that should be moved to self-driving cars.

People just look at the total number of accidents and don't realize that "Porsche on the highway driven by an experienced drive bringing his kids back from school during daytime" is not anywhere near as likely as having a serious crash compared to "teenager on alcohol or drugs driving on a secondary road on a Friday night".

They're not even comparable. A rounding error.

Put it this way: it you were to get a 90% by moving everybody to self-driving car, you'd get an even bigger than 90% drop by moving only those in the segment at risks to self-driving cars.

And then, later on, once self-driving cars drive near perfection, you can move everybody to self-driving cars.

P.S: and to put things into perspective, we're talking about self-driving cars being fooled by stickers or bird poo on street signs. There's still some way to go.

bdcravens3• 3 years ago

It's all a matter of the context that humans understand. I've seen stop signs for the left hand turn lane that the main lanes didn't have. I've also seen signs a bit askew (probably from kids messing with the sign) that were angled in such a way that an automated system may think belongs to the wrong street, but a human has no issues with.

malf• 3 years ago

I assume https://xkcd.com/1958/ (2018) was in response to this.

MichaelNolan• 3 years ago

Maybe I missed it, but neither this article or the linked car and driver article said which “self driving” card they fooled.

Fooling a 2017 Tesla or fooling a 2017 Waymo is a big difference.

jrm4• 3 years ago

When I see efforts like this, I feel like I'm taking crazy pills; i.e. the only reasonable way any of this works is that you'll need not just the "car" to be "smart," but the environment as well. Sensor/GPS/location thingies on other cars and roads, etc.

Researchers fool self-driving cars with stickers on street signs (2017)

Related Stories