Camera crushes Lidar, claims startup

https://spectrum.ieee.org/camera-crushes-lidar

by pseudolus • 3 years ago

62 171 3 years ago

spectrum.ieee.org

hirundo• 3 years ago

It's extraordinary that the article mentions Waymo and Cruise in this context but not Tesla.

adrr4• 3 years ago

Tesla doesn’t have autonomous vehicles. It has level 2 vehicles. Waymo and cruise have level 4 vehicles.

ramesh312• 3 years ago

Waymo and cruise have level 4 vehicles.

In a single small extremely constrained geofenced area using vehicles with hundreds of thousands of dollars worth of sensors each. Those companies are complete dead ends. They are not even remotely solving for the general case.

Fricken• 3 years ago

It seems Tesla is finally starting to get serious about L4 autonomy, but they don't have much to show for it yet. What's extraordinary is how many people have fallen for Elon's bullshit and remain committed to it in spite of it being obvious bullshit.

In any case, many journalists are still trying to spin it as cameras vs lidar when it's really single modality vision systems with no redundancy versus multi-modal vision systems that conventionally include camera as well as lidar.

Various Lidar startups have also made inroads in deriving vision from laser range finding technology. It's hard to speculate on what will win in the end, but nitpicking these things is a form of bike shedding. Ultimately it comes down to the software, which is a much bigger problem and not as straightforward to wrap one's brain around.

jeffreygoesto• 3 years ago

I assume the downvotes are about mentioning that one person that gets mentioned a lot? Factually this post is pretty accurate IMHO. As a human you can almost always extract "this could have been seen" from fusion sensor data. That does not mean the computer can (some of this also vibes from you judging post mortem and typically having additional information about the scene). Neither hand made nor trained classifiers master the "long tail" that is needed for L4. This does not change with some percent improvement of one sensor.

vm• 3 years ago

This article doesn’t address when cameras are blocked, which is the obvious issue with camera-only self driving. Teslas have crashed when cameras were blinded by the sun. Now throw in snow, rain, dust… Is that solvable with lots of cameras and different types? …Does it need to be solved?

Maybe the bigger question - anyone know the status of low cost lidar? Dozens of startups and larger companies were working on it 10 years ago, yet Lidar still costs “thousands” according to the article

deeviant• 3 years ago

Maybe the bigger question - anyone know the status of low cost lidar?

"Solid state" lidars would fit the fit bill for likely low cost lidar. They are probably 3-4 years out, and have been for the last 10 years.

coolspot2• 3 years ago

iPhone 13 Pro has solid state LiDAR.

tanjtanjtanj3• 3 years ago

There's something bizarre going on with lidar manufacturing/pricing. When I was looking into it for a specific application it was much cheaper to buy entire made-in-china products containing the exact lidar module than it is to buy the part separately, even at bulk pricing.

krisoft• 3 years ago

This is very common with all kind of components. There are economies of scale your vendor can achieve when they sell someone a million of the same thing. Also the company buying a milion of the same thing is going to pay the vendor a significant sum, even if they get all kind of discounts, and that puts them at a much better negotiating position than you buying a single one.

Hobbyist buying a few units of a component, even if they are buying it with a significant margin, will net the producer peanuts. So not surprising they don’t worry much about serving them that market.

asdff• 3 years ago

Maybe this explains the reason why its somehow too costly to put lidar in a $50,000 tesla, but I can have it in my $200 xiaomi vacuum.

jstanley2• 3 years ago

What does a human driver do when their vision is obstructed?

1. Attempt to use the vehicle's built-in windscreen wiper to remove the obstruction.

2. Failing that, stop the car. Preferably before the vision gets so badly obstructed that the car cannot safely be brought to a stop. But stop the car even so.

3. Get out and clear the obstruction. Admittedly the AI will have trouble with this, but it is vanishingly rare anyway, and if the car is carrying passengers, this task can be given to the passengers.

demondemidi• 3 years ago

This is whataboutism for ADAS.

People and ADAS have their own, different, and critical weaknesses. Neither is a panacea. (Which is why mass transit investment should be prioritized over scifi fantasy ADAS.)

bipson2• 3 years ago

Yeah, right. Since if there is snowfall we get out of the car and shoo the flakes away. Or scream at the sun to stop blinding us.

Humans have something called perception and cognition, we can make sense of things we don't see.

AFAIK we don't have cameras yet that can do that. We need better sensors.

mrguyorama13• 3 years ago

When the human continues driving in incliment weather, and runs into the back of a van full of kids and kills all of them, we put them in jail for making a bad judgement call.

How do we handle the AI mowing over a pedestrian when it makes a bad judgement call? Right now, the status quo is that we do jack and shit, and I can't help but feel like that's not a good plan.

xnx2• 3 years ago

This page from March 2023 says $1,000/unit for Lidar: https://www.sae.org/news/2023/03/adas-and-autonomous-vehicle...

adolph4• 3 years ago

Teslas have crashed when cameras were blinded by the sun. Now throw in snow, rain, dust…

I used to think the more sensors the better, but after listening to George Hotz talk about it I can see the logic of focusing on ambient spectrum in visual and near range. Of course, he will talk up his approach as best, but here it is as best as I recall:

  1. more sensors ~= more signal
  2. more sensors means 
    a. longer processing pipeline for fusing data streams (timing, registration)
    b. more software, thus more surface area for defects
    c. decisions about response when 1 sensor modality fails
  3. visual range spectrum is 
    a. well adapted for environment
    b. has inexpensive and high quality sensors
    c. sufficient for humans so is sufficient to get to human-like driving by a computer

The answer to blocked cameras is:

  1. to have protocols to slow down and stop gracefully
  2. maintain enough of a spatial model of the vehicle surroundings to perform the above (Simultaneous Localization and Mapping, SLAM)

Both of the above are basically what humans do.

mrguyorama13• 3 years ago

Our eyeballs are not cameras and have way more depth info from their function than just two arrays of pixels that you can derive parallax from, and all the claims that "humans only use their eyes" fundamentally ignore all the other parts we use, up to and including an intrinsic simulation of physics in our brain.

Jemm• 3 years ago

Hasn't this already been settled after Tesla's claim?

2rsf2• 3 years ago

Was it?

nomel4• 3 years ago

claim

Claims never settle anything.

sublinear• 3 years ago

To see how their system performed in these situations, NoDar conducted a series of tests on a remote airstrip in Maine with almost zero light pollution.

Isn't this the opposite of realistic?

In broad daylight, NoDar’s setup generated 40 million 3D data points per second compared to the lidar’s 600,000. In extremely heavy rain the number of valid data points dropped by only around 30 percent, while for lidar the drop was roughly 60 percent. And in fog with visibility of roughly 45 meters they found that 70 percent of their distance measurements were still accurate, compared to just 20 percent for lidar.

These results sound very cherry picked.

porphyra2• 3 years ago

Yeah and having 600k high quality lidar points is better than having 40 million noisy stereo points. Honestly their depth map is so full of holes and artifacts, despite the fact that a depth map is the easiest way to hide poor quality (since range inaccuracies only manifest as subtle hue changes).

Also not sure why they picked a 600k points per second lidar instead of a 3 million point per second one like an Ouster 128 beam one.

joshvm2• 3 years ago

despite the fact that a depth map is the easiest way to hide poor quality

I did my PhD on stereo and LIDAR; also in industry building these things. This is something that really annoyed us about camera companies selling stereo systems: you cannot tell anything from a colormapped depth plot. It doesn't matter if it's grayscale or jet or a nice perceptually uniform one. Zed are really bad for this in their promo material.

Small errors might be really significant for reconstruction, but depth maps make it easy to hide errors, fuzzy bits, holes, discontinuities etc. Really you want to test against a known calibration object at a distance and present the reconstruction error, show the 3D reconstruction top-down (or in a way which lets you see how much depth variation there is) or compare to a simultaneous LIDAR capture which might be sparse but will be more accurate (absolute) at distance.

hennell• 3 years ago

The other challenge for cameras is that, unlike lidar, which has its own light source, they rely on ambient light. That’s why they often struggle at night or in bad weather.

They were testing to disprove the idea it doesn't need ambient light. It's 'worst case scenario / point making testing' not realistic real world testing.

Also I'm not sure I see how you're deducing 'very cherry picked' data there. They've given a range of situations and some general results that look pretty good. I'd assume it's cherry picked in as much as it's not a full data set, but it doesn't exactly yell suspiciously missing anything core to me. Are you expecting the system to fall apart in light rain, but excel in heavy?

CharlesW• 3 years ago

"Camera crushes lidar" is marketing spin, because different types of sensors "win" for different scenarios.

You really want a sensor fusion strategy for devices making life-or-death decisions on your behalf.

mensetmanusman3• 3 years ago

You really want what works best given the ms time budget in the context of your on-board compute.

mynameisvlad2• 3 years ago

If it’s a life or death decision, you should opt for the best strategy, not the cheapest.

lgrebe• 3 years ago

How do you manage disagreements in fusion?

bipson2• 3 years ago

That's the essence of fusion.

batmansmk• 3 years ago

Kalman filters.

unregistereddev• 3 years ago

The old school way is to add logic based on the strengths and weaknesses of each sensor type. My example is not specific to automotive sensors (I haven't worked in the automotive sector, but I do have now-outdated experience in obstacle detection and ranging, along with avoidance algorithms).

Sonar sensors are most accurate at medium ranges, but they are notorious for detecting ghost objects that do not really exist. Infrared range sensors are more reliable but are only accurate at very short range. So when a sonar sensor detects an object 8.4 meters away, you use the infrared sensor to double check. If the infrared sensor says there's an object 9 meters away in the same direction, you assume the object is real but is actually 8.4 meters away. If the infrared sensor says the nearest object in that direction is 20 meters away, you assume the sonar sensor made something up.

If you have enough types of sensors, you can also use a "majority rule". If two of 3 sensor types agree, you assume the 3rd is an anomaly. Lidar is excellent for this because it is accurate across a very large range, so it tends to overlap with most of your other sensors. This increases that odds that when there is a disagreement, one of the agreeing sensors will be capable of accurately measuring the distance to the object.

mrguyorama13• 3 years ago

Statistics. "Sensors disagree" is the EXPECTED result when you get a reading from multiple sensors, and the whole point of sensor fusion is that, if the sensors have independent error models, that disagreement IMPROVES your output.

esafak• 3 years ago

"Robust" methods; e.g., robust particle filtering.

https://en.wikipedia.org/wiki/Robust_statistics

https://doi.org/10.3390/s21020438

ajross3• 3 years ago

That's mostly spin too. You want a strategy that works well enough, because that's the regime we live in currently. Other human drivers certainly aren't implementing a LIDAR-based sensor fusion strategy and you have to share the roads with them. Your algorithm for evaluating their safety is "how likely are they to kill you" and not an absolutist position on the specific equipment in their heads or cars, and you're clearly OK with that.

As far as LIDAR itself: sure, yeah, you get depth info out of it. But depth info is only part of the problem, and frankly it's clear at this point that it's one of the easiest. The hard parts are in the recognition side: not "is that pedestrian in your path" (easy), but "is that pedestrian going to step into the street or not" (hard). And that's a computer vision problem. You can use LIDAR output as vision input, sure, but it's has no advantages.

Tesla was right, basically.

BugsJustFindMe3• 3 years ago

But depth info is only part of the problem, and frankly it's clear at this point that it's one of the easiest.

That's proven false by the cars continuing to drive into stationary objects. This failure mode is not ambiguous.

Camera input is garbage for interpreting geometry, especially from very smooth or very discontinuous surfaces, and especially with the shitty low resolution and low dynamic range cameras they use, and especially with non-stereoscopic cameras with no motion freedom relative to the vehicle body. Lidar is a necessary crutch for working around the fact that, while hypothetical cameras that don't exist might work well, all available cameras are unsuitable for the purpose, and calculating multi-view geometry accurately costs time.

The hard parts are in the recognition side: not "is that pedestrian in your path" (easy), but "is that pedestrian going to step into the street or not" (hard). And that's a computer vision problem.

Pedestrian motion is not strictly a vision challenge but a general category of environment understanding (mass, momentum, motion mechanics). Vision is only one possible input mode preliminary to modeling.

andbberger• 3 years ago

Other human drivers certainly aren't implementing a LIDAR-based sensor fusion strategy and you have to share the roads with them

humans are absolutely doing sensor fusion, brains are bayesian inference machines. do not underestimate the power of the visual system.

and no, that the brain does is not an argument in favor of LIDAR-less cars. the eyeball + visual cortex system is alien technology compared to our feeble models. beware the hubris of a man who has learned to classify golden retrievers.

ggreer2• 3 years ago

The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust. But LiDAR is subject to interference, and it can’t see lane markings, traffic lights, speed limit signs, or emergency vehicle lights. If you ignored camera data and just relied on LiDAR, you wouldn’t be able to drive safely for very long. If you choose to ignore the LiDAR, then the sensor fusion is just adding noise to your camera vision model of the world.

If you somehow do sensor fusion perfectly and the models never disagree, then why even have the LiDAR? At that point you’ve solved vision using cameras.

nomel4• 3 years ago

LiDAR easily passes the “featureless white wall” test. That’s why I’ll be waiting.

yomlica8• 3 years ago

Don't you just make a conservative decision in that case? My eyes and ears disagree all the time, but if one of them says danger that is the one I listen too.

CamperBob22• 3 years ago

You remove one of the sensors and pretend the disagreement didn't happen?

bafe4• 3 years ago

The idea of sensor fusion is to have some kind of error model for all sensors (and frequently for navigation also a dynamic model of the vehicle) so that you can weigh the various sensor differently based on the uncertainty. The most primitive incarnation of this would be a simple linear Kalman Filter, but you can use similar concepts with more complex non-linear observation and dynamic models

progbits• 3 years ago

I keep hearing this argument which makes no sense. I guess it comes from Tesla marketing because they keep repeating that and it rubs off?

On technical level two sensors are clearly better than one even if you just pick one in case of disagreement, but as others have said Kalman filters and other more advanced techniques exist. There is a reason airplanes or spacecraft have multiple redundant sensors like this for decades.

The argument only makes sense if you want to save money, but then say you are being cheap up front.

ijustlovemath• 3 years ago

I think you're misunderstanding sensor fusion. Any sensor fusion worth its salt functions closer to a weighted average, but instead of linear combination, you're relying on the error model and overall system model to figure out the actual state from the variety of sensors

oxfordmale2• 3 years ago

Humans drive both with their eyes and ears, although Tesla marketing like to state otherwise.

We are perfectly capable combining both inputs and act accordingly. An AI system can easily learn how to combining visual and LIDAR input to make the right decision given the circumstances.In the end it is just a decision tree.

throwawaylinux• 3 years ago

The problem with sensor fusion is what do you do when the sensors disagree?

Like one sensor says you have a bus stopped in front of you and the other says it's all clear? And your choices are full steam ahead or prepare to not ram the apparent bus?

i_am_jl• 3 years ago

The problem with sensor fusion is what do you do when the sensors disagree? You have to decide which sensor to trust.

There are two cameras in this computer vision system. If one of them goes offline, is obscured, becomes dirty, or malfunctions you lose stereo vision and depth perception. So you'd obviously need 3+ cameras. And now we're right back at sensor fusion challenges again.

callalex2• 3 years ago

I mean this in the kindest way possible: you need to mix up your sources of information, as you have become blinded by lies from marketers and a cult of personality. Machines have been using redundant sensors since machines and sensors have existed, with great success.

kaba0• 3 years ago

Similarly to how ML vision also a black box, there is no reason why its error handling can’t also be that.

If you hear a sound but see nothing, you do become much more aware of the generic area afterwards, something like that should also be possible.

ChrisMarshallNY• 3 years ago

Sounds cool.

But these days, I’ve learned to wait for a working commercial prototype, before cheering.

2rsf2• 3 years ago

Clickbait as usual, it is one company's own claim.

snarkyturtle• 3 years ago

it'll also crush the kids they run over

fjfaase2• 3 years ago

It is known that in neurological vision there is are also top-down pathways, where later stage processing influences early stage processing. Especially in poor conditions, the interpretation of images is based on an already established 3D model of the environment and information about ones own direction of movement.

It is nice that the system can generate 40 million 3D data points per second, but those point still need to be processed (and interpreted) in later stage processing.

fjfaase2• 3 years ago

The cameras being 1.2m apart is nice for accurate triangulation at a larger distance, but in a crowded location with obstancles near the road, it could lead to substantial blind areas at closer distances. I also guess that in the dark, reflections of lights in puddles on the road could lead to a stark reduction of valid 3D points or even mismatches.

ck22• 3 years ago

So mud and dirt and dust and oil and antifreeze and all kinds of garbage spray on the roads covers all parts of a car. Some people never wash their car if they don't want to.

What happens with these sensors? Can they detect physical interference? Will they refuse to let an assist mode of any kind activate if there is a malfunctioning sensor?

What kind of self-diagnostic do they run, how often and how fail-proof is it?

agab• 3 years ago

Volvo has had this solved for ages. Bring them back!

https://imgur.io/gallery/C3Kww

mrguyorama13• 3 years ago

And when those break down and the owner doesn't fix it?

Self driving cars will require regular and stringent inspections of functionality, otherwise it's a time bomb.

vel0city• 3 years ago

Both of my cars have some level of ADAS systems on them. Both have ways to have sprayers clean cameras or have the cameras in the path of the standard wipers. When they cannot sense well enough they will disallow activating their ADAS features.

What happens when the windshield gets to coated in junk that the driver can't drive?

alphabetting• 3 years ago

The most performant AVs right now aren't using solely lidar. They have cameras and many other monitors in their sensor suites.

squokko• 3 years ago

I thought a deranged camera had crushed a LIDAR and then claimed the life of a startup.

GabeIsko• 3 years ago

Would solid state Lidar really be that much more expensive than a CMOS camera if it were manufactured at scale?

4ndrewl• 3 years ago

I read the title to mean "a camera crushed a lidar sensor which caused a startup to go into administration"

Tagbert• 3 years ago

and scissors cut paper

intrasight2• 3 years ago

If time-of-flight sensing made sense, we would have evolved that over millions of years - oh wait...

Vecr3• 3 years ago

I'm actually interested in this, are you claiming humans do time of flight measurements?

intrasight2• 3 years ago

My "we" was really animals in general, where there are several examples: bats, dolphins. But I have seen some academic papers on human echolocation over the years.

Hippocrates• 3 years ago

I highly recommend listening to this podcast episode. Lex Friedman interviews Andrej Karpathy (former director of AI at Tesla): https://podcasts.apple.com/us/podcast/lex-fridman-podcast/id...

1:32:39 Andrej explains why Tesla removed Radar and relies on vision. Part of it is that other sensors bloat and complicate the software needed to interpret data coming in. What if they get conflicting signals? There may be calibration or manufacturing inconsistency to account for. You need to normalize that in your software and the entropy quickly gets out of control. I find this explanation very compelling and feel like vision is the more "necessary" sensor anyway. After all, roadways are designed to be navigated using vision.

bloodyplonker22• 3 years ago

He "forgets" to make a point of the number one reason they don't use lidar: cost.

yreg3• 3 years ago

For lidar surely, but what about the ultra sonic sensors?

callalex2• 3 years ago

If something is required for the core functioning of your product, that thing is not “bloat”. That’s like saying “It’s really hard to do security and authorization for this server I’m setting up, I’ll just call it bloat and leave everything open.”

pedalpete• 3 years ago

I'd be curious to understand why the they think the technology would work in fog. The article suggests it does, but the science behind how that worked wasn't explained (or I didn't get it).

Does anyone understand this? The problem with fog as I understand it is that it defuses light, washing out vision. My understanding is that Lidar does not bounce off fog, or at least to the same degree.

Is my understanding there correct?

tb_technical• 3 years ago

¿Por qué no los dos?

iamleppert• 3 years ago

Long baseline stereo matching has a trade off of poor depth resolution in short ranges and requires wide angle lenses which result in an uneven distribution of data after correcting for distortion. The wider the lens, the more distortion increasing as you move away from the center of the lens as pixels are no longer square; you also need higher and higher resolution sensor to get good matches around the edges.

This makes it very hard and computationally expensive because you need to search progressively larger areas of the image to do matching to get the same FOV at longer range. Large baseline also is more likely to suffer from occlusion (even self-occlusion) by objects in the foreground resulting in holes in the depth map. In addition to being harder to keep the cameras rigid between calibrations (but I guess they figured this part out).

Instead of a depth map like they show it would be much more useful if they instead show a map of the reprojection error of a flat calibration target at the minimum and maximum depth ranges. Even if they are not aiming for geometric accuracy it would give a much better idea of the actual performance of their system.

The problem with these stereo camera companies is they get obsessed with maximizing certain metrics when the fact is there is a fine balance of trade offs to maintain that is highly specific to the desired application and even scene. Sure, you can increase the baseline and figure out how to mitigate calibration but that creates many more problems to solve that only become apparent when you try to apply it to some application.

Anyone interested in long stereo baseline should check out this blog which has been around forever: https://www.elphel.com/blog/2017/09/long-range-multi-view-st...

Andrey_Filippov• 3 years ago

We do it now with thermal images: https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...

mikewarot• 3 years ago

If you're going to deal with multiple cameras, why wouldn't you at least have a 4x4 array of cameras, with a spacing of at least 10 cm? You could deal with small object occlusion, and gain some redundancy.

Camera crushes Lidar, claims startup

Related Stories