Flash Memory Wear Killing Older Teslas Due to Excessive Data Logging: Report

https://www.tomshardware.com/news/flash-memory-wear-killing-older-teslas-due-to-excessive-data-logging-report

by rbanffy • 7 years ago

230 170 7 years ago

bookofjoe• 7 years ago

Because a Tesla is highly dependent on its electronics, once the flash memory in a Tesla’s infotainment unit goes bad, it essentially bricks the entire car. Yikes!

metalforever• 7 years ago

Someone that has had a few classes in electronics should be able to replace this.

SXX• 7 years ago

Unless your car has DRM that prevents you from tampering with it's flash memory...

Avamander• 7 years ago

It takes more than that to replace a BGA flash chip.

spinach• 7 years ago

Seems a bit more complicated than just unsoldering the old chip and putting in a new one.

https://web.archive.org/web/20190621052637/http://teslaservi...

joezydeco2• 7 years ago

However, if it's similar to how Apple does it, the chip might be bound to the processor via a serial number or some other magic that's burned into the NAND at manufacturing, or OTP on the processor, or something else.

It's not just a matter of "oh just go get a Hynix NAND part from DigiKey and reball the PCB". At minimum you need to copy the data over. At worst you need to bless the part with whatever secret sauce Tesla is using at the factory.

wasdfff3• 7 years ago

Good luck finding the part and not voiding your warranty

snag• 7 years ago

Cars are not "bricked" : this issue is well known and some Tesla hackers have solutions to fix. Rich rebuilds YouTube channel did a report on fixing flash cards on a Tesla : https://youtu.be/o-7b1waoj9Q?t=510

slg3• 7 years ago

I really dislike how this issue is being presented in the headline and in quotes like this, because it is fundamentally no different than a critical part going bad in an ICE vehicle. The Tesla isn't "bricked" when this flash memory goes bad anymore than a Toyota is "bricked" when its transmission dies. You fix the faulty part and get the car back on the road.

That said, if there is a systemic problem with this part failing faster than it should, which certainly seems to be the case, Tesla should be pushed to either recall the affected vehicles or waive the service costs for those owners who are impacted. But that isn't any different than if there was a design flaw in a critical part of an ICE cars manufacturing.

vorpalhex• 7 years ago

In an ICE vehicle, something like a transmission is very expensive and takes up amazing amounts of space, and so any kind of redundancy is very difficult.

But we're well aware that flash memory degrades, corrupts and gets a bit screwy. Flash memory chips are also dirt cheap and quite small - couldn't Tesla of baked in a bit more redundancy in this case?

In our transmission example, any fault in the transmission is not catastrophic, but with electronics, a single bad joint or bridged trace can take the whole thing down.

umvi3• 7 years ago

And why should "the flash memory in a Tesla’s infotainment unit" be a "critical part"? This seems like the opposite of a critical part.

That would be like if the CD player in my ICE car broke, suddenly I can't drive my car. Absurd.

justapassenger• 7 years ago

The Tesla isn't "bricked" when this flash memory goes bad anymore than a Toyota is "bricked" when its transmission dies.

That's why Toyota isn't releasing updates to transmissions every few weeks, that are changing it's behavior and using off the shelf parts without proper attestation and testing in all possible configurations.

Tesla moves fast, which is great for progress. But extremely risky for long term reliability of complex machines.

dmitrygr• 7 years ago

No, because Toyota does not use consumer-grade parts in cars. This eMMC (H26M42002GMR) is not even rated for automotive temp range! (flash wears out faster and retains data worse at higher temps)

Unklejoe4• 7 years ago

it is fundamentally no different than a critical part going bad in an ICE vehicle

The main difference is that the probability of a typical ICE ECU or TCU failing is much lower than a Linux-based infotainment system running on non automotive grade flash.

joezydeco2• 7 years ago

Previous discussion:

https://news.ycombinator.com/item?id=19912065

0xcde4c3db• 7 years ago

It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car?

jschwartzi• 7 years ago

They probably used a reference design and then failed to perform the step of analyzing the design for weaknesses like this one. Or they did but decided it wasn't their problem.

myself2482• 7 years ago

I assume the decision to leave datalogging turned on to this degree was not made by the same folks who decided to put datalogging and firmware in the same flash IC.

PinguTS2• 7 years ago

In mass production every tenth of a penny pays. That is the thing in car mass production.

taneq3• 7 years ago

Yeah but there's Toyota Corolla style mass production and Tesla Model S style mass production. In the former? Sure, save $5 here and there and it adds up. In the latter? Spend the money. Any small saving from corner-cutting will be far outweighed by the loss of reputation if your $100k car bricks itself for want of a $5 part.

CalChris• 7 years ago

In mass production every tenth of a penny pays if it pays. This is a cost benefit analysis and not just a cost analysis. If in the lifetime of the car, this part will fail then it will have to be replaced at great expense, $1,800 to $3,000. If that happens under warranty, that will cost Tesla a lot. If that happens out of warranty, that will cost the customer a lot and will cost Tesla a lot in customer good will.

I sure hope Tesla has figured this out for 3 and the Y. (It has. According to the FA, this has to do with MCUv1 Tesla Model S and Model X units up to 2018.)

vonmoltke3• 7 years ago

Depends on the definition of "mass". $0.001/unit doesn't pay when "mass production" is 500,000 units per year. That's a savings of $500/year.

danesparza• 7 years ago

This twitter thread from last year might enlighten you:

https://twitter.com/atomicthumbs/status/1032939617404645376

pseudosavant• 7 years ago

That thread is a trip.

microtherion• 7 years ago

Discussed heavily on HN at the time: https://news.ycombinator.com/item?id=17835760

Slartie• 7 years ago

Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars. This is especially evident when looking at comparable cases of part failure, like the infamous touchscreen failures: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows...

You are absolutely right in that this failure should not have happened, because they should use some other medium to store their logs that is not getting cramped by increasingly bigger firmware versions and that is appropriately sized and engineered to withstand at least a century or two of heavy usage. But since cloud thermostat and media player box vendors also get away with just stuffing some embedded Linux distro on a single chip of the cheapest Flash they can find on the spot market, Tesla probably didn't put much thought into why they shouldn't just do the same. After all, this system is just a media player box mounted in a car, right?

(Except that for some totally absurd reason, the car ceases to function entirely if its media system breaks, which is the next big WTF in this story...)

perl4ever9• 7 years ago

"Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars."

Who didn't assume this to be the case? Teslas are automotive iPhones, and I've always assumed they would be scrap in 3-5 years rather than 15-20. This is why I don't think it was ever valid to compare the purchase price to a regular car.

com2kid2• 7 years ago

It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car?

It is strange no one performed back of the envelope calculations on flash life vs logging size.

I've worked in embedded, we had a time series based rolling log. Flash life calculation was done at regular intervals.

IIRC with what we shipped, the flash chip would outlive the battery by 3x or 4x. And that was without wear leveling!

We were smart enough to continue the rolling log at the last point on flash we uploaded from, avoiding the mistake I've seen some products do of starting over from the very "beginning" after every log sync. Do that and you end up with 10% of your flash worn down to nothing and 90% of your flash going unused.

Also, unless needed, please stop using JSON. Use a binary format instead. Protobufs is trivial to get up and running almost anywhere. Other binary encoding formats may serve a product's needs better. JSON is fine if everything is low bandwidth web services talking now and then, or low friction HTTP APIs, but for other use cases... use something else.

Worst I ever saw was XML being used to RPC invoke every single function call, even local ones. All params got packed up in XML, sent over to the destination, who'd then unpack the XML, execute, and package up the return value. A double digit % of the CPU was being used on XML serialization and de-serialization. (If your choice is that or JSON... probably go with JSON!)

Logging flash being same as storage flash is super weird though. Though if their MCU only supports hanging a single flash chip off of it, I could see it happening.

catalogia• 7 years ago

Estimating log growth rate is troublesome. It might be easy enough to measure under normal operating conditions, but sometimes something breaks, maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate.

Assuring yourself that under no circumstance is that possible can sometimes be tricky.

wyldfire• 7 years ago

FRU sizes are optimized to minimize warranty cost and must be balanced against things like design complexity, reliability and service procedure count/complexity.

This was a no-brainer IMO: Tesla wanted their cars to be superior and time to market was key. Why do handwringing about the cars that might be still on the road in five or ten years when that's beyond the warranty and beyond Tesla's life in business if they don't deliver a Model S with the target specs? They made the right tradeoff. This is probably one of dozens of design weaknesses that Tesla cars have, ones that allowed them to get way ahead of the competition.

Now that we have arrived at five or ten years later there is some nonzero risk to their brand. Do they have unreliable cars? How many Tesla owners still own the original car and how many of these were purchased second-hand? Folks who bought the original car and they encounter this failure mode might think about this expensive out-of-warranty repair when they go car shopping next. If there are a lot of them, Tesla could consider the financial tradeoff between extending the warranty to cover this failure mode and the impact on future sales.

gowld• 7 years ago

Today, it looks like Tesla will fix the problem if you're within the warranty, and outside-of-warranty repairs can cost $1,800 to $3,000

Any other car manufacturer would issue a recall notice and repair it for free, regardless of warranty status.

I smell a class action lawsuit -- Tesla installed software that invisibly destroys user-inaccessible hardware throughout the entire warranty process and then fails afterward, and then charged people to fix it.

lallysingh• 7 years ago

"Any other car manufacturer would issue a recall notice and repair it for free, regardless of warranty status."

I don't buy this one bit. E.g. GM ignition switches, Ford Focus transmissions, LR coolant crossover pipes. They don't do this by default, and mostly get away with it.

dawnerd• 7 years ago

I doubt any car company would recall this. There’s plenty of production cars with flaws that completely destroy the engine that were never recalled, like the north star engine Cadillac used.

gonesilent• 7 years ago

North Star is sealed for life!

happycube• 7 years ago

The Cadillac "High Tech" V8 used in the early 80's was even worse for that...

michaelt• 7 years ago

The Tesla firmware wasn't very big at the beginning, [...] the firmware has grown, leaving very little room for the logging to take place. That means individual sectors suffer from lots of data being written in a short amount of time, further accelerating the wear.

My understanding was modern flash memory wear levelling would swap regularly-written and seldom-written sectors around, allowing for even wear across the disk even if it was 95% full all the time. Is it established that Tesla doesn't have this feature?

planteen• 7 years ago

I'm not sure exactly what eMMC part they are using, but the SDHC spec doesn't explicitly require wear leveling. A part could implement it, but it's not a given.

codeulike• 7 years ago

It does, but the article says that the car logs a lot of data and so consequently all sectors get wear. See here: https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issu...

oarsinsync• 7 years ago

It's also possible that older Tesla vehicles do not have modern flash memory.

benj1115• 7 years ago

That would be bad for this use case?

I want the firmware working, I don't much care if data logging stops working.

rcxdude• 7 years ago

While I might expect an SSD to use such a strategy, an embedded flash chip doing so is just as likely to be a misfeature, because when the flash does fail you may lose data other than that already being written, which makes it impossible to keep the system even limping along without any logging.

evancox100• 7 years ago

This article is unreadable due to the ads

taneq3• 7 years ago

What ads?

crankylinuxuser• 7 years ago

Adblock on the web is equalivalent to antivirus on Windows.

If you're not running it, you absolutely should be, precisely because it is a security issue.

brokensegue• 7 years ago

serious question: do people still run antivirus on windows other than MSFT's free one?

lithos• 7 years ago

Of course there are absurd ads. As adblocking gets more mainstream and less technicalml, the users left to monetize are less skilled and less valuable. Meaning publishers are stuck trying to monetize a smaller slice of their total traffic, that's also worth less. Leaving only more advertising "mass" as a way to get the same income.

gok• 7 years ago

Especially annoying since this article is really a retelling of another article [1], which is itself a retelling of a forum thread [2].

[1] https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issu...

[2] https://teslamotorsclub.com/tmc/threads/likely-mcu-failure.1...

Merrill2• 7 years ago

Which auto maker's electronics are likely to be reliable for the 25-year nominal life of the vehicle (US fleet average age is 11.8 years and rising)?

And which will have spare parts available for that long?

I'm thinking about buying a new car, but I'm worried.

PinguTS2• 7 years ago

My Volvo S60 is getting 20 years in 3 month. Many premium cars from the 1990s are getting to the 30 year mark, which will mean they are getting a historic car badge. Most of them still work. Only some with failed sealings need replacements.

Merrill2• 7 years ago

I have a 2005 Accord which has been quite reliable, but it is an LX and doesn't have much of an infotainment system. I'm worried about repair of things like the video cameras, LCD screens, and infotainment systems that are in the new cars once a decade or more has passed. Cars are a pretty hostile environment for those systems.

jbarberu• 7 years ago

We have two 03 vehicles that are going strong. The electronics are simple enough that I expect them to keep going unless there's ever water intrusion :) It's a Honda and a BMW for what it's worth.

JohnJamesRambo• 7 years ago

My family’s ownership outcomes strongly suggest getting a Toyota. I’m driving my Dad’s old 2001 Tundra pickup with 350k miles on the same engine. I was amazed when I bought it from him how every switch still worked, the original AC still blew cold. This is not my normal experience buying used cars from other manufacturers. Usually at about 150k miles they turn into a clown car of various systems failing. Someone put thought into each component on the truck and how it would age.

LeonM2• 7 years ago

Toyota has been known for their reliability for a long time now. I can attest that with my Lexus, it's 13 years old now and in the 7 years that I have owned it, it has had zero defects. Last year it even got recalled (12 years after production) because the fuel filter might not have been installed correctly in the factory, so they replaced it for free.

That said: I still do every scheduled maintenance, which helps tremendously. The main problem with older vehicles is that people stop following the recommended services as the car gets older. Which is of course the opposite of what you should do with an old car.

kipchak• 7 years ago

Generally speaking Toyota is conservative in terms of rolling out new technologies and sticks with what's proven. For example the base Camry has a 2.5L engine and 8 speed auto transmission and the base Accord a 1.5 turbo and 10 speed auto. Toyota also goes more in depth terms of QA and testing on individual parts than others.

From an article on the Toyota - BMW Supra/M4 collaboration,

"...BMW couldn’t believe how extensive some of our quality and efficiency studies were as parts came into shape one by one. We would take every bit down to a fastener or rivet, and put it through our stringent quality control and a dozen other testing, we’d ship thousands of parts back to Japan for analysis. That is normal to us."

http://club4ag.com/chief-engineer-tetsuya-tada-reveals-the-a...

celticmusic2• 7 years ago

When I was in HS I had a friend whose family had this old beat up orange smallbed truck from the 70's (this was mid 90's). That thing was beat all to hell, ugly as sin. It was used as a truck.

But you know what? That damned thing ran like a beauty. It literally ran until their son started driving it and wrecked it.

I was impressed enough with it, that when it was time for me to purchase a vehicle, I went with Toyota. 15 years later and that thing has never left me on the side of the road, not even once (I do keep up maintenance on it). I'm considering purchasing another vehicle because it's time to get something larger, and when I do it will most likely be Toyota.

I've seen so many people say that at 200k miles Toyota is just getting warmed up, and there's some truth to it. You obviously have to keep up with maintenance on the vehicle, but if you do I can guarantee it will treat you right. And I'm not saying other manufacturers are any worse, but I am saying Toyota is a great vehicle to buy.

tyfon• 7 years ago

It highly depends on which Toyota you pick.

RAV4 for instance is plagued with faults while my 2013 Avensis ran quite ok the first 90000 kilometres I drove it before getting a tesla.

It did have to change brakes, a spring in one of the keys failed and the open/close window button on the passenger side broke. It also rusted under the chrome strips..

But compared to other cars I think it did quite well :)

anderspitman• 7 years ago

Our '91 Tacoma (simply called SR5 back then) made it well past 200k miles and only stopped then because it got caught in a garage fire a couple years ago and melted. Loved that rig

mywittyname3• 7 years ago

There are plenty of 90s era car electronics that are going fine. The sensors are going to be maintenance items after 20 years, but there's no reason to think the solid state components won't normally last 25+ years.

Spare (electronic) parts shouldn't be an issue with major brands. I've seen the same sensor used on a dozen models from several manufactures produced over 10+ years. As Takata exposed, there aren't that many parts suppliers around the globe, so a lot of commodity components are shared across the industry.

For long-term part replacement, I'd be far more concerned with construction techniques like plasma-transferred wire arc sprays used for creating cylinder linings. This technique cannot be replicated outside of the factory, thus, they kind of make engine blocks disposable (they can be sleeved, but they won't perform the same and will require different piston heads).

slg3• 7 years ago

but there's no reason to think the solid state components won't normally last 25+ years.

Except this article is specifically about solid state components failing in a much shorter timeframe.

wasdfff3• 7 years ago

If you have a used audi and the mechanic comes back and tells you its the wiring harness, thats probably it for the car. Unless you like spending 3k to fix a 4k car.

kube-system• 7 years ago

Good process ensures good results: https://en.m.wikipedia.org/wiki/The_Toyota_Way

bluGill2• 7 years ago

Depends. Electronics manufactures won't promise any CPU will be available for that long (Odds are someone reading this should be thinking I'm partially responsible for that policy in my company). When spare parts are not going to be made the companies will try to predict how many will be needed in the future and order that many. This will be your biggest limit: you need a part there is a good chance the part is failing more often than expected and so the manufacture is out of stock and they cannot get more. (or sometimes they can, at a cost of $1000 each, minimum order 10,000 - the manufactures want to sell the new shiny replacement even though it isn't compatible)

Even if the above isn't an issue, storing electronics not on a circuit board is not easy, but I'm probably not supposed to talk about those issues. (You can research them)

throwaway57522• 7 years ago

Prius's routinely get to 300-400,000 miles. The used market is full of 10+ year old Priuses with 200k+. You may have to replace a transaxle and hybrid battery, but they are $1k each. Part of it is Toyota, part of it is the Atkinson cycle engine.

beerandt5• 7 years ago

hybrid battery

This isn't a trivial cost

i_am_proteus• 7 years ago

When the alternator in my 1983 Mercedes 300D quit, I started the car off a jump and drove it dead-electric to the store for a replacement part.

I believe the only semiconductors in the car are in the timing circuits for indicators and wipers, the tach and fuel gauge amps, and the radio.

Past results do not guarantee future performance: no new cars are this simple.

coolspot• 7 years ago

1983 Mercedes Diesel has electronic fuel injection and ABS.

wiggles_md2• 7 years ago

Up until around 89 (I think) you could have even tow started it! The instructions are even in the manual. They are incredible machines.

noonespecial• 7 years ago

I once drove my '76 300D for more than a week with no electricity whatsoever. No generator, no battery. Hill started it or borrowed a fresh battery when I needed to start it and let it run until I was done with it.

Electricity was strictly optional on those cars.

brandonmenc3• 7 years ago

I bought a 2000 Toyota 4Runner a few years ago at around 160k miles, put a few thousand dollars into required routine maintenance, and it runs great.

Sale price plus maintenance divided by time owning it currently has me at $275/month and dropping.

benj1115• 7 years ago

What's "a few years"?

$275/month is $3300pa, so you're already at a minimum of $6600 for a near 20 year old car, which doesn't sound particularly cheap.

I'm on a lower monthly cost of ownership for 3 years ownership of a 6 year old car.

perl4ever9• 7 years ago

I feel like airbag-era cars are time bombs, so I have to reluctantly pass on a lot of cars I otherwise would enjoy.

You can argue the risk comparison between an old airbag and no airbag, but the thing that sways me is, beyond a certain point, the manufacturer just doesn't care if it works or not. When 95% of the cars have been scrapped, nobody cares if it doesn't work right anymore.

I also consider that on an old car without an airbag, I still have a seatbelt that I will wear, and on an old car with an airbag, I still don't have modern crumple zones and reinforcements.

So I'm sticking with either relatively new vehicles or pre-airbag ones.

dsfyu404ed• 7 years ago

EEC-IV and EEC-V (Ford) ECUs haven't yet found the "late" stage of the bathtub curve and they're 15-35yo give or take. I'm not familiar with other brands of ECU but I suspect that they tend to be similar. If you look at all brands there's probably some that don't last because they were subject to the capacitor plague but they're clearly so few and far between that no brand has developed a reputation for ECUs crapping out. I wouldn't personally be worried about electronics failures. Physical failures of the electronic systems are far more likely.

Because this is a decently upscale crowd on HN everyone is going to tell you Toyota/Honda. I'm not going to write the mini-novel it takes to explain why but it suffices to say that is one filter bubble's opinion and other bubbles will have other opinions. As someone who has fairly extensive experience with vehicles that most here would consider far beyond end of life I assure you the badge on the grill matters far far less than what you use the vehicle for and how you treat it. Anything will "last" 25yr if you're willing to baby it there, put a bunch of money into it or just deal with it having all sorts of NVH issues in years 20-25. Trying to predict what vehicle and configuration will last 25yr with the least work (the unstated assumption in these discussions) is a complete and total fools errand because the equation is dominated by variables outside, or mostly outside your control.

crooked-v• 7 years ago

If you like niche cars, the Mini Cooper line has dealerships with replacement parts going back to at least the 2001 models (and probably can custom-order stuff for earlier models), and an endearingly nutty online fanbase that sells both custom aftermarket parts/upgrades and original OEM parts for both the 'modern' BMW-made line and for the classic cars.

The word online is that these cars take a lot of maintenance, but I haven't had any trouble with mine (2012 model, 90K miles, original electronics) aside from needing a new fan belt and a new battery. And of course the Countryman line is basically just a BMW SUV with the Mini branding and dealership network.

If you want a vehicle that's less 'will the parts be available' and more 'will it just last forever', though, you can't go wrong with a Toyota Hilux.

rodgerd• 7 years ago

Toyota commit to 20 years of parts.

(I can probably still get new mechanical parts for my 23 year old Ipsum, but I doubt the in-car electronics are replaceable except from the second-hand market)

Animats• 7 years ago

That's a problem.

The design life for the Ford EEC IV, used in 1980s Ford vehicles, was 30 years. My 35 year old Ford Bronco is still using the original electronics.

perl4ever9• 7 years ago

I thought there were laws and regulations requiring automakers to reserve a certain number of spare parts for everything they make and support it for a period of time.

vijaybritto2• 7 years ago

A failure in the infotainment will brick the entire car?! I thought they would be made up of individual systems talking to each other in messages. Tesla needs Erlang!

takumo• 7 years ago

This is the case for many, if not most, modern/high-tech cars, especially EVs.

The on-board systems have become very tightly coupled, if not monolithic to the point where a broken info unit will prevent the car from starting.

I had the infotainment system replaced in my e-Golf a few weeks ago, it took two technicians three days to install and configure to work with all the car's on-board systems.

Edit: If I _had_ to guess, I’d say this is because the car industry (especially electric) has been pushing to innovate features so rapidly that the time hasn't been available to engineer, test and prove decoupled systems.

vijaybritto2• 7 years ago

Why won't they think about these things? Do they really don't care or is it that we don't understand something very important in making cars?! I want to know !

smolder3• 7 years ago

There's anti-theft considerations at play with how car electronics are coupled, reducing part count for cost savings, and I think a bit of the classic right to repair vs corporate interests dynamic happening. Automakers have been known to intentionally make things difficult to work on.

bri3d• 7 years ago

Yes, electronics are highly coupled, but in the case of the e-Golf it will at least function as a car with no infotainment system (unlike the Tesla). And MIB replacement in the VW MQB platform is simple so I don’t understand what took the techs so long. It’s one unit in the glovebox and guided coding using ODIS, the VW dealer tool (which is also available to independent shops albeit at a $1000/year fee, which is stupid and should be illegal). The VW units do have Component Protection which keeps them from simply plugging in, but with the aforementioned dealer tool, so long as it is legally obtained, a subscription to VW’s anti theft coding service Geko is already included and the process is very straightforward.

linsomniac• 7 years ago

Not exactly new to integrate multiple systems: My 2004 Land Cruiser has the heating/cooling and a few other less urgent systems integrated into the radio. Which means I can't replace it with a radio that supports bluetooth audio, without giving up heat/AC...

lazyguy22• 7 years ago

You can... it just means modifying the PCB for the controls extensively. I had a friend who did this on a Subaru once. Was a huge PITA.

fred_is_fred• 7 years ago

Same for my 2011 Explorer, and since it was such a PoS [1] the system would reboot it self while I was driving. When rebooting it would put the A/C/heat into halfway fan on 70 degree mode, whether you wanted it or not.

[1] - there's currently a class action settlement about how terrible it was

Unklejoe4• 7 years ago

I thought they would be made up of individual systems talking to each other in messages.

That's how my 2011 BMW is. It's not exactly the most reliable car, but I'd imagine things would be much worse if it weren't designed that way.

For example, my body control module (FRM) just recently died, but the engine still runs and the car can still be driven, just none of the lights work.

The non-critical systems communicate with the powertrain bus through a special CAN gateway in an attempt to isolate the systems.

LeonM2• 7 years ago

With electric vehicles now on the market in masses and coming of age, we’re starting to see the real culprits of aging

The issue with the flash memory resides in the MCU (media control unit), which has nothing to do with the electric drivetrain. Cars with combustion engines can have this issue too.

once the flash memory in a Tesla’s infotainment unit goes bad, it essentially bricks the entire car.

Essentially, yes. But as far as I am aware (correct me if i'm wrong) a Tesla with a bricked MCU will still drive.

I am annoyed that articles like this make it sound that these types of faults are only happening to electric cars, which it is not. It might be a Tesla issue, as they are both ambitious and inexperienced in car component design, but it could have happened with other brands as well.

Fortunately, Jason Hughes, known as a ‘Tesla hacker,’ can service these units at a much lower cost.

Yes, many people can, there is no secret technique to it. Just like a local garage can fix any part with any brand of car for much cheaper than the dealership. That is because dealerships have to replace parts, rather than repair them. There are many good reasons why they do this, I wont go into details here.

Sure, a conventional car mechanic probably can't do it, but anyone with electronic repair experience can. That is a shift that is happening with all car brands. As cars are shifting towards computers on wheels, we must learn to distinguish mechanical repairs from electronic repairs.

conception• 7 years ago

It might be a Tesla issue, as they are both ambitious and inexperienced in car component design, but it could have happened with other brands as well.

For me this is a trend Tesla has been showing. (e.g. https://www.thedrive.com/tech/27989/teslas-screen-saga-shows... stuff is trying to use off the shelf but aren't designing for what their cars will actually be experiencing. It -could- happen to other brands, but it is happening with Teslas. This combined with their reported code practices (https://twitter.com/atomicthumbs/status/1032939617404645376) generally put in the camp of recommending people lease, don't buy and definitely don't buy used. These first/second gen Teslas are going to be trash is a few more years.

_ph_2• 7 years ago

Which is why any flash drive that is regularly written to, should be exchangeable. Be it a car or a laptop. Putting things on SD or M.2 seems like a much more robust solution.

rasz6• 7 years ago

Apple thinks otherwise.

_ph_2• 7 years ago

I wrote my post because I disagree with Apple on this point. And any other company building their laptops in this fashion.

flash_zombie2• 7 years ago

Today, it looks like Tesla will fix the problem if you're within the warranty, and outside-of-warranty repairs can cost $1,800 to $3,000, depending on your location. Tesla’s method is to replace the entire MCU.

This is ridiculous. Can we all agree that this is ridiculous? Because that's what it is. If we can at least all agree that this is indeed ridiculous, perhaps the collapse of modern civilization can be avoided.

uniformlyrandom• 7 years ago

Which part? The price? Yes, it is ridiculous.

The method of replacing entire MCU? Maybe not, flash is likely not the only component in there that has an age limit.

flash_zombie2• 7 years ago

Replacing the entire MCU. The price for that isn't necessarily ridiculous.

Maybe not, flash is likely not the only component in there that has an age limit.

The MCU itself should outlast the lifetime of the car in the vast majority of the cases.

omgwtfbyobbq• 7 years ago

In case anyone's interested, here's a TMC thread with good discussion on it.

https://teslamotorsclub.com/tmc/threads/preventive-emmc-repl...

And a doc the OP put together on replacement.

https://docs.google.com/document/d/1ZH8oP4AgdVxmCN0saKA1Dxc2...

Unklejoe4• 7 years ago

I'm going to be a little petty here, so sorry in advance:

It's common for Tesla fans to mock other automotive manufacturers for being "old and slow" - as if they're incompetent because they take so long to develop new features and adapt to changes in technology. It's like a "why does it take 5 years for BMW to update their infotainment system when I can do the same thing with a Raspberry Pi in 5 minutes?" mentality.

What I think goes unappreciated by some is the level of testing that normally goes into automotive grade development.

That's not to say that modern ICE cars are more reliable than a Tesla, but their electronics are usually designed for a higher level of robustness.

Here are my issues:

- Why is the car rendered useless if the infotainment system fails? This should be a non-critical component. If it is critical and the critical part can't be isolated, then the system as a whole should have been tested more extensively.

- Why does the infotainment system fail if a log message write fails, and why was this failure mode not tested? It is very common knowledge that flash wears out, and log messages aren't critical.

- Why were they not using automotive grade flash?

This, coupled with the recent story about the non-automotive-grade displays failing really demonstrates that maybe, just maybe, the rest of the automotive industry isn't totally clueless. Maybe they're just more scared of a failure.

Who knows - it's probably a mixture of both.

vokep• 7 years ago

This is giving me deja vu...I'm not alone right? This same comment is posted on the same story but from the past, no?

alanthedev• 7 years ago

- Why were they not using automotive grade flash?

Cool. I did not realize there were such products.

Unklejoe4• 7 years ago

Well, for a given chip, it usually just refers to a wider temperature range.

But there are other types of flash that are better suited for this environment. I see parallel and SPI NOR flash a lot in ruggedized embedded devices (usually with a minimum of 100,000 erase cycles), but they're generally a lower capacity.

I don't know the exact cause of this failure, but there are ways to design around unreliable flash too. They could partition it such that the firmware image is in a read-only partition, and it runs out of a ramdisk, with only user settings and log messages stored in a R/W partition. That may be what they're doing, and perhaps the issue is due to the software crashing when the write fails. I'm not sure.

PossiblyKyle• 7 years ago

Reminds me of a comment I’ve seen a couple of years ago: “Tesla is not an automotive company, it’s a tech company”

perl4ever9• 7 years ago

In all fairness, if you look at cars preserved from the 80s, when electronics started becoming pervasive, there are a lot of failures there too.

For instance, the multi-segment displays tend to fail in little bits. I have a car that doesn't have that problem, but the main ECU tends to fail and have to be resoldered.

Of course, mostly people don't care because after 20 years you expect the vast majority to be scrapped, but in any case, you still design for a certain lifespan.

mjevans• 7 years ago

As a suggestion for a general re-design to solve this issue...

Cluster the consumable components together. Flash and battery. Attach the storage via iSCSI or something similar, have the car's main computer perform encryption/decryption (rather than trusting a component expected to be swapped).

In the case of a battery swap the bulk contents of the (encrypted) external storage can also be cloned. That can probably happen in the timespan that a battery is topped up to charge, but would be more difficult for automated swaps at refueling points. A home/cloud backup and transferring only the new data might be sufficient.

Flash Memory Wear Killing Older Teslas Due to Excessive Data Logging: Report

Related Stories