Tesla Gets Carried Away Hyping Crash Test Results, Has Hand Slapped By Feds

http://www.forbes.com/sites/michelinemaynard/2013/08/22/tesla-gets-a-little-carried-away-has-hand-slapped-by-feds/

by w1ntermute • 13 years ago

75 70 13 years ago

forbes.com

nonchalance6• 13 years ago

Official announcements on http://www.safercar.gov/ (image) and http://www.nhtsa.gov/Misc/HeadLines (first headline)

Kylekramer• 13 years ago

So a problem with precision and judgment, but not integrity?

nonchalance6• 13 years ago

and integrity:

NHTSA does not rate vehicles beyond 5 stars and does not rank or order vehicles within the star rating categories.

So the following statement is flat-out untrue:

SETS NEW NHTSA VEHICLE SAFETY SCORE RECORD

If NHTSA does not rate vehicles beyond 5 stars, the maximum safety score is 5 and thus 5.4 is irrelevant

I think if the headline was "sets new vehicle safety score record", dropping the NHTSA acronym, then it would be fine.

thezilch2• 13 years ago

NHTSA doesn't PUBLISH ratings above 5 stars. Regardless, the 5.4 IS relevant.

invisible3• 13 years ago

The Tesla blog post makes it abundantly clear that the 5.4 is the report the manufacturer receives (not the issued rating which is capped at 5). Why does it stop at 5 if you can still die in those cars?

revelation• 13 years ago

It says as much in the official Tesla press release:

NHTSA does not publish a star rating above 5, however safety levels better than 5 stars are captured in the overall Vehicle Safety Score (VSS) provided to manufacturers, where the Model S achieved a new combined record of 5.4 stars.

You can't blame Tesla for taking that number and running with it, given that (1) modeling car safety from two seaters to SUVs on a range from 1 to 5 stars is ridiculous and (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further.

apalmer• 13 years ago

Nothing wrong with the 5 star rating system as far as it goes... the marketing 'usefulness' of a measure is not the be all and end all, in that regards it is very different from video game and movie reviews.

jlgreco3• 13 years ago

This makes me wonder, do they adjust the rating system over time, and if so, are they doing it frequently enough? You can buy five star cars today, and I assume you could buy 5-star cars 15 years ago, but 5 star cars from 15 years ago wouldn't get rated as highly today, right?

Maybe they need to be more aggressive with their adjustment so that the minimum expected really is 1 star.

gamblor9562• 13 years ago

Safety ratings are adjusted annually, so a car that was rated 5-stars a decade ago would probably be lucky to get 2 stars today.

smackfu4• 13 years ago

The VSS is not a star score though. It looks like Tesla got a 0.43 VSS. Tesla was the one who converted that to a 5.4, by making up a methodology, and their wording on that is definitely a bit unclear.

CatMtKing• 13 years ago

I demand p-values!

sum(p(accident type) * p(estimated of surviving corresponding safety test))

throwaway_yy2Di• 13 years ago

Nitpick: Those things are not p-values.

Strilanc• 13 years ago

P-values aren't probabilities you should multiply or add.

At best you're computing odds according to the null hypothesis (probably not what you indented). At worst it's as meaningless as adding meters to seconds.

ars2• 13 years ago

They have those. Maybe not exactly in the format you specify, but there are statistics for various star ratings and accident survivability.

lmkg• 13 years ago

(2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further.

That's not a bad thing. We wanted safer cars. We started measuring safety, and publishing the measurements. As a result, safety improved. Working As Intended.

gamblor9562• 13 years ago

The threshold for each star is increased each year, based on the available safety features and safety performance of cars. A car that received 5 stars under the 2000 rating system would receive only 2 stars today because it lacks some of the now-required safety features.

3 is the new horrible simply because that is the lowest that a commercially viable car can go--Chevy and Subaru have released sub-3 cars in the past and suffered for it with poor sales. They and others learned that consumers wouldn't buy cars that were rated below-average safety-wise and have vastly increased R&D spending on safety features. That's a good thing for consumers.

ars2• 13 years ago

rated below-average

It better not be a real average if everyone is always above average!

You mean to say poorly rated, not below average, since it's not an average.

stcredzero2• 13 years ago

Here is how star rating actually works:

For a truly excellent product, the distribution should approximate a power law graph:

    XXXXXXXXXXXXXXXX
    XXXXXX
    XX
    X
    X

If you see something like these:

    XXXXXXX
    X
    X
    X
    X


    X
    X
    X
    X
    XXXXX


    XXXXXXXXXXXXXXXX
    XXXXXX
    XXX
    X
    XXXXXXXX

Then ask yourself if the sample size makes this plausible, in which case the star rating isn't worth much, because there isn't enough data. Ask yourself if it's possible that someone has been gaming the system for or against.

The uppercase gamma-shaped graph is either someone gaming the system or it's a power law graph hidden by aliasing. (In which case, the thing is of godlike quality.)

We should do away with star ratings, and instead just present an aggregate "rating." Genuine customer ratings are valuable, but are almost immediately gamed by cheaters. If I were Apple or Amazon, I would build stealth captchas and "honeypot" traps for ratings manipulation, and figure those into the rating. Be sure to separate detection from consequences by a random time interval of at least a few days. This makes the "development cost" for the cheaters much higher.

The problem with star ratings is obviously what PG refers to as "schlepping". Is anyone tackling this already?

EDIT: I would surmise that AirBNB is somewhat tackling this problem already.

hamburglar• 13 years ago

ObHeadlineHypeComplaint: that's a "hand slap"?

asperous• 13 years ago

I take it from the article, nothing but a statement from the safety agency asking "car manufactures" (not even mentioning Tesla) to not misrepresent their data to the public.

smackfu4• 13 years ago

I wonder how many other cars got more than five stars by Tesla's methodology? From the graph, it looks like some have very close ratings.

And clearly, the goal here is to stop every manufacturer from just coming up with their own rules for interpreting the score that give them a different rating from the one that the NHTSA gives. That's definitely worthwhile.

jcampbell12• 13 years ago

"The agency has dominion over any part of a vehicle sold in the United States, from the headlamps to the owner’s manual, but can’t regulate the speech of a corporation or its employees."

True, but the agency gives legal cover. When there is a question in court as to whether the safety systems performed "as advertised", the NHTSA is free to say that Tesla didn't follow the advertising guidelines, and therefore cannot comment on the performance of the safety system.

Drug companies can also advertise drugs outside the FDA guidelines, and would win a court challenge on 1st Amendment grounds. None do it because it would expose them to unbounded liability.

lvs2• 13 years ago

Drug companies can also advertise drugs outside the FDA guidelines, and would win a court challenge on 1st Amendment grounds. None do it because it would expose them to unbounded liability.

No, no, no. This is utterly and confirmably false. It is illegal under the Federal Food, Drug and Cosmetic Act to advertise a prescription drug for a use that is not FDA approved, and it is within the FDA's mandate to recommend investigation to the Department of Justice in such circumstances. The FTC is authorized to oversee advertising for over-the-counter drugs under the Wheeler-Lea Act. As a recent example, Pfizer agreed to settle with the DoJ earlier this summer for half a billion to end an investigation brought under these very circumstances[1].

[1] http://www.newstatesman.com/business/business/2013/07/pfizer...

jcampbell12• 13 years ago

Jesus man. You seem awfully certain. Here is a ruling from December that affirms 1st amendment rights to market off label uses:

http://www.reedsmith.com/files/uploads/DrugDeviceLawBlog/Car...

" Caronia argues that he was convicted for his speech -- for promoting an FDA-approved drug for off-label use -- in violation of his right of free speech under the First Amendment. We agree. Accordingly, -3- we vacate the judgment of conviction and remand the case to the district court. "

It is uncertain whether marketing off label is protected by the 1st amendment. The courts don't all agree.

will_brown2• 13 years ago

This is an interesting 1st Amendment discussion (relating to the article...I am not touching the separate FDA/drug issue).

The most analogous case that comes to mind, is the recent California small claims case against Honda.

Facts: According to the EPA (who independently tests vehicles fuel efficiency) the Prius gets an average of 51 MPG. The Plaintiff claimed she only got an average of approximately 30 MPG. The Plaintiff won the lawsuit against Honda for fraudulent misrepresentation winning $10K the small claims limit in the jurisdiction. However, one of the most interesting arguments (and legitimate) was that while Honda does advertise the 51 MPG, that Honda could not be fraudulently misrepresenting the MPG, because Honda had no control over the testing or findings handled by the EPA, and going further Honda does not have a right (1st amendment) to contradict the "EPA sticker". Ergo according to the argument Honda should not be liable, rather the EPA - nevertheless the Court found Honda liable for fraudulent misrepresentation.

Assuming this law is on point and controlling, for example, if the Tesla were to under perform in a real crash and Tesla was sued, Tesla may be liable despite having no control over the NHTSA test or findings. Going further, if the dicta from the Honda case is also on point, if Tesla's internal crash tests contradict NHTSA's ratings, Tesla would not have a 1st Amendment right to contradict the NHTSA findings publicly.

Note: That SCOTUS makes a distinction between "commercial speech" and other forms of speech, and the government has a lot more power to regulate commercial speech than say political speech.

tadfisher• 13 years ago

Are you referring to a suit against Toyota or a Honda hybrid vehicle? Prius is manufactured by Toyota, not sure why Honda would be involved.

skue2• 13 years ago

It was easy to see that coming once the media story took off the way it did. But this just generates more coverage of Tesla's results.

I doubt that anyone at Tesla is feeling the least bit chastised. Musk is certainly not the type of person who is afraid to generate a little controversy.

hauget• 13 years ago

why is shit like this on the HN frontpage?

650REDHAIR• 13 years ago

Because it's getting upvotes, mostly.

skue2• 13 years ago

HN isn't just about software/engineering, it's also about entrepreneurship.

Tesla is a startup with cool technology. They're taking on a market where startups never succeed and are navigating significant challenges regarding new technology, consumer bias, competitors, regulatory issues, business model, press relations, and more. Some of us find it fascinating and enlightening to watch them take this journey.

Edit: Reworded.

nonchalance6• 13 years ago

The Tesla Motors post that heralded this news (https://news.ycombinator.com/item?id=6240862) was on the front page for nearly a full day and garned 794 upvotes. HN readers are clearly intrigued by TM

Uchikoma• 13 years ago

Because Elon Musk is god here and worshipped.

robterrell• 13 years ago

I really like Tesla and want them to succeed, but they've got some weird ideas about good PR. The cost calculator that factored in the cost of pump time at your billable rate; the over-the-top posture against the NY Times reviewer who discovered that the battery heater could draw enough power to affect range; and now this, where they made up their own methodology to convert VSS into stars and found themselves to be the best. (I'm not even considering the weird stuff like Elon Musk offering publicly to consult on Boeing's batteries, although I think that points to the source.) Aren't the "best car ever made" reviews enough?

eli3• 13 years ago

You left out Tesla suing television show Top Gear for libel. (Which they lost despite the UK's infamously plaintiff-friendly libel laws.)

TrevorJ• 13 years ago

I think that may have been less about winning the case, and more about getting the word out that the 'test' Top Gear did was edited in a way that maximized humor vs. being 100% accurate.

tokipin• 13 years ago

part of the issue is how quick people are to judge, combined with the particular historical perception of electric cars. Tesla has to be very aggressive about the image of its vehicles, otherwise the prevailing prejudices about electric cars in general will override the image of its own cars regardless of how awesome they may actually be

etler2• 13 years ago

Their vehicle safety score was 0.42, the closest one was 0.50. That's a big difference. The lowest with a 5 star rating was 0.66 and the average was 0.60. That's totally something they should be allowed to brag about, but they should have just used those numbers or clearly explained their methodology. I think they were just trying to show off that number in a familiar way, and I can't blame them too much for that.

eridius3• 13 years ago

Lower VSS is better? I wouldn't want to try and explain that in marketing.

cma• 13 years ago

the over-the-top posture against the NY Times reviewer who discovered that the battery heater could draw enough power to affect range

Actually he claimed to have the heater keeping the cabin at near-freezing temperatures, and the cabin temperature graph captured from the car strongly disputed that. Going over-the-top at the guy was all they could do after he had made so many different false claims. He was probably right with some of the claims, and the proof of him circling around in the parking lot wasn't as open-shut as initially presented, but when you are hit by that many misrepresentations at once you have to fight back fast.

lnanek2• 13 years ago

You mean the reporter who drove around in circles trying to kill the car off and lied about the heater settings he used, also in an attempt to kill the car? I think you missed the whole week there where the actual drive data proved he was a fraud.

stcredzero2• 13 years ago

I don't like that reviewer, because he's evidently a very sloppy reviewer with a pretty poor understanding of physics and mechanics involved. However, I no longer think he was gaming the system deliberately. Assume incompetence before malice.

asdfs• 13 years ago

He drove around a parking lot once, searching for the charging station.

jhonovich• 13 years ago

This is nonsense: "can’t regulate the speech of a corporation or its employees"

Entities have rights over how they are used in promotions, i.e., the right of publicity / personality rights http://en.wikipedia.org/wiki/Personality_rights

For example, Consumer Reports No Commercial Use Policy - http://www.consumerreports.org/cro/about-us/no-commercial-us...

etler2• 13 years ago

It just seems like the NHTSA is just reiterating that there's no official higher than 5 star rating to protect themselves, rather than scolding Tesla. Tesla did say that a rating higher than 5 stars does not exist, and that the extra high rating should be taken with a grain of salt seemed implied to me, but I guess they could have been clearer. The blog post was very in depth, so it's not like they were misleading people with their wording.

marvin• 13 years ago

Yeah. This is a non-story; it makes a good headline but no one has been slapped. Tesla just noted that they received some papers which noted that for the internal purposes of the ranking, it got 5.4 stars. Then the NHTSA made a clarification that the official ranking actually doesn't go past 5 stars.

It's good for Tesla shareholders that Forbes decided to run with this story, though. Free advertising. There is no way anyone will ever perceive a fight about whether the car should have 5 or 5.4 stars as negative.

Aqueous• 13 years ago

Better to ask for forgiveness than permission.

twoodfin• 13 years ago

Does it really hurt consumers if carmakers are bragging about doing well on tests?

I'd argue that it might: NHTSA wants to encourage automakers to build safer cars. One way to do that is to provide them with detailed information on the results ("scores") of its tests, so that automakers can benchmark their engineering against each other and tune individual safety features as needed. Another way is to aggregate those results for each model and publish a consumer-friendly figure that automakers are motivated to be able to advertise as a "stamp of approval".

What they don't want is automakers getting into NHTSA test "hot-rodding" games, focusing their efforts on topping the individual tests by some small relative figure in order to claim things like "safest car ever". In an environment like that, NHTSA has to either stop publishing detailed results, or keep the mechanics of their testing secret in an attempt to prevent "benchmark gaming" of the kind we see all the time in technology. That sounds like an NHTSA that is competing with automakers rather than working with them, and I suspect that is how consumers could be ultimately hurt.

Tesla Gets Carried Away Hyping Crash Test Results, Has Hand Slapped By Feds

Related Stories