Data scientists need to learn about significant digits

https://lemire.me/blog/2019/01/29/data-scientists-need-to-learn-about-significant-digits/

by ibobev • 3 years ago

68 65 3 years ago

Omg yes. My coworkers often specify parameters (the results of what we would have called "regression" or "fitting" but what is now called "machine learning") to like 14 decimal places when we only know them to 20%. It would have yielded a failing grade in my high school chemistry class.

They are like "Chesterton's digits" when they appear unexplained in parameter files. Do we really know this number or is it just a wild-assed guess?

dickersnoodle2• 3 years ago

puts on grumpy old man hat Yeah, back in my day you had to learn about significant digits because we still used slide rules and the calculators didn't have as many functions and we liked it! removes grumpy old man hat

tobinfricke5• 3 years ago

That's not the reason sigfigs are important.

bigbillheck2• 3 years ago

Do we really know this number or is it just a wild-assed guess?

Does it matter? You have 23 bits of mantissa to fill one way or the other, and if you don't know the lower, say, 12, one pattern of bits is as good as any other pattern.

diarrhea• 3 years ago

Precision beyond what can be justified by the uncertainty of the original input is wrong, not just as-good-as.

tobinfricke5• 3 years ago

That would be fine if the number were accompanied by a confidence interval.

pessimizer• 3 years ago

The reason you get rid of those extra digits is because it will always be tempting to find meaning in them.

7402• 3 years ago

I was taught about significant digits in Junior High School science classes. (in a public school in US, late 1960s.) Answers were marked wrong if they had too many digits that weren't significant. Is this no longer taught in school?

atkion• 3 years ago

This was definitely still taught in my public high school in the 2010s - if they stopped teaching this, it was very recently.

Granted, the teacher that tried to teach me this had no understanding of it themselves and would incorrectly mark answers wrong when they had the correct number of significant digits, so it was pretty botched in my case. Still was attempted in the curriculum though.

2muchcoffeeman• 3 years ago

I think some of the problem is that you can get into computing through many pathways now, and you don’t have to come through some science and math stream from high school. Even when I was doing physics I always had to remind myself of the accuracy.

koito17• 3 years ago

Significant figures have always been taught in chemistry courses. I remember students getting confused at notation like "100. g" representing 100 grams with 3 significant figures.

SoftTalker• 3 years ago

I don't remember it in junior high, which for me was the late 1970s. High school yes, definitely.

bluenose69• 3 years ago

I was taught this in High School and I know that it is still taught. We learned it because the option for such (and many other) errors was a zero on the work and likely an F in the class overall.

Teachers have a lot less latitude nowadays; see [1] for an example that is shocking in the 1960s context.

1. https://www.cbc.ca/news/canada/edmonton/teacher-who-gave-zer...

lamename2• 3 years ago

Do people really report 5 decimal places?

In my experience reporting models in industry, even 1 or 2 decimal places are irrelevant to the overall decision to be made, at least if the number is around "87%" as stated in the example.

Of course it's context dependent. If you're evaluating incremental improvements perhaps that warrants more careful precision.

After all, by definition each decimal place is 10x less important.

ta988• 3 years ago

Highly field dependent. In most of biology, two figures is often way above pipetting skills of most mortals (and robots). In physics in my meager understanding they are looking at close to 10 in some cases.

lamename2• 3 years ago

Agreed. Based on the title, I took the context of the article as generic data science/analytics in industry, not cutting edge bio or physics.

fipar• 3 years ago

After all, by definition each decimal place is 10x less important.

I’m not sure about that, since it depends on the error margin tolerance one has.

Regardless, my biggest gripe with most reports of quantitative information I stumble across is that, lacking mention of significant digits accuracy, it’s very easy (and I’ve seen this happen a lot) for people to attribute more accuracy than what is there to a value.

One example: AWS cloudwatch reports external replication delay in MySQL Aurora in milliseconds, which is impossible, since the value reported by MySQL is in seconds. Yet many times I’ve been involved in discussions with people who say “our system can tolerate at most 500ms” of replication delay for reads, because they see the value reported in that unit.

atoav• 3 years ago

I have a 9 1/2 counts multimeter, and when I calibrate DACs I need all of those. The last two digits are a bit noisy but hey.

There is no rule of thumb, except that people should understand that I can give the illusion of precision to every measurement by capturing crap. Those 32 bits on your laptops onboard audio interface are the real banger, when even the least significant digits in 16 bit mode are filled with electromagnetic interference and noise from the power rails. Real, actual precision is way more hard to reach and you will know when you reach it, because you will have spent a lot of money on the way.

Whether you need it, or wield it correctly is a different story.

allenrb• 3 years ago

What sort of meter is this? I’m aware of a few 8 1/2 digit units but thought that was it.

SamBam• 3 years ago

The other thing people get wrong is treating significant figures and decimal places as the same thing.

123.45 Kg and 7.32 Kg have the same decimal places, but the former has more significant figures, and implies greater precision.

jimmaswell• 3 years ago

Always seemed odd to me you can have a ruler that measures millimeters and get more significant digits measuring something to 9mm vs 12mm. The measurements are equally precise but one has more significant digits. Write it in hex and now they're the same, 9mm and Cmm. How does this make any sense? If I measure 123mm and divide it by 1.23 seconds I get to keep 3 sig figs but if it was 99mm I'd be forced to drop one? At least writing everything in scientific notation sidesteps this.

prepend• 3 years ago

I hate how values are rounded for presentation and precision is lost.

boredemployee2• 3 years ago

business ppl ~always~ prefer that approach.

When we go to industry or "the real world", it's as if we are forced to unlearn all good practices

SAI_Peregrinus3• 3 years ago

If you have a material sample with measured length of 1cm ±0.5cm, and a measured mass of 3g±0.5g, then reporting 3.3333333333g/cm for mass per unit length is just lying. It's between 0.14 and 0.6, so "about 0.3g/cm" is not lying, whereas "0.333333333333g/cm" is.

mitthrowaway22• 3 years ago

What do you think about 0.333 ± 0.222 g/cm?

tobinfricke5• 3 years ago

This is the issue that is addressed by paying attention to "significant figures." Digits that are significant are shown. Those that aren't, aren't. It's appropriate to round where your precision ends.

ayhanfuat• 3 years ago

I think a distinction should be made for measurements and predictions. If you are measuring something then you have a scale, and you probably have a sense of the margin of error. That is easy to translate into significant figures. When you make predictions, things are much harder. There is also the question of whether you should take the variance of the model into account, or its accuracy.

montebicyclelo• 3 years ago

In addition to the uncertainty of the value, there's also the context of how it will be used. E.g. for some quick business decision that needs to be made, it may be fine to round to 1 sig fig, e.g. 71.244% -> 7/10 -> the majority.

uxp8u61q• 3 years ago

How can someone call themselves a "scientist" and not know about significant digits? "Data technician" would be a more accurate job title.

dekhn• 3 years ago

data scientists are not scientists, they are statisticians who make more money and work in the bay area.

dmd2• 3 years ago

My favorite example is GPS locations stored as and then displayed as floats or even doubles, giving a location precise to a handful of nanometers.

yodon• 3 years ago

It's not uncommon to have 10m accuracy with consumer gear, which is 4 digits to the right of the decimal in degrees, and differential GPS can do around 3 orders of magnitude better, or around 7 digits. That's hitting the limits of float precision. By the time you start doing calculations and error propagation on the calculations, doubles are absolutely reasonable choices.

Are all location estimates that accurate? No. Should the precision of your location estimate be limited by your storage format? Of course not. That's why doubles are frequently used.

dmd2• 3 years ago

The point isn’t how you store them. The point is how you display them.

zoomablemind• 3 years ago

It'd be even more insightful if the significant figures discussion touched upon rounding that may be inherent to the calculations.

Take sin(x), C math would give a different approx value for sin(x+2*pi) even though it should be the same. Well, if only pi was known exactly...

boredemployee2• 3 years ago

In my experience, non tech/engineer/math people dislikes more than 1 or 2 digits.

Obs: Don't wanna sound a douche here but I can't help but notice that they're talking about accuracy citing Wikipedia.

SAI_Peregrinus3• 3 years ago

Software often ignores significant digits. It'd be buch nicer if math libraries had built-in significance propagation.

cowthulhu• 3 years ago

Working in finance, significant digits are (of course) super important. You always want to assume a stakeholder doesn’t have a perfect understanding of the data definition, so provide less precision (even if you know the actual value to a cent) to avoid the stakeholder misusing the data given a slightly flawed understanding of some obscure edge case.

If the number stays at 32.56m for a period of time, you want them to consider it sideways movement rather than start worrying about the fact that it did technically go up by a few hundred dollars.

lifeslogit• 3 years ago

I'm pretty sure most of us DS know about significant digits and are usually calculating the maximum to enable future flexibility. For a single output, I can understand how you'd be upset we don't round. But for the 20+ column tables we usually build, I've found most will calculate the maximum scale allowed by the database and call it a day. The best of us certainly find the right formatting for the presentation.

cauch4• 3 years ago

I agree with that.

It reminds me of the tomato. Some people don't know the tomato is a fruit. Some people know the tomato is a fruit and treat them like a fruit. Some people know the tomato is a fruit and still treat them like a vegetable because it is what makes sense.

In practice, I have never saw someone mistreating a badly written number in a way that had any impact. If they don't know themselves the concept of significant digit and believe the number is 1.234567890 precisely and not 1.234567889, what would be the wrong decision they will take that they would not have taken if the number was 1.234567889?

It starts to matter when you have 10% or ~100% uncertainty, in which case, writing 1.2 or 1 is still not enough to convey the meaning to someone who does not get significant numbers, because for them, 1 = 1.0000 anyway. In this case, you need to explicitly explain the limitation on decision due to the uncertainty.

In practice, splitting hairs on the significant digit convention is just missing the point: if you apply the convention to people who are not informed on the precision, they will make bad decision anyway even if technically the number has the correct significant number.

nicklecompte4• 3 years ago

The problem is that it's not just "presentation": being sloppy about significant digits (or precision generally) early in the computation leads to bad statistical reasoning much later in the problem.

If your variable is x +/- 0.05, then 1/(x +/- 0.05) != 1/x +/- 0.05. If you're not careful about this when doing computations, you'll end up with answers that aren't actually meaningful. The computational implementation of these equations is only concerned with machine epsilon, but each one of those 20 database columns has a real-world +/- delta which isn't being correctly considered.

cauch4• 3 years ago

But the error propagation is not transmitted by the significant number. x, y being written with the correct number of significant number will not lead to f(x, y) being written to the correct number of significant number.

Usually, the best approach is to propagate the uncertainty, for example by saving the uncertainty as another variable in the database and using it directly when the number is used. If you do that, there is no practical needs to lose time to format the numbers. Using significant numbers seems a "cheap trick" that risk to mislead you more often than help.

wdkrnls• 3 years ago

No, they need to learn the importance of transparency. Chemists need to learn that too but I am not holding my breathe. They are too busy trying not to blow themselves up to care. Significant digits, like prediction intervals and confidence intervals are all rendered meaningless by misunderstanding from just one thing of a great many implicit assumptions upstream of the analysis.

CrazyStat3• 3 years ago

I have mixed feelings. On the one hand, yes, people working with data should be more thoughtful about how many digits they report. But I’m not a big fan of the proposed solution (report everything to a small fixed number of significant digits, like two).

Massive computer experiments are common and (often) cheap. If I’m comparing two classification models on a large dataset a difference between 97% and 96.8% accuracy might well be both statistically and practically significant.

An error I’ve seen professional statisticians make is asserting that averaging samples (or performing more complex statistical analyses like regression) justifies increasing the number of significant digits. For example, if I have a hundred observations measured with two significant digits of precision, the mean of those observations is asserted to be accurate to three significant digits, the standard deviation of the mean being 1/10th that of each observation. This is a natural conclusion if you treat the significant digits as accounting only for independent random errors, but in fact significant digits are also used to account for systematic errors in measuring equipment. A scale that measures down to 0.1g, for example, may be miscalibrated by 0.03g. This makes reporting the mean of 100 samples down to 0.01g obviously nonsensical.

In short, how many digits to report deserves more thought than either most data scientists or the author give it.

YetAnotherNick3• 3 years ago

97% and 96.8% accuracy might well be both statistically and practically significant.

I don't think author ever objected to this.

CrazyStat3• 3 years ago

Which answer do you give? Whatever your software tells you (e.g., 87.14234%) or a number made of a small and fixed number of significant digits (e.g., 87%).
The latter is the right answer in almost all instances.

and

Most human beings are happy with a 1% error margin.

While he doesn't say it as a rule outright, the author repeatedly uses a "small and fixed" two significant digits. He later says this:

You must choose the number of significant digits deliberately.

which I agree with, but is at odds with the "small and fixed" dictum with which he leads off the post.

pella• 3 years ago

Xkcd Coordinate Precision: https://xkcd.com/2170/

NASA: pi 15 digits

"For JPL's highest accuracy calculations, which are for interplanetary navigation, we use 3.141592653589793. Let's look at this a little more closely to understand why we don't use more decimal places."

"Let's go to the largest size there is: the known universe. The radius of the universe is about 46 billion light years. Now let me ask (and answer!) a different question: How many digits of pi would we need to calculate the circumference of a circle with a radius of 46 billion light years to an accuracy equal to the diameter of a hydrogen atom, the simplest atom? It turns out that 37 decimal places (38 digits, including the number 3 to the left of the decimal point) would be quite sufficient. "

https://www.jpl.nasa.gov/edu/news/2016/3/16/how-many-decimal...

esafak• 3 years ago

Data scientists have better ways of quantifying uncertainty. They can estimate statistics (functions of noisy estimates) by modeling the noise distribution, and report the credible- or confidence interval. And this is standard practice, so there is nothing to worry about.

kstrauser• 3 years ago

Everyone does. I grind my teeth at things like “blue whales can be up to 30 meters (98.43 feet) long” or “my dog weighs 20 lbs (9.072 kg)”. Stop that, please.

slooonz• 3 years ago

Yes, you have 10,000 samples and properly classified 5,124 of them so your mathematical precision is 0.5124

That’s the point. That’s the correct usage of significant figures. Your inputs have significant figures ; it determines exactly the significant figures of your output. Significant figures have nothing to do with things like sampling variance.

What you’re looking for is not significant figures, it’s confidence intervals.

Data scientists need to learn about significant digits

Related Stories