How to defeat the purpose of IEEE floating point (2008)

http://yosefk.com/blog/consistency-how-to-defeat-the-purpose-of-ieee-floating-point.html

by luu • 12 years ago

35 29 12 years ago

markrages4• 12 years ago

The quest for consistency seems misguided.

If your code changes behavior based on the timing of the various sections, you have a bug, a race condition. The solution isn't to lock down your code and build with only one configuration. The solution is to fix the bug.

In the same way, if the behavior of your code is dependent on the order of operations in floating point, it is very likely that you are doing something wrong. The solution is not careful ordering of floating point operations, but using algorithms appropriate to floating-point (and avoiding floating point when it isn't applicable.)

Code that has been ported to many platforms tends to be more stable, because the heterogeneous environments have exposed bugs and flawed assumptions in the code's construction.

lmm2• 12 years ago

Simple use case: a multiplayer game containing physics. I'd like the physics to be reasonably accurate, but more than anything it needs to be consistent, because I want to get the same results on both players' machines, and then the only thing I have to send over the slow internet is a command stream.

markrages4• 12 years ago

But if infinitesimal disturbances to the results makes a practical difference to the game, then your physics engine is not stable.

idlewords• 12 years ago

I read the article so you don't have to! The author dreams of consistent behavior so he can get:

- Reproducible results across platforms

- Reproducible results with different build settings (such as debug builds or optimization levels)

- Useful test cases for regression testing

- Test cases that help him find misbehavior in the optimizer

- An improved chance of finding more subtle bugs, absent 'noise' from small changes in floating point operations

In particular, he gives examples of how changes to nearby code can change the output of floating point operations, due to hairiness in how registers are allocated. You can see how this would make testing quite painful.

These all seem like worthy goals that deserve a better response than "you're doing it wrong", along with some non-sequitur about race conditions.

markrages4• 12 years ago

Regarding race conditions, I was referring to this:

It's not just floating point that's inconsistent across modes – it's code snippets with behavior undefined by the language, buggy dependence on timing, optimizer bugs, conditional compilation, etc.

pjc50• 12 years ago

There's quite a few algorithms that are generally vulnerable to precision issues because they can affect comparison order, such as k-means clustering. "Don't do that then" is not a useful answer.

marcosdumay• 12 years ago

Lots of problems dictate code that depend on the ordering of floating point operations. It's not something that you can design away.

lmm2• 12 years ago

This kind of thing makes me happy I work on the JVM. strictfp may be "cowardly" but it's relatively easy to use.

Terr_3• 12 years ago

I understand that avoids some platform-capability differences, but does it also protect from issues with execution-order or ghostly-coupling between threads?

tomlu2• 12 years ago

I have never heard of those problems, what are they? Hardware bugs?

pcwalton• 12 years ago

In C terms strictfp just sets FLT_EVAL_METHOD to 0, which in today's SSE and 64-bit world most C compilers do anyway.

shawnhermans• 12 years ago

Maybe I am wrong, but is the author just looking for a decimal type? Those exist in most major languages, right?

wolf550e• 12 years ago

The performance of decimal arithmetic on hardware not recently made by IBM means it cannot be used in most places.

maxlybbert2• 12 years ago

The original post said he'd be happy to trade performance for consistency.

He could also use fdlibm ( http://netlib.org/fdlibm/index.html ), at least for anything other than primitive operations. But, again, it trades performance for consistency.

TheLoneWolfling2• 12 years ago

Decimal types suffer from many of the same issues.

Sure, 0.1 can be represented exactly. But that's about the only issue with floating-point that that solves.

srean2• 12 years ago

I have often found myself closer to the opposite end of the spectrum. Many machine learning algorithms and applications (certainly not all) can tolerate a fair degree of inaccuracy. This is often justified by the fact that the training data is typically noisy to begin with and is a random sample from the space it lives in. If your algorithm can be stable under those conditions it ought to be stable under floating point jitter.

This usually means you need regularization, or smoothing, or low pass filter, or averaging in some guise or the other (in some abstract way they are the same thing). There are some frequently used operations in ML that are very unstable under inaccuracies, numeric differentiation is one, just dont do it without smoothening. Inverting matrices is another, but you better have a real solid reason to ever invert a matrix, yes there are reasons when you do really want the inverse.

In any case the final point is that I have had to baby talk the compilers into, "yes, please please dont worry about it, just do these commutative (for real numbers) operations in any order that you want". This exposes more parallelism and other compiler optimizations. Often I would gladly take the hit over some precision in higher order decimal places to gain speed. But this should not be done blindly, and a smattering of numerical analysis helps gauging when it would be safe to do so. I doubt whether numerical analysis figures prominently in the trajectory of a graduating machine learner, although I think it ought to.

TheLoneWolfling2• 12 years ago

One of the advantages of languages with N-ary operators.

* (a,b,c) can take advantage of those tricks, whereas the programmer can still be explicit and go * (a, * (b, c)), for example.

srean2• 12 years ago

Indeed. Do you have some specific examples in mind, I would love knowing more. The usual suspects, for example fold_left, fold_right would typically adhere to fixed order for floating point operations. A reason Java is a FLOPs killer is that its standard is too strictly specified, it does not leave much room for the runtime to take advantage of these features. Even order of argument evaluation for functions are specified to be left to right.

jephir2• 12 years ago

If you're a game developer, you should look at this Floating Point Determinism article, which shows some additional techniques for implementing deterministic physics with floating point.

http://gafferongames.com/networking-for-game-programmers/flo...

rurban• 12 years ago

Getting consistently worse results? :) He didn't even mention when matlab 5 started using his 64b only recommendation, the results were getting in-precise and matlab was getting unusable. http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf

I cannot take a floating point rant seriously without knowing Kahan's arguments. And a crazy requirement to compare it to slow and false debugging code does not really help.

How to defeat the purpose of IEEE floating point (2008)

Related Stories