Posits, a New Kind of Number, Improves the Math of AI

https://spectrum.ieee.org/floating-point-numbers-posits-processor

by cpeterso • 4 years ago

220 101 4 years ago

spectrum.ieee.org

novaRom• 4 years ago

"They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption."

Yes. Nothing for free.

evdubs• 4 years ago

Part of it is not wasting so many bit patterns on valid NaN values. That is sort of free, I think.

Whereas floating point numbers are polluted with NaN values, posits are cleansed of such unclean special values.

https://www.cs.cornell.edu/courses/cs6120/2019fa/blog/posits...

js84• 4 years ago

I think NaNs are pretty useful. I think we should have reserved 0x80000000 as a NaN value for integers.

vintermann2• 4 years ago

Of course nothing for free, but the point is to trade things you don't need for things you do.

I also expect that chip area and power consumption for IEEE floating point has been optimized quite a bit more than for this novel number format, so there may be some efficiency left on the table on those particular metrics.

bXVsbGVy4• 4 years ago

The posit is a compressed floating point encoding with a flexible data structure with a trade-off of precision (the number of bits after the dot) and magnitude (the absolute value).

It uses more hardware because the posit is decoded into a floating point larger than a equivalent IEEE754 with the same number of bits. So, the logic units needs to be larger.

MikeDelta• 4 years ago

I thought this was about different numbers like rational and complex, but it is about number representation (of floats), see also https://en.wikipedia.org/wiki/Unum_(number_format)#Unum_III

parasti• 4 years ago

Unfortunately the Wikipedia article is extremely light on details of the actual format.

zeotroph• 4 years ago

The(?) posit website has a pdf[1] which goes into more details under "3.3 Posit format encoding". It also mentions a "quire" (and the footnote for the quire value is self referential?) , can anyone explain that concept?

1: https://posithub.org/docs/posit_standard-2.pdf

btdmaster• 4 years ago

I have found [1] and [2] but it is not clear how that helps with small numbers.

[1] http://raden.fke.utm.my/blog/positnumbersystem

[2] https://www.johndcook.com/blog/2018/04/11/anatomy-of-a-posit...

FabHK• 4 years ago

Interesting. So, Posits are basically Unums, v3.

I’d love to know what Kahan (the father of IEEE 754 floats) has to say about them. He thought that the „merits of schemes proposed in“ earlier versions were „greatly exaggerate[d]”.

http://people.eecs.berkeley.edu/~wkahan/UnumSORN.pdf

zasdffaa• 4 years ago

I did some reading on unum a while back, inc. Kahan's criticisms. There's another side.

There's a vid on youtube with Kahan and Gustafson talking over it. Kahan goes in like a wolf onto a lamb, much to gustafson's shock and hurt. Gustafson points out some of Kahan's criticisms are plain wrong, even citing the page in the book, then says something like "we should be working together on this, why aren't we working together?" Kahan doesn't respond.

Kahan may be right or wrong but his attitude is weirdly hostile, and Gustafson is no n00b about floats.

Edit: I think this is it (but it's been a while) https://www.youtube.com/watch?v=LZAeZBVAzVw

dr_dshiv3• 4 years ago

Beautiful idea: a number representation where the distribution of accuracy aligns with the distribution of typical use.

Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them. To fit into a designated number of bits, many real numbers have to be rounded. The advantage of posits comes from the way the numbers they represent exactly are distributed along the number line. In the middle of the number line, around 1 and -1, there are more posit representations than floating point. And at the wings, going out to large negative and positive numbers, posit accuracy falls off more gracefully than floating point.
“It’s a better match for the natural distribution of numbers in a calculation,” says Gustafson. “It’s the right dynamic range, and it’s the right accuracy where you need more accuracy. There’s an awful lot of bit patterns in floating-point arithmetic no one ever uses. And that’s waste.”

boberoni• 4 years ago

I like this idea of choosing a representation for numbers based on the use case. Like, "ints" and "floats" but even more specialized.

For example, what would be the most efficient binary representation of probabilities values between 0 and 1? In the future, I can imagine hardware + software that is specialized for examples like this.

dr_dshiv3• 4 years ago

Makes me wonder about quantum computing possibilities.

js84• 4 years ago

I feel like probabilities are best stored as inverse logistic function, because the closer the probability is to 0 or 1, the less you care about significant digits, because the events are going (not) to happen anyway. (That's why I am proponent of LNS.)

Animats• 4 years ago

Ah. So it's because the machine learning people mostly work in the -1 .. 1 range, but it's possible for their numbers to go outside that range. So they need a compact representation which uses most of the possible values for the range of interest. If you're down to 8 bit numbers, something like this makes sense for custom machine learning hardware.

I wonder if it has graphics applications for high dynamic range images. Probably not.

threatripper2• 4 years ago

The machine learning space probably has the easiest time to adopt a new number format. The players are big enough to define their own silicon and training/inference don't even need the same number format. As long as it works on their new specialized training hardware it's good to go.

The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.

KingMob• 4 years ago

It might. The whole reason behind gamma correction is human brightness perception is logarithmic. By having more resolution at lower values, posits could correct for the reason we need in gamma in the first place, though maybe, not as well as gamma itself.

But if it mitigates error accumulation in perceptually-friendly ways, it might still have value in calculations (once there’s native support).

philipov2• 4 years ago

Real numbers can’t be perfectly represented in hardware simply because there are infinitely many of them.

There are also infinitely many integers, but we can represent them just fine inside a finite bound. The problem with reals (and rationals to a lesser extent) is that, within any range we are interested in, a single real number can be infinitely 'long'.

LittlePeter• 4 years ago

That is a nice insight.

kgwgk• 4 years ago

We could even say that within any range we are interested in there are infinitely many of them.

visarga• 4 years ago

A single real number can contain all the information ever produced by humanity.

LordDragonfang• 4 years ago

Similarly put, the infinite number of reals is also a bigger infinity than the infinite number of integers (aleph-null).

ouid• 4 years ago

real numbers can't be represented in hardware because most of them are incomputable. It's not that there are too many of them, it's that any one of them is unlikly to be representable as the output of any program.

vintermann2• 4 years ago

Novel number representations are cool. I came across Paul Tarau's "hereditarily binary natural numbers" a few years ago, and via them, Knuth's earlier TCALC representation (one of the many small side projects Knuth has done over the years, which for some reason hasn't gotten much attention).

They go pretty much the other way than this representation: they allow huge numbers, much larger than can be represented in regular notation (incl. floating point) to be represented and calculated with efficiently and exactly. Tarau's system improves slightly on Knuth's in that representations are unique. He also proves that they at worst require twice as many bits as regular representation.

Knuth's and Tarau's system are variable-length and limited to naturals, but it seems like it would be easy enough to extend them to rationals and fix the representation length.

black_puppydog• 4 years ago

Kinda burying the lede there... The actual paper [0] is called "PERCIVAL: Open-Source Posit RISC-V Core With Quire Capability". :)

[0] https://ieeexplore.ieee.org/document/9817027/

freemint• 4 years ago

I kinda wish Gustafson would focus on something more productive but he is still trying to boil the ocean.

js84• 4 years ago

I don't understand why don't we just use https://en.wikipedia.org/wiki/Logarithmic_number_system.

It seems much simpler than the posit proposal and it has the same advantage (most precision around 1 and -1).

hakuseki• 4 years ago

A logarithmic representation is great for multiplication, but adding numbers becomes an expensive operation.

js84• 4 years ago

Today, both integer addition and multiplication is IMHO done in similar speeds in silicon, although multiplication circuits are way larger.

In the logarithmic representation, multiplication becomes addition. I don't think that just by doing everything in logarithms you really increase the overall complexity, rather you transfer it from multiplication onto addition, so I don't think adding logarithms should be more expensive than integer multiplication..

Should it then matter for neural networks, which IIRC require similar amount of additions and multiplications?

Even floating-point is actually trading off the simplicity of addition in order to make multiplication easier, because they are quasi-logarithmic representation.

However, I wonder if there is a numeric representation where the circuits for addition and multiplication are of similar complexity. Something like half-logarithm, which if applied twice, you get the logarithm.

eurasiantiger• 4 years ago

I like the concept, but hate the name. There is no semantic connection to what it represents, unlike with float, for example.

xeonmc• 4 years ago

floats = floating decimal point (or perhaps it should be called binary points)

Posits have floating range as well as floating precision, so doubly floaty.

Therefore I propose that they should be called "Lofties".

throwawaymaths4• 4 years ago

The name was created so someday someone might use it for machine learning tasks (perceptrons) and thereby finally create positronic brains.

bXVsbGVy4• 4 years ago

Something like "compressed floats" would express well the name.

vitus2• 4 years ago

Hm.

The article links to the posit paper [0]:

A posit processing unit takes less circuitry than an IEEE float FPU. With lower power use and smaller silicon footprint, the posit operations per second (POPS) supported by a chip can be significantly higher than the FLOPS using similar hardware resources.

The argument seems to be predicated on the cost associated with NaN-handling. The exposition comes across as somewhat arrogant, IMO:

If a programmer finds the need for NaN values, it indicates the program is not yet finished, and the use of valids should be invoked as a sort of numerical debugging environment to find and eliminate possible sources of such outputs.

Meanwhile, from the article where people actually tried implementing this thing on an FPGA:

They also found that the improved accuracy didn’t come at the cost of computation time, only a somewhat increased chip area and power consumption.

I wonder where the mismatch comes from.

[0] http://www.johngustafson.net/pdfs/BeatingFloatingPoint.pdf

NohatCoder6• 4 years ago

I think Gustafson may be the kind of liar that doesn't know he is lying. He knows saying that his design improves upon power and die size sounds good, so he says it.

throwawaymaths4• 4 years ago

Nope, I have seen it tested on an fpga, many years ago. The reference was Berkeley hardfloat. This might not be the best implementation ever, but it is the most accessible open source one.

doug-moen• 4 years ago

Posits have a variable size mantissa, and the largest mantissa for a given bit size (eg, 32 bit floats) is larger than the mantissa in the corresponding IEEE float. For example, in a 32 bit float, the IEEE mantissa is 23 bits, while the maximum size Posit mantissa is 27 bits. So a Posit FPU requires more mantissa bits than the IEEE FPU of the same bitsize, and this is why the Posit FPU will have a larger silicon footprint. IEEE needs a bit more silicon to manage all the special cases of IEEE logic, but apparently this is less than the extra transistors required for the Posit mantissa. (The extra cost for the mantissa is related to the better accuracy reported for posits.)

Gustafson's paper is old, and doesn't reflect the language of the recent Posit standard. He may have had a different silicon implementation in mind than what has been implemented. Gustafson says there is no NaN. But in the Posit standard, there is a single NaN-like value called NaR (Not a Real). In IEEE, 0/0 is NaN, while in Posit, 0/0 is NaR. The rules for NaN and NaR are different, so they have different names.

vitus2• 4 years ago

Aha. Thanks, that explains it.

It's possible Gustafson's paper doesn't consider the scaling as you get to wider types, or the presence of NaR removes enough of the savings from killing off NaN that it's a wash.

thuum7• 4 years ago

Simplest description by Gustafson https://posithub.org/conga/2019/docs/13/1430-John-Introducto...

loicd5• 4 years ago

I don't know about regime notation, but it is nice to see a new format with the most annoying ideas from IEEE754 removed:

- Two's complement instead of one's complement

- No infinities, no signed zeroes

- Only one exceptional value with the same encoding as the smallest signed integer: 10...0

The above means that comparisons work exactly like for signed integers (with the unique exceptional value behaving like -∞). Also, one can test whether x is invalid using `x == -x && x != 0` rather than the ugly `x != x`, i.e. no need to break reflexivity. Even if posits do nothing but remove the one's complement dust off IEEE754, that would be a positive change in my view.

phkahler• 4 years ago

There are a whole bunch of people who insist that lots of the "baggage" of IEEE 754 is really important and must not be abandoned. For most applications it is just extra garbage and for the few there are other ways to do what they want.

NohatCoder6• 4 years ago

That may be true, but you don't need a fundamentally different number representation in order to throw away most of it. Also, the die size cost is quite small.

The biggest saving you could make is probably foregoing exactly rounded results, and only stipulate that the result has to be within ±1 lsb of the true value. That would save multipliers from computing a bunch of bits that don't end up in the result anyway, except for the rare case where they decide the rounding. That would probably be a good trade-off for most AI chips. For general purpose CPUs I don't think it is worth the breakage.

noobermin• 4 years ago

We literally had a post about "old not being broken" on the front page yesterday with a significant discussion on developer churn and now people are already clamoring on other pages to deprecate floating points. Let's have a little memory to shake us out of our habits, shall we?

throwawaymaths4• 4 years ago

The big problem with posits is that its relative error depends on its value, this is terrible for a lot of engineering work and scientific simulations where you need to present an error estimate that includes computational error (for ML it's probably fine)

loicd5• 4 years ago

Isn't that also the usual argument against subnormal numbers? Anyway, you're right. According to that paper, section 4.3:

https://people.eecs.berkeley.edu/~demmel/ma221_Fall20/Dinech...

the relative error is bounded by 2^{-24} only in the interval [1.0e-6,1.0e6] for the Posit32 format, whereas it is [1.17e-38,3.4e38] for the IEEE binary32 format.

adgjlsfhk15• 4 years ago

IMO, this isn't a big deal. If you want rigorous error estimates, you need to use some form of interval arithmetic (or ball arithmetic). Also, these types of engineering and scientific work are pretty much all 64 bit, while POSIT is mainly useful for <=32 bit. My ideal processor would have 64 bit floating point (with Inf/NaN behavior more like posits) and possits for 16 and 32 bit.

LeanderK• 4 years ago

I think the problem is that you can't really restrict IEEEs values. I would like to have an error for infinities unless explicitly constructed (so number/number = infinity is an error but number * infinity = infinity is not).

So I understand the use case but they can't replace floats with those infinities remove imho. They complement them. You can work around those things by wrapping posits but sometimes you just need them (for example: I needed it yesterday). I don't always use them, but still somewhat frequently. They are not error codes, I work with the extended real number line [1] so infinity is a valid return value. What's the alternative...working with PositOrInfintiy?

I would probably use an exceptional value and later treat the exceptional like if I would encounter an infinity (usually they are later involved in calculations where infinites turn to 0). But that's just hard to understand for anyone but myself.

Maybe I work not low-level enough but I've never used `x != x` would use `x == -x && x != 0` because that's just not readable. I want a function (isinf, isnumber etc) and I am happy.

[1] https://en.wikipedia.org/wiki/Extended_real_number_line

loicd5• 4 years ago

I want a function (isinf, isnumber etc) and I am happy.

Sure, isnan(x) is what one should use. The fact that it is `x != x` is an implementation detail. The problem is that it is also a hack that breaks the usual mathematical axioms for equality and for order relations. For instance, if you want to sort floating-point values, you have to write your own comparison predicate in case there is a NaN, because a NaN is neither smaller, greater or equal to itself.

As for infinities, they somewhat work for real numbers, but it gets more complicated for complex numbers. For instance, Annex G of the C standard stipulates that an infinite complex number multiplied by a nonzero finite complex number should yield an infinite complex number. Sounds reasonable, but consider:

  (∞ + i∞)×(0 + i1) = (∞×0-∞×1) + i(∞×1+∞×0) = NaN + iNaN

So Annex G recommends some complicated functions to be executed at each complex multiplication and complex division, which makes little sense for most applications, and I suspect few people do that. As an aside, Annex G breaks the whole point of NaNs, because it stipulates that numbers like (∞ + iNaN) should be considered infinities rather than NaNs, which means that NaNs are no longer necessarily viral.

All in all, what I find frustating with these aspects of IEEE754 is that they complicates things under the hood, but the benefits seem to me limited to some specialized applications.

Sniffnoy2• 4 years ago

IEEE754 uses sign-magnitude, not 1's-complement. It doesn't use any sort of complementation.

Edit: Also, it looks like posits also use sign-magnitude and not 2's-complement? I am confused as to where you are getting this from. [Edit again: I was wrong about this, see below.]

loicd5• 4 years ago

Yes, sign-magnitude is the right terminology, my mistake, but too late to edit. As for two's complement, look at slide 18-19 from the link somebody produced below:

https://posithub.org/conga/2019/docs/13/1430-John-Introducto...

mr_gibbins• 4 years ago

As an aside, this is why I love HN (no /s). I have absolutely no idea what a posit is, and here is a community of experts nitpicking it, discussing it at length and in detail on this thread. No matter what the subject, SMEs always pop up out of nowhere and give the rest of us a crash-course education :)

skybrian• 4 years ago

Unfortunately it works the opposite way sometimes. It all depends on how political the topic is and whether any experts show up.

Kalanos• 4 years ago

"four-order-of-magnitude improvement in the accuracy of matrix multiplication"

derbOac• 4 years ago

Wouldn't this depend on what's in the matrices?

I've done work with big numbers and this format screams problems.

I can see how it would be great depending on the domain, and wish there was more diversity out there in the wild in this area but this seems a little hyped? Maybe I'm missing something. I'm not saying this doesn't seem useful, just that the article seems to be presenting it as a replacement for floats, when it might be better thought of as another option?

sweezyjeezy• 4 years ago

One particularly stark 'issue' I've noticed with floating point values in AI (which posits would seem to help with) is the difference between 0 and 1. In binary classification problems we generally assign 1 to one subset of our data, and 0 to its complement - there is not necessarily an inherent asymmetry here and the problem will often not meaningfully change if you switch labels 0<->1.

But... if we take for example float16, the smallest model prediction you can represent greater than 0 is 2^{−24} = 5.96*10E−8, but the largest you can represent _less_ than 1 is (binary) 0.111... = 1-2^{-10} = 1-9.77*10E-4. So values around 1 are about 4 orders of magnitude more quantized than those around 0. I don't know if that's necessarily a 'problem' but I have noticed this fact when looking at model predictions.

NohatCoder6• 4 years ago

That is fundamentally the same for Posits. If you want constant quantization you should use fixed point, or you could switch your inputs to be +1/-1 for symmetry in this specific case, but I doubt that this is much of a practical issue.

infogulch2• 4 years ago

Posits are pretty damn cool.

The posit standard was ratified in March this year. It's only 12 pages and shockingly comprehensible for a standard document: https://posithub.org/docs/posit_standard-2.pdf

Also in March Gustafson hosted the Conference for Next Generation Arithmetic (CoNGA) 2022 sorta 'inside' the Supercomputing Asia (SCA2022) conference. They covered a bunch of different areas not exclusively related to posits. Lots of cool topics including hardware implementations and optimizations, comparisons between different number formats, and a talk and panel from the posit committee about writing the standard and what's next for posits. The session videos are somewhat difficult to find so I made a playlist: https://youtube.com/playlist?list=PLBH9oUUfaYoQNl6-zr_ScsMp4... (Note the descriptions have talk titles and authors.)

The best video intro to posits is probably the one in 2019: https://www.youtube.com/watch?v=JJgT-YphE1Y

infogulch2• 4 years ago

Probably my favorite CoNGA2022 talk is the first one, A Case for Correctly Rounded Elementary Functions (starts at about 10m), which presents a methodology and library, already implemented into LLVM for floats, of elementary functions (lg, sin, cos, etc etc) that are 1. correctly rounded to the last bit, 2. are faster than the previous algorithms, 3. and can be downcasted to any smaller format while preserving correct rounding.

The posit standard requires correctly rounded elementary functions in order for an implementation to be considered compliant. This means that every standards-compliant posit implementation is now also deterministic.

gwbas1c2• 4 years ago

I remember going to an AI meetup where the guy presenting was proposing a totally new computing architecture.

I quickly suspected he wasn't very competent about how computers worked, and tried to probe a bit. I suspected he was mentally ill but could "talk the talk" enough to convince people his ideas had merit.

But this: A more efficient way of handling floating-point numbers, is probably something worthwhile. I wonder how hard it will be to recompile existing software to take advantage of posits?

7373737373• 4 years ago

https://en.wikipedia.org/wiki/Outsider_art

gwbas1c2• 4 years ago

Well, art is art. It doesn't matter who's making it, what their background is, or if they follow preconceived rules.

That doesn't apply to engineering because it either works or it doesn't. The presenter misunderstood basic concepts of information. If you don't understand those, you can't design a working computer.

peter_d_sherman• 4 years ago

"With their new hardware implementation, which was synthesized in a field-programmable gate array (FPGA), the Complutense team was able to compare computations done using 32-bit floats and 32-bit posits side by side. They assessed their accuracy by comparing them to results using the much more accurate but computationally costly 64-bit floating-point format.

Posits showed an astounding four-order-of-magnitude improvement in the accuracy of matrix multiplication"

This is nothing short of amazing!

I can just imagine future GPUs on FPGA's using Posits rather than (now old!) IEEE floating point formats...

(Also, on a probably unrelated note -- might Posits be where we get the future equivalent of Star Trek TNG's character Mr. Data's Positronic (Posit-ronic!) -- AI "brain" from? ???)

Posits, a New Kind of Number, Improves the Math of AI

Related Stories