Skip to content
Prev 33056 / 63421 Next

Floating point precision / guard digits? (PR#13771)

Dear Stavros,

Thank you for your fast reply!

So if I request a calculation of "0.3-0.1-0.1-0.1" and I do not get 0,
that is not an issue of rounding / underflow (or whatever the correct
technical term would be for that behaviour)?

I thought that guard digits would mean that 0.3-0.1*31 should be
calculated in higher precision than the final representation of the
result, i.e., avoiding that this is not equal to 0?

I am sorry if I am not from the field and although I have had some
basic background in numerics that has been more than a decade ago. If
you can suggest an online resource to help me use the right vocabulary
and better understand the fundamental concepts, I am of course
grateful.

Best regards,
David.


2009/6/20 Stavros Macrakis <macrakis at alum.mit.edu>: