Skip to content

protential rounding error concern

3 messages · Jie, R. Michael Weylandt, Petr Savicky

#
On Tue, Jul 31, 2012 at 2:45 PM, Jie <jimmycloud at gmail.com> wrote:
This would be a canonical reference:

http://www.validlab.com/goldberg/paper.pdf.

Hope this helps,
M
#
On Tue, Jul 31, 2012 at 03:45:14PM -0400, Jie wrote:
Hi.

Numbers of the order 10^(-100) are still relatively far from the lower
limit and are represented with full precision, which is 53 bits.

The smallest number representable in full precision is 2^-1022 = 2.225074e-308
and the smallest representable number (in limited precision) is 2^-1074 =
4.940656e-324.

See 

  http://en.wikipedia.org/wiki/Double_precision_floating-point_format

for more detail.

Hope this helps.

Petr Savicky.