Skip to content
Prev 199179 / 398503 Next

unexpected results in comparison (x == y)

Peter Tillmann wrote:
This looks like a side effect of roundoff error due to the finite precision
of numbers stored by a computer.  There was a large discussion of this issue
on the list about two weeks ago, you could read the posts at: 

  http://old.nabble.com/Rounding-error-in-seq%28...%29-ts25686630.html

They may provide some insight.  The punchline is that:

  0.2 = 2 * (1/10)

And 1/10 is not representable as a form of:

  2^n

For some integer n. Therefore the number 0.2 cannot be represented exactly
in floating point arithmetic and so inconsistencies will occur-- as you just
observed.  It's one of those realities of computer science that always
catches us non-computer scientists by surprise.

Hope this helps!

-Charlie

-----
Charlie Sharpsteen
Undergraduate
Environmental Resources Engineering
Humboldt State University