Floating point conversion error?

(2241.18 * 100).to_i
=> 224117

Anyone know what is up with that?

Brandon H. wrote:

(2241.18 * 100).to_i

=> 224117

Anyone know what is up with that?

Floating point arithmetic isn’t entirely accurate.

“%.30f” % (2241.18 * 100)
=> “224117.999999999970896169543266296387”

And since 224117.999999999970896169543266296387.to_i is 224117, that’s
what
you get.

HTH,
Sebastian

Sebastian H. wrote:

Brandon H. wrote:

(2241.18 * 100).to_i

=> 224117

Anyone know what is up with that?

Floating point arithmetic isn’t entirely accurate.

“%.30f” % (2241.18 * 100)
=> “224117.999999999970896169543266296387”

And since 224117.999999999970896169543266296387.to_i is 224117, that’s
what
you get.

HTH,
Sebastian

Thanks… found this after poking around… guess I should have spent
more time looking.

http://www.ruby-forum.com/topic/132986#new

Hello!

On Jan 23, 2008 4:25 PM, Brandon H. [email protected] wrote:

(2241.18 * 100).to_i
=> 224117

Anyone know what is up with that?

Floating point numbers are never 100% accurate. See
http://en.wikipedia.org/wiki/Floating_point for a detailed
explanation.

On Jan 23, 2008, at 10:25 AM, Brandon H. wrote:

(2241.18 * 100).to_i
=> 224117

Anyone know what is up with that?

What Every Computer Scientist Should Know About Floating-Point
Arithmetic

http://docs.sun.com/source/806-3568/ncg_goldberg.html

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs