I need acuracy!

how come <%= 19043.0 - 19032.6 %>

display 10.4000000000015

how can I avoid this and get a proper result?

Thanks

On 9/26/06, rosoft2001 [email protected] wrote:

how come <%= 19043.0 - 19032.6 %>

It’s a floating point error. See here:

http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

display 10.4000000000015

how can I avoid this and get a proper result?

Thanks

Use integer arithmetic instead. For example, use cents instead of
dollars by multiplying by 100 and converting to Integer. For other
numbers you will need to decide what precision you need. 4 decimal
places? Multiply by 10,000.

Cheers,
Dave

On 9/25/06, rosoft2001 [email protected] wrote:

how come <%= 19043.0 - 19032.6 %>

display 10.4000000000015

how can I avoid this and get a proper result?

Look up bigdecimal

http://www.ruby-doc.org/stdlib/

John W Higgins
[email protected]

On 9/26/06, John H. [email protected] wrote:

Look up bigdecimal

http://www.ruby-doc.org/stdlib/

You’ll also need edge rails if you plan on storing accurate values as
decimal/numeric in your db.

Isak

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs