Float calculation error

Hi

I have the following values

pActualCost = 33.00
pPaymentCost = 29.99

So this calculation should leave me with 0.01
pPaymentDifference = pActualCost - pPaymentCost

however when doing this in rails, it returns 0.00999999999999801

has anyone got any suggestions to whats going wrong and how I can
correct this

Thanks
Scott

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

ss schrieb:

Pete wrote:

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

ss schrieb:

Thanks

for anyone who is having a simular problem I used this to round to 2dp

pPaymentDifference = (pPaymentDifference * 102).round.to_f / 102

ss wrote:

for anyone who is having a simular problem I used this to round to 2dp

pPaymentDifference = (pPaymentDifference * 102).round.to_f / 102

You could make it generic so that you don’t repeat yourself everywhere
you need to round to decimal places.

I have this in my app:

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

Richard L. wrote:

Thanks

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

Rails already provides a helper, number_to_currency, described here:

http://api.rubyonrails.org/classes/ActionView/Helpers/NumberHelper.html#M000449

regards

Justin

On 6/23/06, Pete [email protected] wrote:

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

Or better yet, use BigDecimal from Ruby’s standard library.

Isak

Justin F. wrote:

Richard L. wrote:

Thanks

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

Rails already provides a helper, number_to_currency, described here:

http://api.rubyonrails.org/classes/ActionView/Helpers/NumberHelper.html#M000449

regards

Justin

Also, facet/round_at is super useful.

gem install facets
require ‘facet/round_at’
3.12345.round_at(2) #=> 3.12

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs