Float calculation error

Rails
1

Hi

I have the following values

pActualCost = 33.00
pPaymentCost = 29.99

So this calculation should leave me with 0.01
pPaymentDifference = pActualCost - pPaymentCost

however when doing this in rails, it returns 0.00999999999999801

has anyone got any suggestions to whats going wrong and how I can
correct this

Thanks
Scott

2

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

ss schrieb:

3

Pete wrote:

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

ss schrieb:

Thanks

for anyone who is having a simular problem I used this to round to 2dp

pPaymentDifference = (pPaymentDifference * 102).round.to_f / 102

4

ss wrote:

for anyone who is having a simular problem I used this to round to 2dp

pPaymentDifference = (pPaymentDifference * 102).round.to_f / 102

You could make it generic so that you don’t repeat yourself everywhere
you need to round to decimal places.

I have this in my app:

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

5

Richard L. wrote:

Thanks

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

Rails already provides a helper, number_to_currency, described here:

regards

Justin

6

On 6/23/06, Pete [email protected] wrote:

2 ways:

  1. store all your money in cents and do integer arithmetic (not prone to
    rounding errors)
  2. use rounding

Or better yet, use BigDecimal from Ruby’s standard library.

Isak

7

Justin F. wrote:

Richard L. wrote:

Thanks

class Float
alias :round_off :round
def round( d=0.0 )
rounded = (self * (10.0 ** d)).round_off.to_f / (10.0 ** d)
rounded = rounded.to_i if d == 0.0
rounded
end
end

Rails already provides a helper, number_to_currency, described here:

ActionView::Helpers::NumberHelper

regards

Justin

Also, facet/round_at is super useful.

gem install facets
require ‘facet/round_at’
3.12345.round_at(2) #=> 3.12