Forum: Ruby truncate trouble

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Greg L. (Guest)
on 2007-03-19 21:41
(Received via mailing list)
This

('40.12'.to_f*100).to_i

results in this integer :

4011

This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.

Greg
Gavin K. (Guest)
on 2007-03-19 21:51
(Received via mailing list)
On Mar 19, 1:37 pm, "Greg L." <removed_email_address@domain.invalid> wrote:
> This
>
> ('40.12'.to_f*100).to_i
>
> results in this integer :
>
> 4011

What I find interesting is that the #inspect result of both f1 and f2
below are the same, when they internally are clearly not.

irb(main):001:0> f1 = '40.12'.to_f
=> 40.12
irb(main):002:0> f1*100
=> 4012.0
irb(main):003:0> (f1*100).to_i
=> 4011
irb(main):004:0> (f1*100).round.to_i
=> 4012

irb(main):005:0> 4012.0.to_i
=> 4012

irb(main):006:0> f2 = '40.12000000000001'.to_f
=> 40.12
irb(main):007:0> (f2*100).to_i
=> 4012
Brian C. (Guest)
on 2007-03-19 22:17
(Received via mailing list)
On Tue, Mar 20, 2007 at 04:40:08AM +0900, Greg L. wrote:
> This
>
> ('40.12'.to_f*100).to_i
>
> results in this integer :
>
> 4011
>
> This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?

Only standard numeric rounding error. The problem is that decimal 0.1
cannot
be represented exactly in binary floating point - it's a recurring
fraction,
like 1/3 in decimal - so you get only an approximation.

> Is there another floating type that avoids this problem? I;m not
> seeing anything in the docs that might have warned me.

irb(main):002:0> require 'bigdecimal'
=> true
irb(main):003:0> f = BigDecimal.new('40.12')
=> #<BigDecimal:b7cb7548,'0.4012E2',8(8)>
irb(main):004:0> (f*100).to_i
=> 4012

BigDecimal stores decimal numbers as decimal. It's not exactly well
documented though (I only came across it because Ruby on Rails uses it).

Another solution is to multiply all your numbers by 100 at source, i.e.
works in cents rather than dollars or whatever.

Regards,

Brian.
Robert K. (Guest)
on 2007-03-19 23:41
(Received via mailing list)
On 19.03.2007 20:37, Greg L. wrote:
> seeing anything in the docs that might have warned me.
No, that's a normal rounding error that comes with floats:

irb(main):001:0> f='40.12'.to_f
=> 40.12
irb(main):002:0> "%20.10f" % f
=> "       40.1200000000"
irb(main):003:0> "%40.30f" % f
=> "       40.119999999999997442046151263639"
irb(main):004:0> "%40.30f" % (f*100)
=> "     4011.999999999999545252649113535881"

If you look at the last line it becomes clear why you get 4011 as
result.

Kind regards

  robert
This topic is locked and can not be replied to.