Forum: Ruby truncate trouble

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ea52304f84449ae1a3feabc6d3cd04f9?d=identicon&s=25 Greg Lorriman (Guest)
on 2007-03-19 20:41
(Received via mailing list)
This

('40.12'.to_f*100).to_i

results in this integer :

4011

This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?
Is there another floating type that avoids this problem? I;m not
seeing anything in the docs that might have warned me.

Greg
852a62a28f1de229dc861ce903b07a60?d=identicon&s=25 Gavin Kistner (phrogz)
on 2007-03-19 20:51
(Received via mailing list)
On Mar 19, 1:37 pm, "Greg Lorriman" <t...@lorriman.com> wrote:
> This
>
> ('40.12'.to_f*100).to_i
>
> results in this integer :
>
> 4011

What I find interesting is that the #inspect result of both f1 and f2
below are the same, when they internally are clearly not.

irb(main):001:0> f1 = '40.12'.to_f
=> 40.12
irb(main):002:0> f1*100
=> 4012.0
irb(main):003:0> (f1*100).to_i
=> 4011
irb(main):004:0> (f1*100).round.to_i
=> 4012

irb(main):005:0> 4012.0.to_i
=> 4012

irb(main):006:0> f2 = '40.12000000000001'.to_f
=> 40.12
irb(main):007:0> (f2*100).to_i
=> 4012
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (Guest)
on 2007-03-19 21:17
(Received via mailing list)
On Tue, Mar 20, 2007 at 04:40:08AM +0900, Greg Lorriman wrote:
> This
>
> ('40.12'.to_f*100).to_i
>
> results in this integer :
>
> 4011
>
> This is a bit of surprise. I'm using 1.8.4 on windows. Is this a bug?

Only standard numeric rounding error. The problem is that decimal 0.1
cannot
be represented exactly in binary floating point - it's a recurring
fraction,
like 1/3 in decimal - so you get only an approximation.

> Is there another floating type that avoids this problem? I;m not
> seeing anything in the docs that might have warned me.

irb(main):002:0> require 'bigdecimal'
=> true
irb(main):003:0> f = BigDecimal.new('40.12')
=> #<BigDecimal:b7cb7548,'0.4012E2',8(8)>
irb(main):004:0> (f*100).to_i
=> 4012

BigDecimal stores decimal numbers as decimal. It's not exactly well
documented though (I only came across it because Ruby on Rails uses it).

Another solution is to multiply all your numbers by 100 at source, i.e.
works in cents rather than dollars or whatever.

Regards,

Brian.
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2007-03-19 22:41
(Received via mailing list)
On 19.03.2007 20:37, Greg Lorriman wrote:
> seeing anything in the docs that might have warned me.
No, that's a normal rounding error that comes with floats:

irb(main):001:0> f='40.12'.to_f
=> 40.12
irb(main):002:0> "%20.10f" % f
=> "       40.1200000000"
irb(main):003:0> "%40.30f" % f
=> "       40.119999999999997442046151263639"
irb(main):004:0> "%40.30f" % (f*100)
=> "     4011.999999999999545252649113535881"

If you look at the last line it becomes clear why you get 4011 as
result.

Kind regards

  robert
This topic is locked and can not be replied to.