I'm sure this has been hashed out somewhere on the list before, but my searches have turned up fruitless. I had a strange issue come up while diagnosing a unit test failure. Notice the first dialog compared with the rest of this IRB session. What causes -3930.0 to be converted to a Fixnum as -3929 only after it's been multiplied by 100 or 100.0? Even weirder is that it seems to only be *this* number. Can I workaround this? >> (-39.30 * 100).to_i => -3929 >> (-39.30 * 100.0).to_i => -3929 >> -3930.0.to_i => -3930 >> (-39.31 * 100).to_i => -3931 >> (-9999.99 * 100).to_i => -999999 $ ruby -v ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-linux] Thanks. -Drew

on 2007-01-19 21:10

on 2007-01-19 21:22

Drew Raines schrieb: > => -3929 > > Thanks. > > -Drew irb(main):001:0> 3930-39.30*100 => 4.54747350886464e-013 Rounding errors are typical for floating point numbers. In this case the result of -39.30*100 ist a little bit smaller than 3930, so to_i works correct.

on 2007-01-19 22:31

```
Drew Raines schrieb:
> Can I workaround this?
If you always expect a result for which "x * 100" is a number without a
fractional part, you can use "(-39.30 * 100).round" instead.
Wolfgang Nádasi-Donner
```

on 2007-01-19 22:51

Wolfgang NÃ¡dasi-Donner wrote: > irb(main):001:0> 3930-39.30*100 > => 4.54747350886464e-013 > > Rounding errors are typical for floating point numbers. In this case > the result of -39.30*100 ist a little bit smaller than 3930, so to_i > works correct. When I think of rounding error, I think of lost precision due to rounding too early in a series of floating point calculations. When multiplying a non-repeating Float point by 100, shouldn't the decimal move over two places without introducing extra precision? And how can 39.30*100 really be 3930.000000000000454747350886464..., but 39.31*100 be 3931.0? BigDecimal seems to be unaffected, so I guess I'll use that as a workaround: >> (39.30*100).to_i => 3929 >> (39.30*100).to_d.to_i => 3930 Thanks for your response. -Drew

on 2007-01-19 22:55

On Sat, 20 Jan 2007, Drew Raines wrote: > rounding too early in a series of floating point calculations. When > multiplying a non-repeating Float point by 100, shouldn't the decimal > move over two places without introducing extra precision? no. check out http://en.wikipedia.org/wiki/Floating_point#Comput... -a

on 2007-01-20 01:34

On Jan 19, 2007, at 4:50 PM, Drew Raines wrote: > And how can 39.30*100 really be 3930.000000000000454747350886464..., > but 39.31*100 be 3931.0? Because 39.30 is a base 10 representation of a number and there is no exact representation of that number in base 2. The Ruby parser has to do a conversion to base 2 for 39.30 long before the multiplication occurs so you aren't really multiplying 39.30*100 but instead you are multiplying 39.2999999999999971578290569595992565155029296875 * 100 to get 3929.99999999999971578290569595992565155029296875 which when truncated to an integer is 3929 Gary Wright

on 2007-01-20 06:00

From: Drew Raines [mailto:aaraines@gmail.com] : #-------------------------------------- # BigDecimal seems to be unaffected, so I guess I'll use that as a # workaround: # # >> (39.30*100).to_i # => 3929 #-------------------------------------- mixing integers and float is like mixing water w gasoline. expect disasters in all directions. irb(main):018:0> 39.30*100 == 3930 => false irb(main):019:0> 39.30*100 == 3939 => false ruby supports integers and floating point numbers (wc in turn are supported by the hw architecture). imho, there's a big gap/hole for the representation of plain decimals (usurped by floating point in this case). #-------------------------------------- # >> (39.30*100).to_d.to_i # => 3930 #-------------------------------------- indeed. 99% of the my time, i find no use for floats (maybe i'm scared of surprises). i'd suggest a global $DECIMAL=1 to instruct ruby to bigdecimal mode so that >> 39.30*100 == 3930 => true >> (39.30*100).to_i # no need for to_d there, 39.30 is decimal, right? ;) => 3930 i really do not care if it would be slower. i chosed ruby anyway. kind regards -botp

on 2007-01-20 08:05

Drew Raines wrote: > rounding too early in a series of floating point calculations. When > multiplying a non-repeating Float point by 100, shouldn't the decimal > move over two places without introducing extra precision? > > And how can 39.30*100 really be 3930.000000000000454747350886464..., Almost all programming languages use binary floating point, so 0.30 can't be represented exactly. A very few languages use binary-coded decimal. Examples: Decimal BASIC and Business BASIC (I think). The BASIC that came with the old 8-bit Atari used binary-coded decimal, so it would have had no problem with 3930-39.30*100. Since computers of today are so much faster than 8-bit computers, one would think that they could afford to incur the speed penalty associated with binary-coded decimal.

on 2007-02-08 19:50

Wolfgang NÃ¡dasi-Donner wrote: > Drew Raines schrieb: >> Can I workaround this? > > If you always expect a result for which "x * 100" is a number without > a fractional part, you can use "(-39.30 * 100).round" instead. Hal Fulton[1] gave me a good idea: class Float FUDGE = 1e-3 def ==(x) (self-x).abs < FUDGE end end This is better for my tests anyway. assert_equal now works on its own. -Drew Footnotes: [1] http://lnk.nu/amazon.com/dng

on 2007-02-08 19:55

On Feb 8, 2007, at 12:50 PM, Drew Raines wrote: > class Float > FUDGE = 1e-3 > def ==(x) > (self-x).abs < FUDGE > end > end > > This is better for my tests anyway. assert_equal now works on its > own. Test::Unit already includes an assertion just for this. It's called assert_in_delta(). James Edward Gray II