Forum: Ruby Numeric precision problem

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
E700f28e33f22aa356f21833cfd7cba4?d=identicon&s=25 Doug Meharry (doug3)
on 2006-05-10 18:36
Greetings all.

I'm having a little problem with numeric precision and being a NOOB to
Ruby, I'm not quite sure where I am going wrong.

When I perform the following calculation:

n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

Thanks.

Doug
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2006-05-10 18:58
(Received via mailing list)
2006/5/10, doug meharry <doug@dmeharry.com>:
> res = (n / d).to_f
>
> Ruby is returning a value of 249,419,480,327.0
>
> However, the more precise number I am expecting is 249,419,480,328.784
>
> How do I do this to get the more precise value?

You can use rational:

>> require 'rational'
=> true
>> n = (9703 * 16346204482307500) + 1
=> 158607222091829672501
>> d = 65537
=> 65537
>> res = (n / d)
=> 2420117217630188
>> res.to_f
=> 2.42011721763019e+15

Floating point arithmetic always has some numeric issues.

robert
E700f28e33f22aa356f21833cfd7cba4?d=identicon&s=25 Doug Meharry (doug3)
on 2006-05-10 19:09
doug meharry wrote:
> Greetings all.
>
> I'm having a little problem with numeric precision and being a NOOB to
> Ruby, I'm not quite sure where I am going wrong.
>
> When I perform the following calculation:
>
> n = ((9703.0 * 16346204482307500.0) + 1).to_f
> d = (65537.0).to_f
>
> res = (n / d).to_f
>
> Ruby is returning a value of 249,419,480,327.0
>
> However, the more precise number I am expecting is 249,419,480,328.784
>
> How do I do this to get the more precise value?
>
> Thanks.
>
> Doug

Oops!  Never mind.  I some of the numbers in my preceeding message were
slightly off.  This produced the expected results.

n = ((9704.0 * 1684654692600.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,445,185,727.0 which is what I also
think it should be.
This topic is locked and can not be replied to.