Numeric precision problem

Ruby
1

Greetings all.

I’m having a little problem with numeric precision and being a NOOB to
Ruby, I’m not quite sure where I am going wrong.

When I perform the following calculation:

n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

Thanks.

Doug

2

2006/5/10, doug meharry [email protected]:

res = (n / d).to_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

You can use rational:

require ‘rational’
=> true
n = (9703 * 16346204482307500) + 1
=> 158607222091829672501
d = 65537
=> 65537
res = (n / d)
=> 2420117217630188
res.to_f
=> 2.42011721763019e+15

Floating point arithmetic always has some numeric issues.

robert

3

doug meharry wrote:

Greetings all.

I’m having a little problem with numeric precision and being a NOOB to
Ruby, I’m not quite sure where I am going wrong.

When I perform the following calculation:

n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

Thanks.

Doug

Oops! Never mind. I some of the numbers in my preceeding message were
slightly off. This produced the expected results.

n = ((9704.0 * 1684654692600.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,445,185,727.0 which is what I also
think it should be.