# Forum: Ruby Numeric precision problem

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
on 2006-05-10 20:36
```Greetings all.

I'm having a little problem with numeric precision and being a NOOB to
Ruby, I'm not quite sure where I am going wrong.

When I perform the following calculation:

n = ((9703.0 * 16346204482307500.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

Thanks.

Doug```
on 2006-05-10 20:58
```2006/5/10, doug meharry <removed_email_address@domain.invalid>:
> res = (n / d).to_f
>
> Ruby is returning a value of 249,419,480,327.0
>
> However, the more precise number I am expecting is 249,419,480,328.784
>
> How do I do this to get the more precise value?

You can use rational:

>> require 'rational'
=> true
>> n = (9703 * 16346204482307500) + 1
=> 158607222091829672501
>> d = 65537
=> 65537
>> res = (n / d)
=> 2420117217630188
>> res.to_f
=> 2.42011721763019e+15

Floating point arithmetic always has some numeric issues.

robert```
on 2006-05-10 21:09
```doug meharry wrote:
> Greetings all.
>
> I'm having a little problem with numeric precision and being a NOOB to
> Ruby, I'm not quite sure where I am going wrong.
>
> When I perform the following calculation:
>
> n = ((9703.0 * 16346204482307500.0) + 1).to_f
> d = (65537.0).to_f
>
> res = (n / d).to_f
>
> Ruby is returning a value of 249,419,480,327.0
>
> However, the more precise number I am expecting is 249,419,480,328.784
>
> How do I do this to get the more precise value?
>
> Thanks.
>
> Doug

Oops!  Never mind.  I some of the numbers in my preceeding message were
slightly off.  This produced the expected results.

n = ((9704.0 * 1684654692600.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,445,185,727.0 which is what I also
think it should be.```
This topic is locked and can not be replied to.