# Numeric precision problem

**URL:** <https://www.ruby-forum.com/t/numeric-precision-problem/60151>\
**Category:** Ruby\
**Created:** [May 10, 2006, 6:36pm UTC](https://www.ruby-forum.com/t/numeric-precision-problem/60151 "2006-05-10T18:36:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![doug3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b3f665/32.png) [@doug3](https://www.ruby-forum.com/u/doug3)\
**Post date:** [May 10, 2006, 6:36pm UTC](https://www.ruby-forum.com/t/numeric-precision-problem/60151/1 "2006-05-10T18:36:41Z")

</div>

Greetings all.

I’m having a little problem with numeric precision and being a NOOB to  
Ruby, I’m not quite sure where I am going wrong.

When I perform the following calculation:

n = ((9703.0 \* 16346204482307500.0) + 1).to\_f  
d = (65537.0).to\_f

res = (n / d).to\_f

Ruby is returning a value of 249,419,480,327.0

However, the more precise number I am expecting is 249,419,480,328.784

How do I do this to get the more precise value?

Thanks.

Doug

---

<div class="post-metadata">

**Author:** ![doug3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b3f665/32.png) [@doug3](https://www.ruby-forum.com/u/doug3)\
**Post date:** [May 10, 2006, 6:58pm UTC](https://www.ruby-forum.com/t/numeric-precision-problem/60151/2 "2006-05-10T18:58:24Z")

</div>

2006/5/10, doug meharry [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> res = (n / d).to\_f
> 
> Ruby is returning a value of 249,419,480,327.0
> 
> However, the more precise number I am expecting is 249,419,480,328.784
> 
> How do I do this to get the more precise value?

You can use rational:

> > require ‘rational’  
> > =\> true  
> > n = (9703 \* 16346204482307500) + 1  
> > =\> 158607222091829672501  
> > d = 65537  
> > =\> 65537  
> > res = (n / d)  
> > =\> 2420117217630188  
> > res.to\_f  
> > =\> 2.42011721763019e+15

Floating point arithmetic always has some numeric issues.

robert

---

<div class="post-metadata">

**Author:** ![doug3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b3f665/32.png) [@doug3](https://www.ruby-forum.com/u/doug3)\
**Post date:** [May 10, 2006, 7:09pm UTC](https://www.ruby-forum.com/t/numeric-precision-problem/60151/3 "2006-05-10T19:09:37Z")

</div>

doug meharry wrote:

> Greetings all.
> 
> I’m having a little problem with numeric precision and being a NOOB to  
> Ruby, I’m not quite sure where I am going wrong.
> 
> When I perform the following calculation:
> 
> n = ((9703.0 \* 16346204482307500.0) + 1).to\_f  
> d = (65537.0).to\_f
> 
> res = (n / d).to\_f
> 
> Ruby is returning a value of 249,419,480,327.0
> 
> However, the more precise number I am expecting is 249,419,480,328.784
> 
> How do I do this to get the more precise value?
> 
> Thanks.
> 
> Doug

Oops! Never mind. I some of the numbers in my preceeding message were  
slightly off. This produced the expected results.

n = ((9704.0 \* 1684654692600.0) + 1).to\_f  
d = (65537.0).to\_f

res = (n / d).to\_f

Ruby is returning a value of 249,445,185,727.0 which is what I also  
think it should be.