I'm trying to understand the argument to BigDecimal#sqrt. I figured it was for precision and it does seem to affect it, but I can't understand the exactly relationship. Can anyone tell me what the argument controls and how? Thanks. James Edward Gray II

on 2005-12-22 23:06

on 2005-12-23 05:13

On Dec 22, 2005, at 4:04 PM, James Edward Gray II wrote: > I'm trying to understand the argument to BigDecimal#sqrt. I > figured it was for precision and it does seem to affect it, but I > can't understand the exactly relationship. Can anyone tell me what > the argument controls and how? Let me try asking me question a different way... Is this the correct way to get a printable sqrt() with 16 digits precision? >> sprintf "%.16f", BigDecimal.new("66543").sqrt(16) => "257.9592991151890260" James Edward Gray II

on 2005-12-23 05:13

On Dec 22, 2005, at 4:04 PM, James Edward Gray II wrote: > I'm trying to understand the argument to BigDecimal#sqrt. I > figured it was for precision and it does seem to affect it, but I > can't understand the exactly relationship. Can anyone tell me what > the argument controls and how? Let me try asking my question a different way... Is this the correct way to get a printable square root with N digits precision (N = 16 in this example)? >> sprintf "%.16f", BigDecimal.new("66543").sqrt(16) => "257.9592991151890260" James Edward Gray II

on 2005-12-23 13:27

yeah, it looks right. interesting format variation: minfieldwidth = 22 precision = 16 sprintf "%*.*f", minfieldwidth, precision, BigDecimal.new("66543").sqrt(precision) => " 257.9592991151898786"

on 2005-12-23 18:08

On 12/22/05, James Edward Gray II <james@grayproductions.net> wrote: > I'm trying to understand the argument to BigDecimal#sqrt. I figured > it was for precision and it does seem to affect it, but I can't > understand the exactly relationship. Can anyone tell me what the > argument controls and how? This is the C source, if you haven't looked at it yet. BigDecimal_sqrt(VALUE self, VALUE nFig) { ENTER(5); Real *c, *a; S_INT mx, n; GUARD_OBJ(a,GetVpValue(self,1)); mx = a->Prec *(VpBaseFig() + 1); n = GetPositiveInt(nFig) + VpDblFig() + 1; if(mx <= n) mx = n; GUARD_OBJ(c,VpCreateRbObject(mx, "0")); VpSqrt(c, a); return ToValue(c); } I don't fully understand what's going on with it, because my C is pretty weak, but hopefully this will provide some insight from someone.

on 2005-12-28 05:29

James Edward Gray II wrote: > Let me try asking me question a different way... Is this the correct > way to get a printable sqrt() with 16 digits precision? > > >> sprintf "%.16f", BigDecimal.new("66543").sqrt(16) > => "257.9592991151890260" Yes. I've written some documentation for BigDecimal which will hopefully be checked in to Ruby some time. mathew

on 2005-12-28 14:43

On Dec 27, 2005, at 10:27 PM, mathew wrote: > James Edward Gray II wrote: >> Let me try asking me question a different way... Is this the >> correct way to get a printable sqrt() with 16 digits precision? >> >> sprintf "%.16f", BigDecimal.new("66543").sqrt(16) >> => "257.9592991151890260" > > Yes. > > I've written some documentation for BigDecimal which will hopefully > be checked in to Ruby some time. Is it available online anywhere before then or could I trouble you to send it to me privately? James Edward Gray II