[Jprogramming] difficulty with extended precision arithmetic

bill lam bbill.lam at gmail.com
Tue Jan 19 17:36:43 HKT 2010


mar, 19 Jan 2010, R.E. Boss skribis:
>    T=:10000000000000306999999999997669x
> 
>    9!:11 ]20	NB. print precision
> 
>    T^0.5
> 3162277660168428
> 

Isn't the result of T^0.5 a ieee floating point?  In that case, increase its
print precision above 16 does not increase accuracy.

-- 
regards,
====================================================
GPG key 1024D/4434BAB3 2008-08-24
gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3


More information about the Programming mailing list