[Jprogramming] dyad ": rounding error

[Joey K Tuttle]

Ah, the perils of floating point numbers....

Even though you say you would never use rational
numbers in calculations, most accountants do.
For example, the calculators you mentioned in
your points often do fixed calculations with
the number of decimals set as a fixed option.

Similarly, many accounting programs store amounts
of money in cents. e.g. in phone call records, the
rates (cents/minute) are usually stored as 10000
times the dollar amount - and so on. Formatting
the final numbers is only a part of the problem...

- joey


At 1:03 PM +0800 3/20/07, bill lam wrote:
>I think session display are also affected by rounding in dyad ": eg
>
>    9!:11[2
>
>    0.295
>0.29
>
>There are many similar numbers so that simple dyad ": alone is 
>inappropriate for formating numbers in reports.
>
>    r1=. <.@(+&0.5)&.(%&0.01)  NB. tolerant rounding
>
>   (0j2&":"0 ; 0j2&":@r1"0) a=. 1000%~205 + 10*i.20
>+----+----+
>|0.20|0.21|
>|0.22|0.22|
>|0.23|0.23|
>|0.23|0.24|
>|0.25|0.25|
>|0.26|0.26|
>|0.27|0.27|
>|0.28|0.28|
>|0.28|0.29|
>|0.29|0.30|
>|0.30|0.31|
>|0.32|0.32|
>|0.33|0.33|
>|0.34|0.34|
>|0.34|0.35|
>|0.35|0.36|
>|0.36|0.37|
>|0.38|0.38|
>|0.39|0.39|
>|0.40|0.40|
>+----+----+
>
>Roger Hui wrote:
>>The difference is that the dyad ": depends on the
>>C library routines whereas 8!:x do the programming
>>themselves and take more care.  I'll look into it.
>>It is likely that a fix will make ": slower.
>>
>>Thanks for the report.
>>
>>
>>
>>----- Original Message -----
>>From: bill lam <bbill.lam at gmail.com>
>>Date: Sunday, March 18, 2007 11:14 pm
>>Subject: [Jprogramming] dyad ": rounding error
>>
>>>On both J504 and J601, dyad ": sometimes does not round properly 
>>>on some numbers, eg.
>>>    0j2 ":  0.295
>>>0.29
>>>    0j3 ":  0.2995
>>>0.299
>>>
>>>incidentally 8!:x family of J601 does not have this problem,
>>>    '0.2' (8!:0) 0.295
>>>+----+
>>>|0.30|
>>>+----+
>>>    '0.2' (8!:1) 0.295
>>>+----+
>>>|0.30|
>>>+----+
>>>    '0.2' (8!:2) 0.295
>>>0.30
>>>    '0.3' (8!:2) 0.2995
>>>0.300