[Jprogramming] Welford's method for standard deviations

[Viktor Cerovski]

Raul Miller-4 wrote:
> 
> On Thu, Sep 15, 2011 at 12:18 PM, Viktor Cerovski
> <viktor.cerovski at gmail.com> wrote:
>> You(r subtracted) mean (is) this:
>>
>>   stddev _1e12+1e12+1e6?@$0
>> 0.288461
> 
> If I am allowed to know the mean of 1e12+1e6?@$0 -- if I am allowed to
> treat the data as non-random -- then, yes.
> 
> But that was not really my point.  The point, I thought, was to come
> up with an implementation of standard deviation that is stable,
> numerically, when the mean is much larger than the deviation.
> 
Well, that's not the point of the example.  Let's see:

   require'stats'
   x =. 1e6 ?@$ 0
   stddev 1e12 + x
0.573404        NB. problem
   stddev@:(- mean) 1e12 + x
0.288938         NB. remedy

So far so good.  But it's *not* because of the large mean. Look:

   stddev 1e12 + x - 0.5
0.288938

Same x translated by tiny (compared to 1e12) 0.5, 
still large mean and the problem is gone. 

So, what you suggest is the correct remedy, but for the wrong
reasons.


> And, of course, Welford's approach does achieve that.  But Welford's
> algorithm assumes a scalar computing architecture.  Meanwhile, it
> seems to me that subtracting the computed mean achieves approximately
> the same thing that Welford's approach achieves, but modularized to
> eliminate the "scalar architecture" aspect.
> 
> That said, if there are generic cases where Welford's approach behaves
> better than subtracting the computed mean from the data set, I would
> be interested in hearing about them.  (But I doubt they exist, except
> in hand crafted special cases with no general utility, because
> Welford's algorithm is doing essentially the same thing, with a
> running average.)
> 
> -- 
> Raul
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
> 
> 
So before getting into all sorts of general questions & quests, 
the first thing to consider here is whether stddev@:(- mean) is 
more accurate than stddev in a sort of generic, standard kind of 
calculations.  I think it is not.

-- 
View this message in context: http://old.nabble.com/Welford%27s-method-for-standard-deviations-tp32458412s24193p32479345.html
Sent from the J Programming mailing list archive at Nabble.com.