std for a growing data range

Hello :),

i am looking for an effective way to calculate following problem:

for i=1:1000
    tmp=data(1:i,:);
    rsStd(i)=std(tmp(:));
end

"data" is a matrix with k rows and j columns. What i dont want is to calculate the std for every column. What i would need is the std for a growing data range. By each loop step the data range for the calculation of the std grows as you can see in the code snippet.

Does anyone know probably a faster solution and could give me a hint? I am thinking something with vectorization but currently i have no idea. Many thanks!

Answer 1

[Roger Stafford]

 n = (1:size(data,1))'*size(data,2);
 sq = cumsum(sum(data.^2,2));
 sm = cumsum(sum(data,2));
 rsStd = sqrt((sq-sm.^2./n)./(n-1)); % <-- Corrected

(I'm assuming you want the unbiased version of standard deviation.)

Answer 2

[Jos (10584)]

Here is the completely vectored code for the running sample standard deviation across the rows of a matrix X:

X = ceil(10*rand(13,4)) % example data

N = cumsum(ones(size(X)),1)
S1 = cumsum(X,1) 
S2 = cumsum(X.^2,1) 
R = sqrt((N.*S2 - S1.^2) ./ (N.*(N-1)))
% this can be a one-liner if you insist ..

% check
k = 4 ; [R(k,:) ; std(X(1:k,:))] % the same

Use it to your advantage!

Answer 3

[mick strife]

Thx both! :)