Fastest way to append file with matrix data

I'm using Matlab to measure data in a rateControl timed loop at a certain frequency, ~ 20 Hz. In my app, I want to plot this data during the measurement, so it is important to do this efficiently (15 channels, 20 Hz, running for a day creates a 10^6 x 15 sized matrix). I'm trying to use animatedline such that I do not have to concatenate and plot an increasingly lengthy matrix. In my timed loop I would overwrite a 1 x 15 matrix containing measured data, use addpoints for plotting, and append the measured data to my file. Appending a file regularly is preferred over saving a large matrix at the end anyway, should the measurement crash for some reason.

My question is what the best/fastest way is of saving/appending a file. The support page for dlmwrite recommends writematrix. However, using the below example I find that dlmwrite is faster.

num_writes = 1000;              
time = zeros(1,num_writes);     % track in case appending long files takes longer
M = randn(1,15);                % random start data
freq = 20;                      % 'measurement' frequency

for mode = ["dlmwrite","writematrix"]
    if strcmp(mode,'dlmwrite')
        dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
        for i = 1:num_writes    % 'measurement' loop
            tic
            N = randn(size(M));
            dlmwrite('test.csv',N,'-append');
            time(i) = toc;
        end
        
    elseif strcmp(mode,'writematrix')
        writematrix(M,'test.csv')
        for i = 1:num_writes
            tic
            N = randn(size(M));
            writematrix(N,'test.csv','WriteMode','append')
            time(i) = toc;
        end
    end
    
    tperoperation = sum(time)/num_writes;
    disp(mode)
    disp(['total elapsed time is ',num2str(sum(time)),' s'])
    disp(['or ',num2str(tperoperation),' s per operation'])
    disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end

Is it possible to improve on the write speed, e.g. using a different file format? I found mex_WriteMatrix on the file exchange, however this seems to improve speed only when appending a large amount of data, which is not the case for me.

If my method of a timed loop with animated lines is flawed in general, I'm also open to suggestions.

[dpb]

Fastest i/o will be to use fwrite to a stream (unformatted) file.

[Koen]

Thank you, I will look into it.

[Walter Roberson]

Yes, binary will always be faster.

Answer 1

[tetra]

I was forwarded to this thread and implemented user dpb's suggestion. fwrite is indeed much faster - roughly 100 times. For others looking for an implementation of that answer I have adapted the code provided by OP.

num_writes = 1000;              
time = zeros(1,num_writes);     % track in case appending long files takes longer
M = randn(1,15);                % random start data
freq = 20;                      % 'measurement' frequency
for mode = ["dlmwrite","writematrix","binary"]
    if strcmp(mode,'dlmwrite')
        dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
        for i = 1:num_writes    % 'measurement' loop
            tic
            N = randn(size(M));
            dlmwrite('test.csv',N,'-append');
            time(i) = toc;
        end
        
    elseif strcmp(mode,'writematrix')
        writematrix(M,'test.csv')
        for i = 1:num_writes
            tic
            N = randn(size(M));
            writematrix(N,'test.csv','WriteMode','append')
            time(i) = toc;
        end

    elseif strcmp(mode,'binary')
        fidbin = fopen('test.bin','a');         % 'a' = append data
        for i = 1:num_writes
            tic
            N = randn(size(M));
            fwrite(fidbin,N,'double');
            time(i) = toc;
        end
        fclose(fidbin);
    end
    
    tperoperation = sum(time)/num_writes;
    disp(mode)
    disp(['total elapsed time is ',num2str(sum(time)),' s'])
    disp(['or ',num2str(tperoperation),' s per operation'])
    disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end

dlmwrite

total elapsed time is 0.27458 s

or 0.00027458 s per operation

this is 0.54916% of loop time (20 Hz)

writematrix

total elapsed time is 1.8607 s

or 0.0018607 s per operation

this is 3.7214% of loop time (20 Hz)

binary

total elapsed time is 0.007412 s

or 7.412e-06 s per operation

this is 0.014824% of loop time (20 Hz)

On Matlab desktop, dlmwrite is slower than when executed as live script: my results are 8.04, 10.55 and 0.02 seconds respectively. You can open the binary file using

bincheck=fopen('test.bin','r');                     % 'r' = read data
bindata=fread(bincheck,[lenght(M),inf],'double')';  % transposing will result in 'correct' size
fclose(bincheck);