load large data set from text file

Hi, I need to load a data file. The data is in n x 3 format.

It is not a problem to load a file if all data available. For example:

3.519549e+002 1.972350e+001 -2.485200e+001

3.523932e+002 1.972350e+001 -2.503600e+001

3.528315e+002 1.972350e+001 -2.520600e+001

However, some files contain bad data points, for example:

1.003707e+002 5.697900e+000 *

1.008090e+002 5.697900e+000 *

1.016856e+002 5.697900e+000 *

3.528315e+002 1.972350e+001 -2.520600e+001

1.012473e+002 5.697900e+000 *

3.523932e+002 1.972350e+001 -2.503600e+001

3.528315e+002 1.972350e+001 -2.520600e+001

What is the quickest way to load the file? I use loop to load line by line and remove all lines with bad data points, however, it takes extremely long time. (my current code below)

data = [];

tline = fgetl(fid);

while tline~=-1

    if isempty(findstr(tline,'***'))

       data = [data;str2num(tline)];

    end

tline = fgetl(fid);

end

Any suggestion? Many thanks.

Answer 1

[Chad Greene]

June--Use textscan to load your data, like

fid = fopen('mydata.txt'); 
DATA = textscan(fid,'%f %f %f'); 
fclose(fid)

Then your columns are given by

col1 = DATA{1}; 
col2 = DATA{2}; 
col3 = DATA{3};

Check out the textscan options like 'TreatAsEmpty'.

Answer 2

[Geoff Hayes]

If you have an idea on how many valid lines are in the text file, then you could consider pre-allocating memory to the data matrix in an attempt to slow or eliminate the re-sizing of this matrix on each iteration of the loop. It may be that the concatenating of the matrix at each iteration (with valid data) is causing a degradation in performance. Try instead

% expected number of lines in file
n = 10000;

% size data matrix
data = zeros(n,3);

% insertion index (of new data) into matrix
dataIdx = 0;

while ~feof(fid)   % an alternative to your condition
    tline = fgetl(fid);
    if isempty(findstr(tline,'*'))  % or whatever string pattern works
       dataIdx = dataIdx + 1;
       data(dataIdx,:) = str2num(tline);
    end
end

% eliminate any blank rows of data
data = data(1:dataIdx,:);

fclose(fid);

There will be some resizing of data if ever the valid number of rows exceeds n but this resizing will be delayed.

An alternative to the findstr and str2num combination, is to use sscanf to convert the data string into the correct format and check to see how many floats were successfully converted. If the answer is not three, then skip this line. So rather than

 if isempty(findstr(tline,'*'))  % or whatever string pattern works
     dataIdx = dataIdx + 1;
     data(dataIdx,:) = str2num(tline);
 end

you could try

    [floatData,c] = sscanf(tline,'%f %f %f');
    if c==3
        dataIdx = dataIdx + 1;
        data(dataIdx,:) = floatData;
    end

Try the above and see if it helps.