How to load numeric & string data from .txt file

Dear all,

It might be a regular and common topic but I am unable to fix this issue properly.

Starting from a .txt file (let's say data.txt):

   data_name1   data_name2    data_name3
        0           OK          0,1,2
        1          NOK          0,2.05E-05,8.5

Goal is to have in matlab workspace:

data_name1 = [ 0 ; 1 ]

data_name2 = {'OK','NOK'}

data_name3 = [ 0 1 2 ; 0 2.05E-05 8.5]

I tried readtext.m from file exchange but take huge amount of time and need lots of specific treatment afterward. Using eval also increased processing time of my function. Note: I have no specific toolbox apart from simulink.

Thanks in advance,

Pierre

[Luca Parziale]

Hi, maybe my code can help you. It's not better than Matt Kindig but it may inspire you :

% load .txt data into string vector
PC = readlines("pouvoir_calorifique_brute_moyen.txt");
PC = split(PC); % separate at each space to create an string array with column
PC = replace(PC,"_"," ") % now change the multi-words names 

labelx = PC(1,2);
labely = PC(1,3);
l = length(PC);
PC_kg = PC(2:l, 2); PC_kg = str2double(PC_kg);
PC_m3 = PC(2:l, 3); PC_m3 = str2double(PC_m3);
labels = PC(2:l, 1);

Answer 1

[Matt Kindig]

This should do it, for your specific case:

fid = fopen('data.txt', 'rt');
C = textscan(fid, '%d %s %f,%f,%f', 'HeaderLines', 1, 'CollectOutput', true);
fclose(fid);
data_name1 = C{1};
data_name2 = C{2};
data_name3 = C{3};

[Petrus]

Thank you very much for your quick answer Matt.

My example was just for convenience. My issue is that I don't know in advance:

1/ the kind of data I will have to extract (data_name1 can represent either a scalar or string or a vector,

2/ the name data_namei are inside the file .txt. I need to get them so as to load them on workspace.

3/ the number of data_namei is unknown.

Regards

[Matt Kindig]

Hmm, so that makes it substantially trickier. How large is the file? If you can read it into Matlab all at once, you might be able to use regular expressions to do this. I don't have time right now to code up a full solution, but this sample code should get you started.

EDIT: Found a better way, updated code to minimize eval() statements.

 str = fileread('data.txt');   %read entire file into string
  parts = strtrim(regexp( str, '(\r|\n)+', 'split'));  %split by each line
  columns = strtrim( regexp(parts{1}, '\s+', 'split'));  %columns
  ncol = length(columns);  %number of columns
  parts(1)= [];  %remove column headers
  nrows = length(parts);  %number of rows
  M = cell( nrows, ncol);  %pre-allocate empty cell array for data
  %now loop through parts
  for k=1:nrows,
    data = strtrim(regexp( parts{k}, '\s+', 'split'));   %split by spaces
    M(k,:) = data;
  end
  %now assign variables
  for k=1:ncol,
     row = M(:,k);
    eval( sprintf('%s=row', columns{k}));
end

Keep in mind that this will regard all text as cell arrays of characters. To convert to numeric arrays, you will have to do some more work-- one of the difficulties is how to distinguish between pure text (e.g. 'OK'), and numeric data with chars in between (like the comma delimited list in data_name3).

[Petrus]

I will try this. Thanks a lot. Question was in a more general situation, yes. Text file can be up to 400Mb and can be loaded into workspace (in fact I have about 10 files of this size but I defined a loop for each file).

My main pb was that it could take hours notably due to the use of readtext and mainly due to the eval function (ncol ~= 200; nrow ~= 1000000) when trying assigning the value (remaining part of code you mentionned). See part of code hereafter (variable 'a' being the cell output from readtext ; which gathers all data).

The treatment of numeric vectors (1000000 x 3) takes much more time (x10000) than numeric scalar treatment ( 1000000 x 1), which seems to be non optimized :)

Thanks in advance. Pierre

    for k = ind_names{K}    % ind_names defines the indices to be loaded for the Kth fille (by default all)

    try 
        ind_numeric = isnumeric(cell2mat(a(2:end,k)));
    catch
        ind_numeric = 0; 
    end
        if ind_numeric == 1 && isempty( min(strfind(a{2,k},',')) ) == 1
            eval( [ a{1,k} '_ref = cell2mat(a(2:end,k));'] );

        else
            if min(strfind(a{2,k},',')) > 1
                for j = 2:size(a,1)
                    eval( [ a{1,k} '_ref(j-1,:) = str2num(a{j,k});'] );
                end
            else
                eval( [ a{1,k} '_ref = a(2:end,k);'] );
            end
        end
    end