Convert a non uniform cell array of cell arrays to matrix

I have the following code:

%% Loading the data
rhoPart = 2540;

files = dir(fullfile(uigetdir,'\**\*.data*'));
[~,Index] = natsort({files.name});
files = files(Index);
folders = {files.folder};
folder_groups = findgroups(folders);

k = 1;
for i = 1:length(files)
    fid = fopen(fullfile(files(i).folder,files(i).name),'r');
    %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    frewind(fid);
    % Write headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
    runData{k} = strsplit(fgetl(fid), ' ');
    % Write only the x, y, and z components of the particles, particle radius,
    % z component+ particle radius and volume of the particle
    expData{k} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    % Write only the vx,vy,vz of the particles and magnitude
    velData{k} = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
    fclose(fid);
    k = k + 1;
end

%% Classify (into a structure)
runData_struc = splitapply(@(x) {x}, runData, folder_groups);
expData_struc = splitapply(@(x) {x}, expData, folder_groups);
velData_struc = splitapply(@(x) {x}, velData, folder_groups);

This will give me the following cell array of runData_struct (see pictures)

As you can see this is a cell array in a cell array. I want to convert the 1x9 cell into a 1x9 double. For example:

runData_struc{1,1}

row 1: 1x9 double

row 2 :1x9 double

row 3: 1x9 double

...

row 2741: 1x9 double

runData_struc{1,2}

row 1: 1x9 double

row 2 :1x9 double

row 3: 1x9 double

...

row 2745: 1x9 double

I tried the following, but it gave me NaN as result:

https://www.mathworks.com/matlabcentral/answers/231291-how-to-convert-a-cell-array-of-cell-arrays-to-matrix

[Matt J]

I suggest attaching a .mat file with runData_struc, so we can work with it directly.

[Tessa Kol]

The .mat file is too large to send. I think the problem is that runData contains the header, thus matlab doesn't see it as numeric values.

Maybe the following pictures are helpfull

The thing that did work was the following line of code, which doesn't give me NaN as output:

str2double(runData_struc{1,1}{1,1})
ans =

   1.0e+03 *

    4.7570         0   -0.0003         0         0    0.0003    0.0001    0.0023       NaN

The last NaN in the output is not relevant for me.

But how can I do this automatically for every 1x9 cell array in 1x2741 cell array? And also for every 1x9 cell array in 1x2745 cell array?

[Sindar]

I may be misunderstanding, but shouldn't this work:

A1 = cell2mat(runData_struc{1});
A2 = cell2mat(runData_struc{2});

if that doesn't work, something like this might (but I'm not the cellfun expert)

A1 = cell2mat(cellfun(@cell2mat,runData_struc{1},'UniformOutput',false));
A2 = cell2mat(cellfun(@cell2mat,runData_struc{2},'UniformOutput',false));

[Tessa Kol]

Well I have some good news and some bad news.

The good news is that I can convert it to a double now with the following code:

A1 = cell2mat(cellfun(@cell2mat,runData_struc{1},'UniformOutput',false ));
B1= str2double(regexp(A1,'\d*','match'));

The bad news is that instead of

row 1: 1x9 double

row 2 :1x9 double

row 3: 1x9 double

...

row 2741: 1x9 double

I get

row 1: 1x16446 double

Which is not quite what I want.

[Sindar]

Ah, sorry, I missed that the innermost cells contained chars not doubles. That changes things a little. But, before altering the solution, is there a reason that you use this line:

runData{k} = strsplit(fgetl(fid), ' ');

instead of something that will read that line as numbers?

If not, it appears something like this should work to get you numbers directly

runData{k} = textscan(fgetl(fid),'%f %f %f %f %f %f %f %f ','HeaderLines',0);

which should then work better later.

In fact, if that works (and you always have the same number of columns), you can probably avoid storing each row in a cell, and pre-sort them by folder

%% Loading the data
rhoPart = 2540;
files = dir(fullfile(uigetdir,'\**\*.data*'));
[~,Index] = natsort({files.name});
files = files(Index);
folders = {files.folder};
folder_groups = findgroups(folders);

% pre-allocate one cell for each folder group
for ind=1:max(folder_groups)
    runData{ind} = [];
    expData{ind} = [];
    velData{ind} = [];
end

% why have i and k separately?
for k = 1:length(files)
    fid = fopen(fullfile(files(k).folder,files(k).name),'r');
    %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    frewind(fid);
    % Read headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
    C = textscan(fgetl(fid),'%f %f %f %f %f %f %f %f ','HeaderLines',0);
    % add the row to the matrix for the correct folder group
    % .' transposes. If this errors or creates 1xN matrices, remove .'
    runData{folder_groups(ind)}(:,end+1) = C{1}.';
    % Read only the x, y, and z components of the particles, particle radius,
    % z component+ particle radius and volume of the particle
    % add the rows to the matrix for the correct folder group
    expData{folder_groups(ind)}(:,end+1) = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    velData{folder_groups(ind)}(:,end+1) = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
    fclose(fid);
end
% no post classification needed

(it may also be worth looking at Matlab tables, but that transition should be straightforward here and can be done after loading in the matrices, with array2table)

[Tessa Kol]

Thank you for your help! The textscan solution worked.

When I searched on how I could read the headerline from a textscan I got the following solution from another post:

https://nl.mathworks.com/matlabcentral/answers/348038-how-to-read-the-first-line-names-variables-of-a-text-file#answer_273561

It works for reading the headerline, but apparently it makes it much harder to use the values within the headerline. I adjusted the code to:

%% Loading the data
rhoPart = 2540;
files = dir(fullfile(uigetdir,'\**\*.data*'));
[~,Index] = natsort({files.name});
files = files(Index);
folders = {files.folder};
folder_groups = findgroups(folders);

k = 1;
for i = 1:length(files)
    fid = fopen(fullfile(files(k).folder,files(k).name),'r');
    %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    frewind(fid);
    % Read headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
    Headerline = textscan(fgetl(fid),'%f %f %f %f %f %f %f %f %f','HeaderLines',0);
    runData{k} = [Headerline{1}(:) Headerline{2}(:) Headerline{3}(:) Headerline{4}(:) Headerline{5}(:) Headerline{6}(:) Headerline{7}(:) Headerline{8}(:) Headerline{9}(:)];
    % Write only the x, y, and z components of the particles, particle radius,
    % z component+ particle radius and volume of the particle
    expData{k} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    % Write only the vx,vy,vz of the particles and magnitude
    velData{k} = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
    fclose(fid);
    k = k + 1;
end

%% Classify (into a structure)
runData_struc = splitapply(@(x) {x}, runData, folder_groups);
expData_struc = splitapply(@(x) {x}, expData, folder_groups);
velData_struc = splitapply(@(x) {x}, velData, folder_groups);

[Sindar]

Yes, that answer was if the header line contained text, e.g.:

time position velocity

1 0 1

2 1 3

So you want to extract the numbers from the body and the words from the header.

Depending on how much time you'll be using this code, I'd suggest checking out my further suggestions - layers of cells tend to be more expensive and tedious to use than other data storage

[Stephen23]

If you reverse the order then you do not need to call frewind, giving simpler and more efficient code.

Simply read the header first and then the rest of the file data (the 1 specifies how many lines to read:

% Read headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
Headerline = textscan(fid,'%f%f%f%f%f%f%f%f%f',1);
% Read all the data from the file
dataRead = textscan(fid,'%f%f%f%f%f%f%f%f%f%f%f%f%f%f');

[Tessa Kol]

Thank you both for the suggestions and I will adapt my code according to it. Can I accept an answer to close this post?

Answer 1

[Sindar]

(see comments on question for details not related to the title question)

A1 = cell2mat(runData_struc{1});
A2 = cell2mat(runData_struc{2});

if that doesn't work, something like this might (but I'm not the cellfun expert)

A1 = cell2mat(cellfun(@cell2mat,runData_struc{1},'UniformOutput',false));
A2 = cell2mat(cellfun(@cell2mat,runData_struc{2},'UniformOutput',false));