Vertically concatenate blocks of arrays

Hi everyone,

I want to concatenate the arrays from 4 files, (each corresponding to a block in a task) for every subject in a list. Each file has 5 variables of interest with 60 rows. I combine them to create a 60x5 array for every block.

I want to concatenate the 4 arrays, to create a single 240x5 array. I repeat this for another phase, where I am trying to generate an output of the same dimensions.

Then, I would like to store these arrays inside cells, to create a Nx3 cell array (N= subjects) with the first column being the ID, then the first phase 240x5 array and then the second phase 240x5 array.

The problem I have right now is that my loop keeps parsing arrays in every cell, so in {2,2} I have a 480x5, in {3,2} a 720x5 and so on.

I know that probably it has something to do with the general organization of my loops, but I am struggling finding the right arrange. Could you please guide me on how to fix this issue? I hope my explanation is clear, otherwise let me know.

Best, Ramiro

%%Empty matrices to populate with results
matLenght = lastID-20;
respMat = cell(matLenght,3);

lrnMatrix = zeros(0,5);
expMatrix = zeros(0,5);

%%Loop through each file to get weights
for k = firstID:lastID
    adjustedPosition = k-20;
    subject = num2str(k);

      for kk = 1:4
          block = num2str(kk);
          lrnFile = [subject '_LP_Block' block '.mat'];
          fileDir  = fullfile(dataDir, lrnFile);
          load (fileDir)
          trials = 60;
          location = trials*block;
          lrnData = horzcat(dE.leftWeight, dE.rightWeight, dE.timeTrial, dE.keyResp, dE.bestDecision);
          lrnMatrix = vertcat(lrnMatrix, lrnData);
      end

      for kk = 1:4
          block = num2str(kk);
          expFile = [subject '_EP_Block' block '.mat'];
          fileDir  = fullfile(dataDir, expFile);
          load (fileDir)
          trials = 60;
          location = trials*block;
          expData = horzcat(dE.leftWeight, dE.rightWeight, dE.timeTrial, dE.keyResp, dE.bestDecision);
          expMatrix = vertcat(expMatrix, expData);
      end

      %%Insert subject ID in the first column of the array
      respMat{adjustedPosition,1} = k;
      respMat{adjustedPosition,2} = lrnMatrix;
      respMat{adjustedPosition,3} = expMatrix;
  end

[KSSV]

Read about horzcat and vertcat.

[Ramiro Rea]

I did, I use them in the code. My question is specific to this situation. I need to know how to arrange the loops so it only concatenate the arrays within a block per every subject, and not all the arrays from all the subjects.

Answer 1

[Guillaume]

The problem I have when reading your code, other than the lack of comment, is that I come across

load(fileDir)

At this point, I have no way of knowing what variables exist. A load unassigned to any variable pops unknown variables into existence, potentially overwriting existing ones without any warning. For all we know, it's replaced all the variables you've just created by new ones.

I'm guessing that the load creates at least one structure called dE, otherwise your code makes no sense. Personally, I would use:

matcontent = load(fileDir);
dE = matcontent.dE;
expData = [dE.leftWeight, dE.rightWeight, dE.timeTrial, dE.keyResp, dE.bestDecision];

Now, if I understood correctly, the problem you have is that you're not resetting your lrnmatrix and expMatrix to empty in between subjects. That's easily fixed with by moving their allocation inside the k loop:

for subject = (firstID:lastID)-20  %your k loop
      lrnMatrix = zeros(0,5);
      expMatrix = zeros(0,5);
      for block = 1:4
          lrnFile = sprintf('%d_LP_Block%d.mat', subject, block);
          lrncontent = load(fullfile(dataDir, lrnFile));
          dE = lrncontent.dE;
          expData = [dE.leftWeight, dE.rightWeight, dE.timeTrial, dE.keyResp, dE.bestDecision];
          lrnMatrix = [lrnMatrix; expData];  %#ok<AGROW> suppress warning about growing a matrix in loop
          expFile = sprintf('%d_EP_Block%d.mat', subject, block);
          expcontent = load(fullfile(dataDir, expFile));
          dE = expcontent.dE;
          expData = [dE.leftWeight, dE.rightWeight, dE.timeTrial, dE.keyResp, dE.bestDecision];
          expMatrix = [expMatrix; expData]; %#ok<AGROW> suppress warning about growing a matrix in loop
      end
      respMat{subject, 1} = subject+20;
      respMat{subject, 2} = lrnMatrix;
      respMat{subject, 3} = expMatrix;
end

I've made some other changes which I think make the code cleaner (including removing the calculation of location since it wasn't used).

[Ramiro Rea]

Thank you very much, Guillaume. Yes, sorry for the mess, I erased some of the code to attempt to make it less distracting, but I only made it confusing. I implemented the changes you mentioned and it worked perfectly. As you pointed out, the problem was that the load did not overwrote the previous variables, creating an evergrowing matrix instead of stopping in the 4 iterations per subject. Also, your suggestions to make it cleaner where very informative.

[Guillaume]

No, the problem wasn't with load, the problem was the location of the two lines

      lrnMatrix = zeros(0,5); %matrix for the learning phase
      expMatrix = zeros(0,5); %matrix for the experimental phase

They needed to be inside the outer loop.

load as you used it was dangerous (if one of the files unexpectedly contained a variable expMatrix, or respMat, or etc. it would have overwritten your variable without you knowing it), but it wasn't the cause of your problem.

To me, it looks strange to have twice the same loop for block = 1:4 particularly as the 2nd loop doesn't depend on the first, so its body could be moved inside the 1st as I did in my answer.

[Ramiro Rea]

Yes, agree. I shouldn't answer that late, sorry, now that I read my answer it doesn't make sense. You are right about the loops, I changed it. I removed the code in the last answer to avoid confusion and I will attach it to the end of this answer. Thanks for taking your time in answering this, it was very helpful.

clc; clear

%%Set data directories
cd('..')
rootDir    = pwd;
dataDir    = fullfile(rootDir, 'eyeDATA'); %location of the mat files
resultDir  = fullfile(rootDir, 'Results'); %where we are saving the output
scriptsDir = fullfile(rootDir, 'Scripts'); %location of the scripts

%%Tell which subjects you want to process
firstSubj = inputdlg('First subject of the list');
lastSubj  = inputdlg('Last subject of the list');

firstID = str2double(firstSubj{1});
lastID  = str2double(lastSubj{1});

%%Empty matrix to populate with final output
matLenght = lastID-20; %get the number of rows to prealocate
respMat   = cell(matLenght,3); %create the empty cell array for the output

%%Loop through each file to retrieve the information
for subject = firstID:lastID

      %%Empty matricex to populate with results of each phase (4 blocks)
      lrnMatrix = zeros(0,5); %matrix for the learning phase
      expMatrix = zeros(0,5); %matrix for the experimental phase

      %%Get the row where each subject's data will be saved in the final matrix
      subjectLoc = subject-20; %the user ID's started at 21

      %%Loop through each phase blocks
      for block = 1:4

          %%Data from Learning Phase
          lrnFile    = sprintf('%d_LP_Block%d.mat', subject, block); %name of the file to load
          lrnDir     = fullfile(dataDir, lrnFile);
          lrnContent = load (lrnDir);
          lrnStruct  = lrnContent.dE; %structure that stores the subjects information
          lrnData    = [lrnStruct.leftWeight, lrnStruct.rightWeight, lrnStruct.timeTrial, lrnStruct.keyResp, lrnStruct.bestDecision]; % Horizontally merge variables of interest
          lrnMatrix  = [lrnMatrix; lrnData]; %#ok<AGROW> %vertically merge merge the 4 blocks' results in one single array per subject

          %%Data from Experimental Phase
          expFile    = sprintf('%d_EP_Block%d.mat', subject, block);
          expDir     = fullfile(dataDir, lrnFile);
          expContent = load (expDir);
          expStruct  = expContent.dE;
          expData    = [expStruct.leftWeight, expStruct.rightWeight, expStruct.timeTrial, expStruct.keyResp, expStruct.bestDecision];
          expMatrix  = [expMatrix; expData]; %#ok<AGROW>

      end

      %%Insert values in each column of the final array (Subject, Learning Phase, Experimental Phase)
      respMat{subjectLoc,1} = subject;   %subject ID
      respMat{subjectLoc,2} = lrnMatrix; %data from the 4 learning blocks
      respMat{subjectLoc,3} = expMatrix; %data from the 4 experimental blocks

end

%%Save output in Result folder
if ~exist(resultDir,'dir')
    mkdir(resultDir)
end

cd(resultDir)
save('allSubjects_Summary.mat','respMat')

%%Go back to where we started
cd(scriptsDir)