batch processing (automating code processing)

Hi all,

I have a question about running a code on multiple datasets. I have seen many examples and methods, but unfortunetly, they did not work for me.

I have over 500 data sets, all have same structure of data but different numbers (arranged in sruct.array) so they are already in .mat form.

Those .mat files are stored on my computer as follow :mainfolder > 5 subfolder > N number of subsubfolder > 15 .mat files

Thus, for each subsusbfolder, there is 15 .mat files.

So, basically I want to process those .mat files by calling a function on all of those data sets automatically, but not really sure how to do so especially that I need a for loop to navigate my subfolders and subsubfolders to reach to the .mat files.

Any help would be appreciated.

Answer 1

[Mario Malic]

This will give you the structure containing all .mat files.

File_Struct = dir('**/*.mat');

Edit:

Use name and folder to get the full path to each .mat file.

File_Struct = dir(fullfile(myFolder, '**/*.mat'))
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
    File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name); 
    fprintf(1, 'Now reading %s\n', File_Path {ii});
    load(File_Path {ii}); %this is changed
    data=myfunc1(data)% calling my function1
    data=myfunc2(data)% calling my function2
    Processed_Data(ii).Value = data; % structure
end

[maryam al labbad]

Thanks for the quick response. this piece of code load all .mat files into matlab, but I am not sure how to run the 2 functions I have to each .mat file.

[Steven Lord]

This does not load anything into MATLAB. It gives you a list of file information through which you can iterate (with a for loop) to load and process.

cd(matlabroot)
cd toolbox/matlab
theFiles = dir('**/*.mat');

In my installation of release R2019b, one of the files in the demos directory is element 40 of theFiles.

>> theFiles(40)

ans = 

  struct with fields:

       name: 'trimesh3d.mat'
     folder: 'C:\Program Files\MATLAB\R2019b\toolbox\matlab\demos'
       date: '25-Nov-2008 12:44:41'
      bytes: 24987
      isdir: 0
    datenum: 733737.531030093

To load this file:

>> F = theFiles(40);
>> fn = fullfile(F.folder, F.name)
fn =

    'C:\Program Files\MATLAB\R2019b\toolbox\matlab\demos\trimesh3d.mat'

>> data = load(fn)
data = 

  struct with fields:

    tri: [2174×3 double]
      x: [1079×1 double]
      y: [1079×1 double]
      z: [1079×1 double]

You can do whatever is appropriate with the data from the file. In this case it's a meshed object so trimesh is appropriate:

>> trimesh(data.tri, data.x, data.y, data.z)

[maryam al labbad]

Thanks Steven, I am a beginner on matlab, so my apologies if I am asking basic questions.

Do I have to indicate the main folder directory? like I am not sure from where to start. I have this piece of code that I am trying to let it work, but it says that unrecognize function or variable data although when I am calling these 2 functions outside of the loop on each .mat file, they work fine. (each .mat file is named data.mat and it has the inputs for each function)

myFolder = 'C:\Users\m\Desktop\try'; % main folder
if ~isdir(myFolder)
  errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
  uiwait(warndlg(errorMessage));
  return;
end
filePattern = fullfile(myFolder, '*.mat');
matFiles = dir(filePattern);
for k = 1:length(matFiles)
  baseFileName = matFiles(k).name;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);
  matData(k) = load(fullFileName);
  data=myfunc1(data)% calling my function1
  data=myfunc2(data)% calling my function2
end

[maryam al labbad]

Mario, I really appreciate your efforts in helping me out, do I have to include main folder directory, so the code should be

myFolder = 'C:\Users\m\Desktop\try'; % main folder
File_Struct = dir('**/*.mat');
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
    File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name); 
    fprintf(1, 'Now reading %s\n', File_Path {ii});
    matData(ii) = load(fullFileName); % matData is not used in the code here?
    data=myfunc1(data)% calling my function1
    data=myfunc2(data)% calling my function2
end

[Mario Malic]

This works if you would like to look for files within other folders.

File_Struct = dir(fullfile(myFolder, '**/*.mat'))

Without specifying the folder, it will look for all .mat files within current folder and subfolders.

[maryam al labbad]

File_Struct = dir(fullfile(myFolder, '**/*.mat'))
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
    File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name); 
    fprintf(1, 'Now reading %s\n', File_Path {ii});
    matData(ii) = load(fullFileName); % matData is not used in the code here?
    data=myfunc1(data)% calling my function1
    data=myfunc2(data)% calling my function2
end

Hi Mario,

With the above code, I am not getting anything. Am I missing a core piece?

File_Struct = 

  0×1 empty struct array with fields:

    name
    folder
    date
    bytes
    isdir
    datenum

[Mario Malic]

I think so, double check your myFolder.

[maryam al labbad]

I may not explain my question clearly, here is a summary of how my files look like. I want to run same 2 fucntions on each .mat file.

C:\Users\m\Desktop\try

+---subfolder_1

+-----data1.mat

+-----data2.mat

+-----data3.mat

+-----data4.mat

+-----data5.mat

+-----data6.mat

+-----data7.mat

+-----data8.mat

+-----data9.mat

+-----data10.mat

+-----data11.mat

+-----data12.mat

+-----data13.mat

+-----data14.mat

+-----data15.mat

+---subfolder_2

+-----data1.mat

+-----data2.mat

+-----data3.mat

+-----data4.mat

+-----data5.mat

+-----data6.mat

+-----data7.mat

+-----data8.mat

+-----data9.mat

+-----data10.mat

+-----data11.mat

+-----data12.mat

+-----data13.mat

+-----data14.mat

+-----data15.mat

+---subfolder_3

+-----data1.mat

+-----data2.mat

+-----data3.mat

+-----data4.mat

+-----data5.mat

+-----data6.mat

+-----data7.mat

+-----data8.mat

+-----data9.mat

+-----data10.mat

+-----data11.mat

+-----data12.mat

+-----data13.mat

+-----data14.mat

+-----data15.mat

+---subfolder_4

+-----data1.mat

+-----data2.mat

+-----data3.mat

+-----data4.mat

+-----data5.mat

+-----data6.mat

+-----data7.mat

+-----data8.mat

+-----data9.mat

+-----data10.mat

+-----data11.mat

+-----data12.mat

+-----data13.mat

+-----data14.mat

+-----data15.mat

+---subfolder_5

+-----data1.mat

+-----data2.mat

+-----data3.mat

+-----data4.mat

+-----data5.mat

+-----data6.mat

+-----data7.mat

+-----data8.mat

+-----data9.mat

+-----data10.mat

+-----data11.mat

+-----data12.mat

+-----data13.mat

+-----data14.mat

+-----data15.mat

[Mario Malic]

File_Struct can't be empty, commands above will return .mat files frorm folder C:\Users\m\Desktop\try and all subfolders within it. Do you have a typo in your folder, or files are not there?

[maryam al labbad]

With the following code, I can access all .mat files in the folder.

% Specify the folder where the files live.
myFolder = 'C:\Users\m\Desktop\try\WSW';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
    errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
    uiwait(warndlg(errorMessage));
    myFolder = uigetdir(); % Ask for a new one.
    if myFolder == 0
         % User clicked Cancel
         return;
    end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
    baseFileName = theFiles(k).name;
    fullFileName = fullfile(theFiles(k).folder, baseFileName);
    fprintf(1, 'Now reading %s\n', fullFileName);
       data=myfunc1(data)% calling my function1
       data=myfunc2(data)% calling my function2
end

However, I cannot call the functions. It shows the following error message:

Unrecognized function or variable 'data'.

Error in Untitled14 (line 20)
 data=myfunc1(data);
 

[Mario Malic]

I can't see what file contains as I do not have the file. Actually, I don't know how your code works, data is not defined in it. I did write this on my answer above.

matData(ii) = load(fullFileName); % matData is not used in the code here?

Documentation for the load function:

S = load(___) loads data into S, using any of the input arguments in the previous syntax group.

• If filename is a MAT-file, then S is a structure array.

    matData(ii) = load(fullFileName); % matData is not used in the code
%     data = load(fullFileName); % I can't tell is this correct
    data=myfunc1(data)% calling my function1
    data=myfunc2(data)% calling my function2

You have to fix these lines according to the documentation above, I don't know what these functions do and is data overwritten or appended, therefore I can't really know how to proceed with this.

[maryam al labbad]

Sorry for making all the confusions. I hope I can make it clear in this post. The last code I posted is different than the first one. The last one showed me all files on my folders and subfolders in matlab, the previous one was showing zero if you recall. So the code that I am working on right now is:

% Specify the folder where the files live.
myFolder = 'C:\Users\m\Desktop\try\WSW';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
    errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
    uiwait(warndlg(errorMessage));
    myFolder = uigetdir(); % Ask for a new one.
    if myFolder == 0
         % User clicked Cancel
         return;
    end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
    baseFileName = theFiles(k).name;
    fullFileName = fullfile(theFiles(k).folder, baseFileName);
    fprintf(1, 'Now reading %s\n', fullFileName);
       data=myfunc1(data)% calling my function1
       data=myfunc2(data)% calling my function2
end

Each .mat file has two type of data stored in structure array such as the following:

data.probe(1).U

data.probe(1).Y

data.probe(1).P

data.probe(2).U

data.probe(2).Y

data.probe(2).P

so my functions :

data=myfunc1(data)
data=myfunc2(data)

are used to compute some statsitics and calculate some paramters based on U, Y, and P. both functions are exactly the same, the only difference is that 1st function will compute data and store it in(1), and 2nd function will compute the data for (2).

The .mat file name is data which has U,Y, and P, which are the input for the functions to calculate rest of parameters. Thus, when I run my functions, it usually compute the rest of parameters on the same .mat file but different cells.

[Mario Malic]

I have edited the code in the answer, please adjust variable names to your liking. Below code is not well understood, these asterisks are called wildcards, where any files with .mat extension will be added to filePattern. This example below is not very intuitive because it also searches in subfolders.

filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);

Consider this example

filePattern = dir(fullfile(myFolder, '*.mat'));

This will get you .mat files only in myFolder.

[maryam al labbad]

Thanks Mario, this seems to be working! but the thing is I think all .mat files are overwritten each another. What I want to do is to save the file after each iteration because this way, I am not able to get the data except for the last one. Please guide me, I really apprecaite your help

[Mario Malic]

I have edited the answer, I don't know what the outputs are so, data for each iteration is saved in structure Processed_Data under field Value.

[maryam al labbad]

Mario, that works perfect, have been trying to resolve this for few days, so many thanks, I really apprecaite your time

[Mario Malic]

You're welcome, for the future questions, you can read the help and a tutorial for the questions so that they are faster resolved. Have a great rest of the week!

[maryam al labbad]

Hello again, I am wondering if there is a way to save those files seperatly in a specific directory? now I am getiing all of them in a structure, just wondering how to save them seperatly? Thanks a lot

[Mario Malic]

You can use function save, you can check it in the documentation.

You can put this line of code in the loop on the appropriate place in the loop. I am not familiar if you're having one or more files in a single folder. You will have to modify the name between each file so it doesn't get overwritten.

save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data');

I did not test this, this will overwrite each .mat file in folder.

However, I would suggest not do it this way, because it will take long to load and save, especially if you have a lot of files. You can add additional identifier in the structure for each analysed file/folder.

[maryam al labbad]

Okay, thank you so much!

[maryam al labbad]

Hi, let's say I want to run more than one function on the code, and I want to save the result of other function. I did that, but it overwrite the saved file. I changed data in the following code to the other function name, but it seems that it overwite the data file in the folder. How can I make it save a new file of the result of the other functions. does it deal with .name thing?

Thanks

save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data');
save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data2');

[Mario Malic]

This time, I'll suggest you to read the documentation on the functions used in these two lines. I want to help, but if I'm going to answer every question you might have, it won't lead you further ahead.

So, read up on save and fullfile.

What value does fullfile return? What input arguments save function takes? Once you learn that, you'll be able to solve your problem. You were on the right track with the answer - does it deal with .name thing?

[maryam al labbad]

Great! thanks