How to move (and copy) specific csv files from multiple sub folders into a single folder

Good evening, I have a series of subfolders representing time steps for a set of simulations I am running e.g. 000 for start and 2800 for the end, with a single CSV file outside of the subfolder domain representing iteration counts. Within each subfolder I have a "node" csv file, that is entilled the same for each time step i.e. subfolder. What I would like to do is the following:

1. Automatically copy each node.CSV file from each subfolder and place into a seperate one (ensuring I do not overwrite the primary files)
2. Extract the data from the data form the 1st and 2nd column for the first time step and the 2nd from thereafter, compiling them within a single CSV file, with each column presentating a timestep

My friend kindly helped me today. The position of the "node.csv" file is not static for each folder, so the code compensates for this. I was wondering if someone could help with the efficency of the code.

% Start with a folder and get a list of all subfolders.
% Finds and prints names of all PNG, JPG, and TIF images in 
% that folder and all of its subfolders.
close;clc;clear    % Clear the command window.
workspace;  % Make sure the workspace panel is showing.
format longg;
format compact;

% Define a starting folder.
start_path = fullfile(matlabroot);
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
	return;
end

% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
	[singleSubFolder, remain] = strtok(remain, ';');
	if isempty(singleSubFolder)
		break;
	end
	listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)
%%

for k = 1 : numberOfFolders
	% Get this folder and print it out.
	thisFolder = listOfFolderNames{k};
	fprintf('Processing folder %s\n', thisFolder);
	
	% Get csv files.
	filePattern = sprintf('%s/*.csv', thisFolder);
	baseFileNames = dir(filePattern);
    numberCSV = length(baseFileNames)
	
	% Now we have a list of all files in this folder.
	if numberCSV >= 1
		% Go through all those image files.
		for f = 1
			fullFileName = fullfile(thisFolder, baseFileNames(f).name);
			fprintf('     Processing CSV files %s\n', fullFileName);
		end
	else
		fprintf('     Folder %s has no CSV files in it.\n', thisFolder);
	end
end
%%

for k = 1 : numberOfFolders
	% Get this folder and print it out.
	thisFolder = listOfFolderNames{k};
	fprintf('Processing folder %s\n', thisFolder);
	
	% Get csv files.
	filePattern = sprintf('%s/*.csv', thisFolder);
	baseFileNames = dir(filePattern);
    numberCSV = length(baseFileNames)
	% Now we have a list of all files in this folder.
	if numberCSV >= 5
		% Go through all those image files.
		for f = 5
			fullFileName{k} = fullfile(thisFolder, baseFileNames(f).name);
			%fprintf('     Processing CSV files %s\n', fullFileName);
		end
	else
		%fprintf('     Folder %s has no CSV files in it.\n', thisFolder);
	end
end

%%
nameee = 'node.csv'
for k = 1 : numberOfFolders
	% Get this folder and print it out.
	thisFolder = listOfFolderNames{k};
	fprintf('Processing folder %s\n', thisFolder);
	
	% Get csv files.
	filePattern = sprintf('%s/*.csv', thisFolder);
	baseFileNames = dir(filePattern);
    numberCSV = length(baseFileNames)
	% Now we have a list of all files in this folder.

    for f = 1:length(baseFileNames)
        if strcmp(nameee,baseFileNames(f).name)==1
            fullFileName{k} = fullfile(thisFolder, baseFileNames(f).name);
        else
            continue
        end
    end
end
%%
node = nan(515,length(fullFileName))
node(:,1) = (1:515);
for k = 2:length(fullFileName)
    data = xlsread(fullFileName{k});
    node(:,k) = data(:,2);
end

disp('finito')
%%
xlsread(fullFileName{1})

Answer 1

[Aylin]

If you have a recent version of MATLAB, you could try using TabularTextDatastore which can make your code simpler (and possibly faster, although you'll need to measure and check if this is the case with your setup). Something like the following might work for you:

% Create a datastore to refer to the CSV files.
ttds = tabularTextDatastore(topLevelFolder, "IncludeSubFolders", true);

% Make sure that only files that end with "node.csv" are in the datastore.
file_indices = endsWith(ttds.Files, "node.csv");
ttds.Files = ttds.Files(file_indices);

t = table([1:515]');
% Iterate through the files in the dataset and append to a table.
reset(ttds);
while hasdata(ttds)
    % Read from each node.csv file.
    [data, info] = read(ttds);
    
    % Copy the second column of each input file into the output table.
    t(:, end+1) = data(:, 2);
end

% Rename the variables in the table so that it is nicer to access each iteration's data.
t.Properties.VariableNames = ["Node", [ "Iteration" + [1:length(ttds.Files)] ]];

This creates a MATLAB table called t that contains variables for each iteration column read from each file. You can index into this using t.Node, t.Iteration1, t.Iteration2, etc.

You can also turn the table into a normal MATLAB matrix using:

node = t{:, :};

I hope this helps!