Batch Process for CSV files

Hi,

I'm trying to open multiple csv files using a loop. The CSV files have two columns that are labeled as DATA1 and DATA2. The problem I encountered is that the loop only reads the last csv file in the directory. I want to be able to create a graph for each csv file and output it as tiff in the same folder where the csv files are located. It only outputs 2 figures and stops. Any help will be appreciated. My code looks like this so far:

if true
  % code
clc
clear
%%Read CSV Files from Directory
CSVF=dir('*.csv');
x=cellstr(ls('*.csv'));
k=size(CSVF,1);
CSVFI={CSVF.name};
str=strcat('C:\Users\gabri_000\Downloads\3-14 Injection Molding\3-14 Injection Molding\', CSVFI);
%%Multiple Files
   for i=1:length(CSVF)
   sfile=CSVF(i).name
   delimiter = ',';
   startRow = 2;
   formatSpec = '%f%f%f%*s%*s%*s%[^\n\r]';
   fileID=fopen(sfile,'r');
   dataArray = textscan(fileID, formatSpec, 'Delimiter',         delimiter,'HeaderLines' ,startRow-1, 'ReturnOnError', false);
   fclose(fileID);
%%Allocate imported array to column variable names
   DATA1 = dataArray{:, 1};
   DATA2 = dataArray{:, 2};
%%Time relative to # of rows in excel file
   Time=find(DATA1==DATA1);
   ADJTIME=(Time/100);
%%Min and Max Values
% DATA 1
   MINLEAD1=min(DATA1);
   MAXLEAD1=max(DATA1);
% DATA 2
   MINLEAD2=min(DATA2);
   MAXLEAD2=max(DATA2);
%%FILTER PZT LEAD 1
   tv3=[ADJTIME DATA1];
   [B,A]=butter(1,0.02,'low');
   h= figure(1);clf
for i=1:1
  tv3f(:,i)=filtfilt(B,A,tv3(:,i+1));     % Filtered signal
  plot(tv3(:,1),tv3(:,i+1),'k');hold on  % Plot raw data
  plot(tv3(:,1),tv3f(:,i),'r');hold on   % Plot filtered signal
  tv3s(:,i)=cumsum(tv3f(:,i));
  plot(tv3(:,1),0.001*tv3s(:,i),'b');hold on
  xlabel('X');
  if i==1,ylabel('Y');end
  print(h,'-dtiff','-r600')
end
%%FILTER PZT LEAD 2
  tv4=[ADJTIME DATA2];
  [B,A]=butter(1,0.02,'low');
  h2= figure(2);clf
for i=1:1
  tv4f(:,i)=filtfilt(B,A,tv4(:,i+1));     % Filtered signal
  plot(tv4(:,1),tv4(:,i+1),'k');hold on  % Plot raw data
  plot(tv4(:,1),tv4f(:,i),'r');hold on   % Plot filtered signal
  tv4s(:,i)=cumsum(tv4f(:,i));
  plot(tv4(:,1),0.001*tv4s(:,i),'b');hold on
  xlabel('X');
  if i==1,ylabel('Y');end
  print(h2,'-dtiff','-r600')
  end
end
end

[Jan]

Please follow the "? Help" link to learn, how to format code in the forum. Thanks. Currently your code is not readable.

Answer 1

[Jan]

As far as I can see, you do not read the last file only, but all files are read. Simply use the debugger to check this by your own: Set a breakpoint in the fopen() line and check, which files are imported. Even a trivial message would clear this question:

fid = fopen(sfile, 'r');
if fid == -1
  error('Cannot open file: %s', sfile);
else
  fprintf('Reading file %s\n', sfile);
end

Please explain where your program stops and why.

[Gabriel Geyne]

You are right, it reads all the files. The problem is that it only does the graphs for the last CSV file, skipping all the other ones. I only get 2 graphs.

Answer 2

[Cedric]

It seems that you have two levels of loops, yet you are using the same loop index for all loops. This creates a conflict (essentially i will be set to 1 by inner loops each time the outer loop sets it to larger values). Note that i and j should be reserved for complex notation, so you could use ii, jj, k, etc, as loop indices.

Also, I suppose that you don't want to use figure(1) and figure(2), but figure(2*ii-1) and figure(2*ii) instead.

I can't test, but if the structure is what I guess it is, you should have something like

 ...
 for ii = 1 : length(..)
    ...
    h1 = figure(2*ii-1) ;
    ...
    for jj = 1 : ...
       ...
    end
    ...
    h2 = figure(2*ii) ;
    ...
    for jj = 1 : ...
       ...
    end
    ...
 end
 ...

[Gabriel Geyne]

I tried this suggestion, but the code stops once it creates the second figure. It pops out a Figure 3 window, but an error comes out. The code looks like:

if    
tv3=[ADJTIME DATA1];
 [B,A]=butter(1,0.02,'low');
for ii=1:length(CSVF)
h1= figure(2*ii-1);
tv3f(:,ii)=filtfilt(B,A,tv3(:,ii+1));     % Filtered signal
plot(tv3(:,1),tv3(:,ii+1),'k');hold on  % Plot raw data
plot(tv3(:,1),tv3f(:,ii),'r');hold on   % Plot filtered signal
tv3s(:,ii)=cumsum(tv3f(:,ii));
plot(tv3(:,1),0.001*tv3s(:,ii),'b');hold on
xlabel('X');
if ii==1,ylabel('Y');end
print(h1,'-dtiff','-r600')
end
%%FILTER PZT LEAD 2
tv4=[ADJTIME DATA2];
[B,A]=butter(1,0.02,'low');
for j=1:length(CSVF)
h2= figure(2*ii);
tv4f(:,j)=filtfilt(B,A,tv4(:,j+1));     % Filtered signal
plot(tv4(:,1),tv4(:,j+1),'k');hold on  % Plot raw data
plot(tv4(:,1),tv4f(:,j),'r');hold on   % Plot filtered signal
tv4s(:,j)=cumsum(tv4f(:,j));
plot(tv4(:,1),0.001*tv4s(:,j),'b');hold on
xlabel('X');
if i==1,ylabel('Y');end
print(h2,'-dtiff','-r600')
end
end

The error I get is:

if 
Attempted to access tv3(:,3); index out of
bounds because size(tv3)=[9020,2].
Error in MathworksAnswer (line 36)
tv3f(:,ii)=filtfilt(B,A,tv3(:,ii+1));
% Filtered signal
end

[Cedric]

When you declare

 tv3 = [ADJTIME DATA1] ;

you define tv3 as a 9020x2 numeric array, where 9020 is the number of rows and 2 the number of columns.

Then, at the beginning of the loop, you have something like:

 for ii = 1 : length(CSVF)
    ..
    tv3f(:,ii) = filtfilt(B, A, tv3(:,ii+1)) ;     % Filtered signal

which saves in column ii of tv3f the column vector (at least it should be one) returned by the function FILTFILT. This function takes as a 3rd argument tv3(:,ii+1), which is column ii+1 of tv3.

Now during the first iteration of the loop ( ii = 1), this expression will extract column ii+1=2 of tv3, which is valid because tv3 has two columns. During the second iteration, however, ii=2 and the expression will try to extract column ii+1=3 of tv3, which doesn't exist.

So you have to determine if you really need a loop, if you really need to access a column of tv3 whose index is increasing with the loop index (+1), and if so, how to define tv3 for it to be valid. It has to have at least length(CSVF)+1 columns if you use a loop that covers the range 1:length(CSVF).