how to read multiple csv files with headers?

Hello everyone,

I have few csv files like SS 2012-07-13 to SS 2012-07-30 with a header as the first line.

And the content inside the file is as follows:

Date,Denuder,Htd Line,Impactor,1130 Case,Pump,RPF,Pyro,1135 Case 2012-07-19 23:58:48,95.9,49.2,49.0,31.2,6.80,559.7,450.2,25.1 2012-07-19 23:58:58,94.2,49.2,58.3,31.3,6.80,551.1,441.5,25.1 2012-07-19 23:59:08,92.6,49.1,64.3,31.3,6.80,542.0,432.9,25.1

I want to read these files consecutively using the following code:

close all;clear all;clc;

strDir = 'G:\Shippo 2012\TEKRAN condition\SS\';

fnames = dir(strcat(strDir, '*.csv'));

numfids = length(fnames);

for i = 1:numfids

   X = csvread(strcat(strDir, fnames(i).name));

end

But within the loop matlab can not read the data because of the header and is showing the following error:

Mismatch between file and format string.

Trouble reading number from file (row 1, field 1) ==> Date,

Error in ==> csvread at 52

m=dlmread(filename, ',', r, c);

Error in ==> SS_file_daily_plots_m at 9

X = csvread(strcat(strDir, fnames(i).name));

Does anybody have any idea how can I read those csv files simultaneously with the header.

Thank you very much in advance for the assistance.

Answer 1

[Ken Atwell]

Moklesur,

If you are using a newish version of MATLAB, try loading the file with the Import Tool, and choose to generate a function when importing. This will give you a template to call in your loop.

Otherwise, you may need to use a slightly lower-level function. textscan will let you specify 'HeaderLines' to skip over the header. It might look something like:

 for i = 1:numfids
   f = fopen(strcat(strDir, fnames(i).name);
   X = textscan(f, '%s%f%f%f%f%f%f%f%f', 'Delimiter', ',', 'HeaderLines', 1);
   fclose(f);
 end

X would be a cell array of parsed columns (note this code reads the timestamp as a string). Read more at http://www.mathworks.com/help/matlab/ref/textscan.html

[Md. Moklesur Rahman]

Thanks Atwell.

Answer 2

[Walter Roberson]

csvread() uses dlmread() to do its work, and it does that in a way that only permits numeric values even for header lines.

The last of the data lines you show does not have a date like the previous two do. Is that the case for exactly the 3rd (useful) data line, or for only the last data line, or for random data lines ?

The normal recommendation for dealing with csv numeric values is to switch to textscan(), but textscan() is harder to deal with when there are string fields with missing values.

[Md. Moklesur Rahman]

Thanks Roberson for your heartiest suggestions. The problem was solved correcting the code based on Atwell's suggestion using textscan.