ASCII (.csv) file data extraction

I am just relearning MATLAB coding for my work environment to streamline data analysis. I am trying to extract numerical values from a .csv that has a larger header of string information in it. Along with the data, there is information such as the frequency of which the measurements were taken at within the header. I am trying to write an extract function tailored for our needs but I am having difficulty getting started. I am not sure whether or not to use textread, csvread, etc. Everything that I have tried thus far doesnt seem to work as errors come up. Below is an example of the header followed by the data

SetupTitle, CV US Protocol
PrimitiveTest, C-V Sweep
TestParameter, Channel.UnitTyp, CMU
TestParameter, Channel.Unit, CMU1:MF/SC
TestParameter, Channel.IName, 
TestParameter, Channel.VName, VBias
TestParameter, Channel.Mode, 
TestParameter, Channel.Func, VAR1
TestParameter, Channel.Index, 
TestParameter, Channel.Time, 
TestParameter, Measurement.Primary.Locus, Single
TestParameter, Measurement.Primary.Scale, LINEAR
TestParameter, Measurement.Primary.Start, -10
TestParameter, Measurement.Primary.Stop, 20
TestParameter, Measurement.Primary.Step, 0.1
TestParameter, Measurement.Aborting.Condition, CONTINUE AT ANY
TestParameter, Measurement.PostOutput.Value, START
TestParameter, Measurement.PostOutput.Retention, OFF
TestParameter, Measurement.Secondary.ACLevel, 0.03
TestParameter, Measurement.Secondary.FName, Freq
*TestParameter, Measurement.Secondary.Frequency, 1000000, 100000, 10000, 1000*                                            

...

AnalysisSetup, Analysis.Setup.Preference.ScalarVisible, true
AnalysisSetup, Analysis.Setup.Title, CV US Protocol
Dimension1, 301, 301, 301
Dimension2, 4, 4, 4
DataName, VBias, C, G
DataValue, -10, 7.5466E-11, 8.1216499999999987E-06
DataValue, -9.9, 7.61209E-11, 7.47184E-06 
DataValue, -9.8, 7.65261E-11, 9.57089E-06

So as you can see, the header is full of useless information until the bold line that consists of the frequencies that each measurement was taken at. The subsequent data, as shown above, then reads as shown and the only interesting pieces I need at the 'VBias" and the 'C'. The other funny issue is that these measurements were saved in the same file consecutively so the order that the frequencies were listed in is the order of the sweep data...meaning the data is continuous from one frequency to the next without any header and the only way you know when this happens is the VBias goes from +10V to -10V in the sweep. Anyway, I would appreciate any help extracting this info into a useful array that I can then write manipulate. Thank you in advance.

[Merged information from duplicate question]

I have a .csv file with a large header. On a line in the header, I need to extract the frequencies at which I performed a measurement. For example:

TestParameter, Measurement.Secondary.Frequency, 1000000, 100000, 10000, 1000

I need to be able to extract the 1000000, 100000, 10000 and 1000 each as a variable but the other side of this is that the number or frequencies needs to be flexible. Some measurements, I will only have one frequency, others I might have 10 and I need to be able to put them in an array of variables that I can assign their correlating data to. Any help with this would be appreciated.

Answer 1

[per isakson]

There are several alternatives. However, if the file fits in memory I would

1. read the complete file with textscan (textread will become obsolete)
2. extract the information I need

    fid = fopen( file_spec, 'r' );
    cac = textscan( fid, '%s', 'Whitespace','', 'Delimiter','\n' );
    sts = fclose( fid );                 
    cac = cac{:};

    is_data = strncmp( 'DataValue', cac, 9 ); 
    str = transpose( char( cac(is_data) ) );
    str = cat( 1, str, repmat( ',', 1, size(str,2) ) );

    num = textscan( str, '%*s%f%f%f', 'Delimiter', ',', 'Whitespace', '' )

This is a start that I haven't tested. See Data extraction from .txt file

[Lukas]

Do I need to use boolean logic to extract the frequencies from the string? I would like to use the string of the number as the variable for the 2xm array containing the VBias and C data. Also, how would you isolate the various frequency measurements when they are continuous with each other? Again, in my mind I am thinking it can only be done with boolean logic but then I would have to use str2num wouldnt I? Thanks for the quick response. This is definitely a good start.

[per isakson]

I'm not sure I understand. num is a cell array that you need to "unpack". I never learn exactly what textscan returns. I try i out in debug mode. Anyhow, num contains three row vectors of double. You might want to concatenate these row vectors. Something like M=cat(2,num{1}{1},num{1}{2},num{1}{3}) or make it in steps: num=num{1}; M=cat(2,num{1},num{2},num{3})

[per isakson]

There is no need for str2num(?). The *second* call to textscan has already made the conversion. Do you want to use the line "DataName, VBias, C, G"?

[per isakson]

"isolate the various frequency measurements". M according to previous comment. Mm98 = M( abs(M(:,1)-(-9.8))<0.01, : );

[Lukas]

Yes, ultimately what I would like to do is have a cellarray called Freq with all the frequencies listed and then assign each discrete frequency the corresponding data of only the VBias and C columns....if that is possible.

Answer 2

[Walter Roberson]

If you use textscan() with 'CommentStyle', {'SetupTitle', 'Frequency,'} and 'Delimiter', ',' and number of lines set to 1 and format set to '%f%f%f%f' then you would read the frequencies and be positioned to start reading with the next line. You can then repeat the same sort of CommentStyle trick to read just the Dataname line.

[Walter Roberson]

If you switch the format to a single '%f' in the above textscan() construct, then you would read the first numeric value in the Frequency line. Then fgetl() the rest of that line, and textscan() with that one string variable in place of a file identifier, using 'Delimiter', ',' and a format of '%f'; that will get you the rest of the values on the line.

Answer 3

[Walter Roberson]

Your existing question is still open and active here, and waiting for your responses. Please do not open duplicate questions; you can edit your existing question or reply to the responses you received. Your duplicate question has been deleted.

[Lukas]

Sorry, I just needed to add that the number of variables to extract will vary. I still am not clear as how it works. I am trying to understand the more detailed workings of textscan in order to figure out how to start the extraction of variables and have it continue to the endofline but I dont see how to do it.

Answer 4

[Lukas]

I still havent been able to figure out if it is possible to extract the data (VBias and Capacitance) as a 2 column array that I can separate into separate arrays by using boolean logic of looking at the VBias step size and when the steps size of the next point exceeds the previous point, start a new variable. Can anyone help with this point?

[Walter Roberson]

When you are positioned at the beginning of the first DataValue line:

celldata = textscan(fid, 'DataValue,%f%f%*f', 'Delimiter', ',', 'CollectData', 1);

YourArray = celldata{1};

Answer 5

[Lukas]

fid = fopen(uigetfile({'*.csv';'*.txt';'*.dat'}, 'Select the CV File'));

Is this a valid way to select and open a file? I get a -1 response from fopen indicating it cannot open the file. I dont know if thats a file format problem or a coding problem

[Walter Roberson]

In order for that to work, the file would have to be in the same directory.

It would be more robust to test the returned value from uigetfile() in order to determine whether the user canceled the uigetfile() operation.

It would be clearer to specify the 'r' or 'rt' flag in the fopen() call.

[Lukas]

It would have to be in the same directory even though the uigetfile prompt lets you change directory to select the file? And isnt 'r' the default if nothing else is specified?