using datastore does not generate exact data files (extra rows or rows missing)

1 visualización (últimos 30 días)
Hi there,
I have been trying to use datastore and apply it to the file attached (the file is an extracted section of one of our data files). The script I use to set up the datastore setting is as follows:
ds = datastore(data_file_loc_ds);
ds.ReadSize = 1000000;
ds.NumHeaderLines = 0;
ds.Delimiter = {'\t',' '};
% ds.Whitespace = {' \b\t'};
ds.MultipleDelimitersAsOne = 1;
ds.CommentStyle = {'%'}
ds.TreatAsMissing = {'-1.#IO'};
ds.TextscanFormats = {'%{dd/MM/yyyy}D','%{HH:mm:ss}D',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f',' %f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f','%f','%f','%f','%f','%f','%f',...
'%f','%f','%f','%f'};
ds.ReadVariableNames = 1;
ds.VariableNames(1:2) = {'date','time'};
preview(ds)
The created datastore seems to do different things for slightly different files. It generally tends to skip the first row (or a few) of the files. At times it repeats a section of the data file and hence for instance I end up with several extra files and sometimes it skips rows. I have been trying different combinations of the "TabularTextDatastore" properties to get the rows exactly as they are but it seems to be impossible.

Respuestas (0)

Categorías

Más información sobre Large Files and Big Data en Help Center y File Exchange.

Productos


Versión

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by