- MATLAB code writes MATLAB variables to RAM drive/txt file.
- Ancient software reads RAM drive/txt file, processes content, outputs outcome to RAM drive/txt file.
- MATLAB code reads RAM drive/txt file, processes content, stores outcome in variable.
- Loop to 1 if not done.
Fast way to read text file of formatted data
11 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
I have a text file formatted as (just showing the first, second, second to last and last lines):
0.00 68.000 68.000
1.00 68.001 68.000
...
1923.00 1871.164 1869.803
1924.00 1871.484 1870.134
The data values are delimited by spaces (number varies, depending by data values).
I want to import these as floating numbers, eventually in a 3 column array. I will always know ahead of time how many columns there are but I will not know ahead of time how many rows there will be.
I can input this easily using one of many commands: dlmread, importdata, textscan, fscanf. For a resulting 1925x3 array, fscanf is the fastest and takes around .004 sec. Since I will have to do this import over a hundred thousand times in my MATLAB script, is there a faster way to do this? Thanks
0 comentarios
Respuestas (3)
Cedric
el 27 de Mayo de 2014
Editada: Cedric
el 27 de Mayo de 2014
You should/could perform this operation using a RAM drive/disk. There is little interest in saving temporary files on disk when you have a lot of them. Once you have the data in MATLAB, save it in one or a few .mat files, to avoid having to deal with that many files in the future. Your processing flow becomes:
When done: store variable in unique .mat file. If too large to hold in memory, split into a few blocks, e.g. 1GB, to minimize the number of files. As mentioned by Star Rider, the .mat format is well suited for storing large amounts of data; it is based on HDF5 [ ref ] (from version 7.3 on). Yet, if you want to push even further, this post is not uninteresting.
To optimize read/write operations on a very large number of files, stick to low level functions, and try to be as specific as possible during calls (i.e. it is more efficient to specify a separator than to let the function find it out by testing, it is more efficient to specify a date format than to let a function finding it out, etc).
Star Strider
el 26 de Mayo de 2014
I would import them once and save them as a ‘.mat’ file, with their variable names included. Then load the ‘.mat’ each time instead.
9 comentarios
Star Strider
el 27 de Mayo de 2014
I rarely program in FORTRAN now (haven’t in more than a decade) but I have a compiler in my DVD software library that will run on this machine (Win 8) that I keep partially out of nostalgia. I was still writing FORTRAN code for my neural nets and genetic algorithms about 20 years ago because MATLAB was very slow on those machines. FORTRAN was significantly faster, and probably still is for large projects.
I strongly suggest you consider Cedric’s idea of a RAM drive for the temporary files. It is much faster in terms of read-write time — you can also wipe the files from the RAM drive quickly — and you don’t have to worry about HDD file fragmentation that would likely slow your processes.
rifat
el 27 de Mayo de 2014
Editada: rifat
el 27 de Mayo de 2014
I did a similar thing before. Hope this helps. result will be on the variable mat.
fclose('all');
fid=fopen('fileName.txt','rt');
frewind(fid);
lnum=0
count = 1;
while(count==1)
line=fgetl(fid);
lnum=lnum+1;
if ~ischar(line)
count=0;
else
count=1;
line=strtrim(line);
len=length(line);
a=line==' ';
loc=find(diff(a)~=0);
num1=line(1:loc(1));
num2=line(loc(2)+1:loc(3));
num3=line(loc(3)+1:end);
mat(lnum,1)=str2num(num1);
mat(lnum,2)=str2num(num2);
mat(lnum,3)=str2num(num3);
end
end
2 comentarios
rifat
el 27 de Mayo de 2014
yeah.. i wasnt worried about performance.. Thats why this rough implementation
Ver también
Categorías
Más información sobre String Parsing en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!