read file variables methods
Mostrar comentarios más antiguos
I got a large data matrix (3Gb) of signal data which I saved to a .mat file. Since the data size is large I figured out it will be wiser to access the file and read each time a block of data, process it and move to the next block instead of loading the complete file. I'm wondering should I use the matfile method, fopen and fread or should I use a datastore and read command. What is faster from the processing point of view and easier to work with? Or any other considerations? Thanks
3 comentarios
per isakson
el 11 de Nov. de 2016
"data set"   is that one numerical array?
"fairly large data set"   how large compared to the physical memory?
Talmon Alexandri
el 12 de Nov. de 2016
per isakson
el 14 de Nov. de 2016
Given
- "large data matrix (3Gb) of signal data which I saved to a .mat file"
- "Total RAM size - 3G[B]"
- "[one] plain numerical array 10 million lines by 48 columns"
I assume the mat-file is version 7.3, because that's needed to store an array of that size.
I would say this a a perfect case for matfile, Access and change variables directly in MAT-files, without loading into memory.
It's the easiest to work with and I don't think there is any other way that is significantly faster.
Respuestas (1)
Walter Roberson
el 11 de Nov. de 2016
0 votos
Categorías
Más información sobre Large Files and Big Data en Centro de ayuda y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!