populating a tall array in a for loop
10 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
*I acknowledge that my approach is flawed, but am curious whether this solution exists.
Can I populate a tall array in a for loop?
I am running a large number of calculations and wanting to store the results in a vector. Lets say the results resemble a 1 by 1*10^12 vector of doubles. Clearly this exceeds the memory of my laptop.
The way I have it coded now is to keep track of how many calculations have been performed. Once a particular number is exceeded, I save of the workspace variable and then clear the variable from memory.
%count = 1
%for i = 1:1*10^12
% if count > 784000000
% if exist('A') == 1
% save('TestsaveforProject.mat','A','-v7.3')
% clear A
% end
% B = zeros(1,784000000);
% B(count-78399999) = calculation
% if count > 2*784000000
% if exist('B') == 1
% save('TestsaveforProject2.mat','B','-v7.3')
% clear B
% C = zeros(1,784000000);
% end
% C(count-2*78399999) = calculation%
% end
%else
% A(count) = Calculation%
%end
%count = count+1;
%end
Can I convert the series of 'if' statements to a few lines to populate a tall table? For 1*10^12 cases I would need to include more than 100 if statements like this...plus the save function is pretty clunky. Open to any other suggestions on data storage.
Thanks
1 comentario
dpb
el 6 de Jun. de 2018
Do you need the large vector of values to do the calculation or is it the result of the calculations (I think I gather)?
If it is the latter, look at the example of using matfile at Growing an array. Also read on the various ML tools for large data Large-files-and-big-data to get an overview of facilities that exist and see what seems to fit most naturally to your problem.
Respuestas (1)
Edric Ellis
el 7 de Jun. de 2018
What I think you should do is something like the following:
% Choose a directory to store the files
outDir = '/tmp/tall_eg';
% Counter indicating which file we'll save to next
fileIdx = 1;
% How many rows of data to save in each file
rowsPerFile = 100;
% How many rows have been written so far
rowsWritten = 0;
% How many rows to write in total
totalRows = 10010;
while rowsWritten < totalRows
% Choose how many rows to write to this file
rowsThisTime = min(rowsPerFile, totalRows - rowsWritten);
% Build the rows
data = rand(rowsThisTime, 1);
% Choose a file name - ensure these progress in order
fname = fullfile(outDir, sprintf('data_%05d.mat', fileIdx));
% Save the data and increment counters
save(fname, 'data');
fileIdx = 1 + fileIdx;
rowsWritten = rowsThisTime + rowsWritten;
end
% Read the data back in as a tall array. First create a datastore ...
ds = fileDatastore(fullfile(outDir, '*.mat'), ...
'ReadFcn', @(fname) getfield(load(fname), 'data'), ...
'UniformRead', true);
% ... and then a tall array
tdata = tall(ds)
Note the 'ReadFcn' argument to the fileDatastore is a little tricky - it loads the file and then simply extracts the 'data' field and returns that. 'UniformRead' is required to ensure that we get a tall numeric vector rather than a tall cell array.
2 comentarios
Margarita Martínez Coves
el 3 de Abr. de 2020
Thank you so so so much! I have a bunch of .mat files of 1,2 GB (in-memory) size, so they don't fit in memory at the same time. I've just needed the last two lines of code, but they were very helpful and able to create a DataStore from them. Then I've run:
write(foldername, tdata)
I was able to save the new datastore as TallDataStore. Of course, this duplicates all the data, but it's worth it since Matlab resizes the blocksize to read small blocks of data each time you use read.
Again, thank you so much for helping people in this community
Ver también
Categorías
Más información sobre Large Files and Big Data en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!