Problem using parfor for reading variable sized chunks of data into a larger pre-allocated container
2 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
Oyvind Heg
el 21 de Oct. de 2020
Comentada: Oyvind Heg
el 27 de Oct. de 2020
Hi,
I have an problem where I have pre-allocated a large matrix or vector, a, and where I will read data blocks from a large number of files that will be inserted at the right indexes in a. The different files, and resulting blocks, will typically have different size.
A simple example:
a = zeros(15,1); % pre-allocated vector
b = [1,10;11,15]; % each row contains the start to stop index for each block
parfor i = 1:size(b,1)
a(b(i,1):b(i,2),:) = i*ones((b(i,2)-b(i,1))+1,1);
end
With 'for' instead of 'parfor' it works as intended.
I have been reading several sites like https://www.mathworks.com/help/parallel-computing/troubleshoot-variables-in-parfor-loops.html and https://www.mathworks.com/matlabcentral/answers/126515-usage-of-sturctures-inside-parfor but I cannot wrap my head around my problem above.
Any tips or solutions which don't reduce the performance I am trying to obtain by using parfor in the first place?
Thanks,
Oyvind
0 comentarios
Respuesta aceptada
Edric Ellis
el 21 de Oct. de 2020
Editada: Edric Ellis
el 21 de Oct. de 2020
There's no simple way to do this without at least some duplication of data. With some duplication of data, you could do something simple like this:
aCell = cell(1, size(b,1));
parfor i = 1:size(b,1)
aCell{i} = <stuff>; % return each block in its entirety
end
a = vertcat(aCell{:}); % concatenate all cell entries into the final result
If that is not sufficiently performant, you could consider using parfeval to give you a little more control, but this is more difficult to code, and may not actually save you much. Here's an untested sketch though:
a = zeros(15,1);
for i = 1:size(b,1)
fut(i) = parfeval(@doStuff, 1, b(i,1), b(i,2)); % invoke doStuff(b(i,1),b(i,2))
end
for i = 1:size(b,1)
[idx, result] = fetchNext(fut); % collect the next result
% (note that 'idx' tells you the index into 'fut' that just
% completed)
a(b(idx,1):b(idx,2),:) = result; % push the result into 'a'
end
4 comentarios
Más respuestas (0)
Ver también
Categorías
Más información sobre Parallel for-Loops (parfor) en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!