How to use parfor with a range from a very large object and cause memory problems
2 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?
This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).
Thanks!
T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);
parfor t=50:T
r = R(1:t,:,:,:); % Broadcast warning here
% Do some stuff with r
c = C(1:t,:);
for s=1:S
for p=1:P
rr = squeeze(r(:,s,p,:));
tmpResults = sum(c.*rr,2);
resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
resultsSD(t,s,p) = std(tmpResults);
end
end
end
2 comentarios
Matt J
el 18 de Nov. de 2021
The solution would depend on the "Do some stuff with r" part of the code.
Respuestas (1)
Raymond Norris
el 18 de Nov. de 2021
Have you considered assigning R in the parfor?
parfor t = 1:T
R = rand(T,S,P,M)
...
end
It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.
Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.
Ver también
Categorías
Más información sobre Parallel for-Loops (parfor) en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!