Retrieving data from createtask is very slow
4 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
I used createJob and createTask to execute a program on 16 notes, then retrived my data from each tasks. The result is very large, I can not use fetchOutputs or getAllOutputArgument.My network is a Gigabit LAN. However, I found that the speed of retrieving data is only 3M/s.It cost too much time is data transmission. This is my code:
Job=createJob(sched); set(Job,'PathDependencies',PathDependency},'FileDependencies','Reconstruction_MLOS_SMART_Parallel.m','IntegratingLOS.m','MARTUpdate.m','MARTInitialize.m'});
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
TaskObj(ii,1)=createTask(Job,@MARTUpdate,1,{VolumeSpace(:,:,PartStart:PartEnd), MFx, MFy, Xmesh, Ymesh, xmesh_jiu, ymesh_jiu, m, oriI, newI, PartStart, PartEnd, ParaSet});
end
submit(Job);
wait(Job,'finished');
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
results_t = get(TaskObj(ii,1), 'OutputArguments');
VolumeSpace(:,:,PartStart:PartEnd)=results_t{1,1};
destroy(TaskObj(ii,1));
clear results_t
end
%%clear the temporary value
clear results_t TaskObj
destroy(Job);
clear Job
The problem arised in ‘results_t = get(TaskObj(ii,1), 'OutputArguments')’. Too much time expended in this step.
I holp some one can help me. Thank you!
0 comentarios
Respuestas (1)
Edric Ellis
el 23 de En. de 2014
The time taken for fetching outputs is driven by the disk access speed, and the speed of MATLAB's own SAVE command. I ran the following test:
c = parcluster('local');
numMBytes = floor(logspace(1, 3, 10));
times = zeros(size(numMBytes));
for idx = 1:numel(numMBytes)
j = batch(c, @randi, 1, {[-10 10], numMBytes(idx), 1024^2, 'int8'});
wait(j);
tic
x = fetchOutputs(j);
times(idx) = toc;
delete(j);
end
plot(numMBytes, numMBytes ./ times);
xlabel 'Number of megabytes'
ylabel 'MB / s'
and saw on my system approximately 140 MB/s. This is R2013b on GLNXA64 using the local cluster which stores the outputs on a network disk.
Ver también
Categorías
Más información sobre MATLAB Parallel Server en Help Center y File Exchange.
Productos
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!