MATLAB Answers

Writing a matfile in a parloop

1 view (last 30 days)
Angelo Cuzzola
Angelo Cuzzola on 4 May 2020
Commented: Angelo Cuzzola on 5 May 2020
I have a very simple code that works pretty well when the size of the involved objects is manageable by the memory.
%%% Variables
% Z -> matrix of dimensions (r,T)
% V -> matrix of dimensions (r,r,T)
% nanY -> matrix of dimensions (n,T)
% y -> matrix of dimensions (n,T)
nanY = isnan(y);
y(nanY) = 0;
denom = sparse(zeros(n*r,n*r));
nom = sparse(zeros(n,r));
parfor t=1:T
nanYt = sparse(diag(~nanY(:,t)))p;
denom = denom + kron(Z(:,t+1)*Z(:,t+1)'+Vsmooth(:,:,t+1),nanYt);
nom = nom + y(:,t)*Z(:,t+1)';
end
vec = denom\nom(:);
In practical application, that bunch of code has to run with the parameter n taking values around 80k leading quickly to an unfeasible memory load. The solution I figured out relies on matfiles, at the expenses of paralellizionation and general performance. This is the version with which I am able to handle it for large n.
nanY = isnan(y);
y(nanY) = 0;
denom = zeros(n*r,n*r,T);
nom = zeros(n,r,T);
save var_cyc.mat denom nom y nanY -v7.3;
v = matfile('var_cyc.mat', 'Writable', true);
for t=1:T
v.nanYt = sparse(diag(~v.nanY(:,t)))p;
v.denom(:,:,T) = kron(Z(:,t+1)*Z(:,t+1)' + V(:,:,t+1),v.nanYt);
v.nom(:,:,T) = v.y(:,t)*Z(:,t+1)';
end
vec = sum(v.denom,3)\sum(v.nom(:),3);
Unfortunately this script cannot be paralellizable because conflicts generate when workers try to write on the matfile. I'm wondering if there is a way to recover the initial parallelization structure using matlab file to handle huge file without incurring in 'out of memory errors'.
Thanks.

  0 Comments

Sign in to comment.

Answers (1)

Jason Ross
Jason Ross on 4 May 2020
Have you looked into using tall arrays?

  1 Comment

Angelo Cuzzola
Angelo Cuzzola on 5 May 2020
Yes, but nom and denom are not exactly tall. They are 60kx60k matrices

Sign in to comment.

Products


Release

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by