How to improve with single 4D arrays when memory limit problem?

I am studying the question in the thread here . I am replacing my 2D cell array of matrices with a single 4D array but I am getting now the memory limit problem. My intuitive data structure would be hashed array tree or using internally Java in Matlab for the task.

> Error using zeros Requested 906x1808x2x1692 (36.3GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

Code where increase the values of `(a,b)` to get enough big array size

a=66; b=77; 
channels=[1 2]; 
Times=(0+5*60):1:(29*60+50);
% Let's use a single 4d array instead of 2d cell arrays of matrices 
Cm = zeros(a,b, numel(channels), numel(Times)); 
for channel=channels
    for k=1:numel(Times)
        Cm(:,:, channel, k) = rand(a,b); 
    end
end

How to improve the internal data structure there? I really would like to avoid such memory problems and/or limits. If Oracle Java is the choice, please propose how to do such a basic processing there from Matlab.

Matlab: 2016b OS Debian: 8.5 64 bit Linux kernel: 4.6 of backports Hardware: Asus Zenbook UX303UA Related: https://se.mathworks.com/matlabcentral/answers/249344-matlab-array-size-limit-error-why

Answer 1

[Neekar Mohammed]

Have you find any solution for your problem? because I'm currently facing same thing?Thanks

Answer 2

[Walter Roberson]

That code as posted does not attempt to produce a 906 x 1808 x 2 x 1692 array, but it would not take a lot of changes to make it do so.

Cm = zeros(a,b, numel(channels), numel(Times), 'single'); 

would take half of the memory compared to double precision.

Cm = rand(a, b, numel(channels), numel(Times), 'single');

would create the entire array at one time without a loop. However, I am not completely sure at the moment whether it would only create single precision numbers, or if it would try to generate a full double precision array and then convert it to single precision. Using a loop to generate a portion at a time is not unreasonable -- though I would suggest rand(a, b, numel(channels), 'single') at a time rather than rand(a,b,'single') in order to take fewer calls.

My intuitive data structure would be hashed array tree

Unfortunately the original stackoverflow postgres related question no longer exists so we do not know what was being asked at the time. What I can say is that in MATLAB, a 4D array is the most memory efficient way that store "dense" numeric values that are logically 4 dimensional. Random numbers that are uncorrelated are barely compressible (though it would in theory be possible to compress single precision rand() results losslessly at 3 bytes instead of 4.) Hash tables and cell arrays just add overhead. Sometimes techniques such as hash tables are useful, especially in sparse situations, but at the moment I do not see any reason to use them here.

[Y.Yang]

Is any way to create sparse random 4-d matrix?

[James Tursa]

You might look into this FEX submission by Matt J:

https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays

MATLAB does not support anything beyond 2D for the sparse format.