OUT OF MEMORY problem

Dear Matlab community,

I am trying to run a very big linear continuos optimization problem (inequality constraints matrix is 674520 x 586979 and equality matrix is 43802 x 674520, both of them very sparse) and I am keep receiving the message:

Out of memory. Type HELP MEMORY for your options.

Error in lipsol>getpu (line 1475)
   P = A * spdiags(vmid,0,n,n) * A';

Error in lipsol (line 694)
    [P,U] = getpu(A,ones(n,1),Dense_cols_exist,idense,ispars);

Error in linprog (line 278)
    [x,fval,lambda,exitflag,output] =
    lipsol(f,A,B,Aeq,Beq,lb,ub,options,defaultopt,computeLambda);

Error in calling_linear_optimization (line 6)
[x,fval,exitflag,output,lambda] = linprog(OBJ_goal,A,b,A_equality,beq,lb,up);

When I type in MEMORY I got this:

memory Maximum possible array: 6734 MB (7.061e+09 bytes) *

Memory available for all arrays: 6734 MB (7.061e+09 bytes) *

Memory used by MATLAB: 2301 MB (2.413e+09 bytes)

Physical Memory (RAM): 8083 MB (8.475e+09 bytes)

Limited by System Memory (physical + swap file) available.

As it seems I have enough memory (my computer has 7.89 GB of usable RAM, 64-bit version of both windows and Matlab), could you help me with the message if something else is actually wrong? I'm using sparse matrices for both inequality and equality constrinats (A and Aeq) and both of them are less than 50 mb in size.

I would appreciate any comment!

Best regards,

Dominik

[Jan]

Did you posts the examples of Matlab's help section? Or are these the values of your machine? If you have a 64 bit Matlab, thet output for "memory command on a 32-bit Windows system" looks suspicious. Did you type the memory command in the command window?

[Dominik Dominkovic]

Hey Jan,

Thank you for your answer. You are right, those were just the examples and sorry for that.

This is what I get when I type in memory:

memory
Maximum possible array:      6734 MB (7.061e+09 bytes) *

Memory available for all arrays:      6734 MB (7.061e+09 bytes) *

Memory used by MATLAB:      2301 MB (2.413e+09 bytes)

Physical Memory (RAM):      8083 MB (8.475e+09 bytes)

*  Limited by System Memory (physical + swap file) available.

It seems that I have enough RAM so I don't know why I keep getting this message?

Best regards

Dominik

Answer 1

[Jan]

The strategies to solve this problem are always the same, as you can examine by a search in this forum:

• Install more RAM
• Run 64 bit versions (as you do already)
• Install more RAM
• Free memory by stopping other programs
• Install more RAM
• Use smaller data types, if possible: A single array uses the half of memory
• Create structs/cells of arrays instead og arrays of structs/cells
• Install more RAM
• Check if you accidently create huge arrays e.g. by rand(1e6) if rand(1, 1e6) was meant
• Check if any matrix became full by accident, when a sparse way of storage is required.
• Install even more RAM

[Walter Roberson]

• Install an SSD and configure the machine to swap to it. This will not be as fast as more RAM but it would be faster than a hard disk (provided you get a good fast SSD and appropriate interface.)

[Jan]

And install more RAM.

[Dominik Dominkovic]

But isn't it clearly written that Matlab uses 2301 mb only, while I have available memory for all arrays of 6734 mb? How could I see then the real amount of memory being used by Matlab?

Why I still get the out of memory statement?

[Walter Roberson]

MATLAB was using that much at the time you asked. Each function has its own workspace that allocates memory and the memory gets returned to the pool when the function returns. By the time that you got to

   P = A * spdiags(vmid,0,n,n) * A';

you did not have enough memory to do the calculation, but when the routine returned because of the error, all of the temporary memory was given back.

[Dominik Dominkovic]

Dear Walter,

Thank you a lot for your answer! Now it's getting me clearer.

Do you know if there is any possibility to track the amount of temporary memory my model is using (let's say a 'peak' memory demand during the code run)? Or to estimate it somehow? I need at least some harsh estimation about the RAM memory I need to have for model being able to run...