Program Optimisation: Out of Memory

I have a program that is on the limit of using all the available memory. I have done all I can to increase the memory limit but because the student edition is 32-bit the limit is 2Gb (correct me if I am wrong please). Below is >>memory when started up.

Maximum possible array:            2046 MB (2.146e+009 bytes) *
Memory available for all arrays:   3275 MB (3.434e+009 bytes) **
Memory used by MATLAB:              474 MB (4.974e+008 bytes)
Physical Memory (RAM):             3882 MB (4.071e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

Therefore I need to optimise my code to work within these small memory limits. The error comes from here:

??? Error using ==> times
Out of memory. Type HELP MEMORY for your options.

Error in ==> couple at 58
        k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
        idx4.*10e6 + idx5.*100e6;

The size of idx is (2.6e5, 200). Is there any way to optimise the code to allow a larger matrix to be computed. The matrix has been generated from ODE45, however, reducing the accuracy even further is undesirable.

Thank you for your assistance.

[Oleg Komarov]

I usually have 1.1 GB of maximum possible array at startup (2011a Win32 Vista) and a dataset of ~450 MB.

I had to split the dataset into a structure and saved it with -struct and -v6 option.

Then I wrote a function that loops a handle through each portion of the struct.

It takes a little to understand how to operate on parts rather than on the entire dataset, but if you think it may be helpful I will share my solution.

Also, you can save the dataset in binary format and fread it with the approariate offsets (alternatively to save with -struct option).

Answer 1

[John]

I solved the problem by making my ODE settings to record fewer time intervals:

tspan = [0 a];
tspan = 0:0.01:a;

It still has the same accuracy but holds fewer numbers.

Answer 2

[Titus Edelhofer]

Hi,

hmm, somehow this looks strange: the size of idx is (2.6e5, 1)? That means, that one of your idx vectors needs about 2MByte of memory. Doesn't sound too much to me ...???

Titus

[John]

Sorry, each idx is (2.6e5, 200).

Answer 3

[Bjorn Gustavsson]

Maybe you can cut up your problem and work with subsets of idx in sequence. This might not be "ideal" but if nothing else works it might get the job done...

[John]

OK. Can you lead me in the right direction by explaining how that may be done?

Do you mean by a series of for loops splitting up idx into sizes of, say, (2.6e5, 50), an adding the resulting [k] matrix to form the larger matrix?

[Bjorn Gustavsson]

Exactly, something like that. I would've guessed that you had a problem with 200 coupled parameters and 2.6e5 solutions for their evolution. Then I kind of hoped that you could calculate the matrix elements from invervalls in/of the first dimension:

i_start = 1:2600:(260001-2600);

i_stop = 2600:2600:260000;

for i1 = 1:length(i_start),

current_idx = idx(i_start(i1):i_stop(i1),:);

currentM = ...

M4results = merge as fit...

end

But I hope you dont need to make a (several) 2.6e5^2 matrices. That would be a lot of matrix elements to look at.

HTH

[John]

Sorry that does not work as there is not enough memory to hold all the new matrices created!

[Bjorn Gustavsson]

OK, if you can't use this subset approach in either dimension, then it looks dire. Maybe you can use memmapfile to store all your big matrices. It might be a workaround - but I haven't used it so I have no grip on if it would work or if there is other pitfalls.

[Walter Roberson]

memmapfile requires virtual memory equivalent to the size of the data stored.

Answer 4

[Walter Roberson]

The limit for the student edition is the same as the limit for the professional edition.

[John]

From what I read on the MathWorks website only 32-bit editions have the 2Gb limit and 64-bit editions have a much higher limit. Can you confirm that I miss-read this?

[Walter Roberson]

The 32 bit versions (student _or_ professional) have a limit of 32 bits of address space, which is 4 Gb total including all memory required for system interfaces.

The 64 bit versions (student _or_ professional) in theory could have a limit of 64 bits of address space, but in practice current processors use no more than 48 bits of address space. I do not know how MATLAB divides up that virtual address space. I _think_ I saw a hint that _one_ of the MATLAB library calls was effectively restricted to 40 bits worth of addressing, but I could easily be wrong about that.

Anyhow, student version and professional version are exactly the same memory limits and have been since the first version that was actually called "Student Version". The original two Springer-Verlog books that included a student matlab *did* have lower memory limits, but it was called "Student Edition". (And yes, I do continue to get "Edition" vs "Version" confused myself.)

Answer 5

[Walter Roberson]

In-place operations might help:

k = idx1.*100e6 + idx2.*10e6 + idx3.*10 +
      idx4.*10e6 + idx5.*100e6;

could become

k = idx1;
k = k + idx5;
k = k .* 10;
k = k + idx2;
k = k + idx4;
k = k .* 10e5;
k = k + idx3;
k = k .* 10;

This would not require any intermediate vectors, just the output vector that is necessary anyhow. In theory any relatively recent MATLAB should be able to handle

   A = A + B

by overwriting A as the computation goes along.

[John]

Thanks for your post. I now get the error on the first line:

??? Error using ==> times

Out of memory. Type HELP MEMORY for your options.

k = idx1.*100e6;

[Walter Roberson]

I didn't start with k = idx1.*100e6 -- I started with k = idx1 .

None the less, if your idx1 is so big that another array the same size cannot be created, then there is really not much you can do about the situation.

You could do hacks such as saving the variables to individual .mat files, and then using system() to run a matlab routine that loaded just enough at a time to take one more step, clearing the variable as soon as it was done with, and then save()'ing the result to a .mat file. What this would get you is that in the second matlab session, the _only_ thing in your workspace would be MATLAB itself, the standard I/O libraries, and the variables you were working with: the effect would be to gain virtual address space.

If you are willing to do some mex'ing, you could probably create a pipe() over to a MATLAB session to do the calculation for you. Note that pipe() might need to be coded differently between Windows and the unix-based systems; in _theory_ from Windows XP SP2 and later the code could be compatible, but I have seen too many hints that the POSIX module built in to Windows doesn't really deliver POSIX.

Answer 6

[Jan]

If the values of idx and k match into a UINT32, use this type to reduce the memory consumption to 50%:

idx1 = uint32(...);
idx2 = uint32(...); ... etc

k = idx1.* uint32(100e6) + idx2.* uint32(10e6) + idx3.*10 + tc...

This together with Walter's inplace method can reduce the needed memory efficiently.

[John]

Thank you for your reply. I tried it and it ran much faster. However, how do I multiply the k vector with another vector which is not uint32? For example:

ans = diff.* k;

k is a vector (n,1) and diff is a vector (n,1) and I wish to multiply each element only e.g. k(n-1,1)*diff*(n,1), k(n,1)*diff(n,1) ...etc.

Answer 7

[HaveF]

In spite of your algorithm, you can try this:

If you use 32bit OS, and have 4GB memory, you can check the biggest contiguous memory space which related your biggest matrix size:

feature memstats

at this section:

Largest Contiguous Free Blocks:

you will get the biggest continued memory space. Then you can calculate if this is big enough for your matrix.

If not, you can enable 3GB switch in system: <http://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240697>

After reboot your system, and type "feature memstats" in Matlab again. You will find your "Largest Contiguous Free Blocks" become larger.

[Walter Roberson]

Note: the above applies only to MS Windows, not to any other 32 bit OS.

feature memstats is, unfortunately, only available in MS Windows.

[John]

HaveF, I have tried the 3Gb switch to no avail. Thanks for the suggestion.