very large matrix manipulation

Hi I have a matrix that could end up being possibly 25x25 large. I need to do large manipulations on this matrix, for instance I need to generate a matrix of zeros B = zeros(n^m,m); where n and m are the dimensions of the original matrix. This obviously gives me an error because there is not enough computing power to do this. If I understand it right, any value in the matrix is a byte so that would be 25^25 bytes right? thats about 8*10^24 gb?! Whatever - Its too big. Can I use multiple cores to deal with the issue? Please can someone suggest how to get around this

Answer 1

[Walter Roberson]

In B = zeros(n^m,m) each value in the matrix will be a double precision number, which is 8 bytes.

You can use B = zeros(n^m,m,'uint8') to get one byte per entry.

Remember too that you have n^m by m, so that would be 25^25 * 25 = 25^26 = 2220446049250313080847263336181640625 bytes.

Using multiple cores is not going to help without a fundamental reorganization of your problem. 25^26 bytes requires a 121 bit address space, but the maximum implemented for any x86 type chips (the only thing MATLAB runs on these days) is 48 bits of address space. Even if you managed to partition the problem up in to 48 bits of address space per instance, you would need approximately 2^(121-48) = 2^73 = 9444732965739290427392 systems (that is close to 1E22)

You need a redesign.

[Alex]

Thanks Walter. If I show you the code, could you suggest an alternative way of doing it. I have a matrix of data, and I need to do a calculation on every combination of the data in the matrix - My analogy is like a number padlock - if you go through every combination you would eventually get the lock open. I am wanting to sum the inverses of the numbers within each combination, and keep the one that is the smallest

[code]

[m,n] = size(entry);

AT = entry.';

B = zeros(n^m,m);

B(1:n,m) = AT(:,m);%B is matrix of all combinations

p = n;

for k = m-1:-1:1

[I,J] = meshgrid(1:n,1:p);

p = p*n;

B(1:p,k:m) = [AT(I,k),B(J,k+1:m)];

end

value = zeros(length(B(:,1)),1);

for i=1:length(B(:,1))

for j = 1:length(B(1,:))

value(i) = value(i) + B(i,j)^(-1);

end

end

[c,ind] = min(value(:))

[/code]

I want c and the values that were used to make the combination

Thanks

Answer 2

[Jill Reese]

If you have access to Parallel Computing Toolbox and MATLAB Distributed Computing Server, you may find that distributed arrays will help you to work around the memory limitation. That is exactly what they are intended for.

[Alex]

Great thanks Jill, I was actually reading about Parallel Computing, but I couldnt work out how to apply it to my problem - have you got any examples, or know of any?