How to extract multiple matrices from a big set (Large Matrix) of data?

Hello Matlab,

I need some help extracting some matrices from a bigger data set. My large 4D-matrix/ dataset has the following dimension: (5*5*14680*30). That's 30 sets of 14680 of five by five matrices. I want to extract multiple (5*5) matrices from all the 30 sets. The 5*5 matrix selection needs to be done based on the matrix location/Order, they are stacked up from 1 through 14680. For example, I want to extract the (5*5)matrices number 500, 550, 700, 755,& 793 out of the 14680 Matrices and in each set (30 sets total).

I was thinking a loop and an "if" statement may do the job because I need to extract 580 matrices then 1170 of them, and I know exactly where they are located so I have a vector (580,1) containing the exact location for the matrices I need. I would greatly appreciate some feedback. Thanks

[Walter Roberson]

A loop might be best in this situation. Although there are ways you could pull out the groups without using a loop, the methods would involve large temporary arrays of indices, and would not necessarily be any faster.

[Amine Ben Ayara]

Hello, Mr. Roberson Thank you for replying. Would you recommend using an "if" statement in the loop? For example, if the 5*5 matrix is located at position/ i=570 then select it, and so on? logically I understand what I have to do, but code wise, I'm unsure. is there a function to extract a 5*5 matrix from the large data set based on the location/indices?

[per isakson]

I'm missing something. "I need to extract 580 matrices then 1170 of them," These two numbers, where do they come from?

M = 1:(5*5*14680*30);
M = reshape( M, 5,5,14680,30);
for kk = 1 : 30
    for  jj = [500,550,700,755,793]
        m5x5 = M(:,:,jj,kk);
    end
end

Yes, m5x5 is overwritten 30*5-1 times.

[Amine Ben Ayara]

Hello & thanks for the reply.

580 & 1170 are the total number of the 2 sets of (5*5) matrices that I need to extract out of the 14680(that’s the total of 5*5)since The large data set/matrix is in 4D: 5*5*14680*30. based on the example you worked above where JJ=[500,550,700,755,783], I’m gonna have 580 diffident identifiers there. Then another 1179, so that’s what I’m having trouble with. How to extract specific matrices based on their location/indice from the 14680(The 5*5 are stacked up in the 14680 matrix).

I was thinking to create a vector column(580*1) for the first set and do some sort if statement to where the code verifies the location of each (5*5) matrix based on that vector then if it is there, it gets extracted. I believe this will need a loop too.

[per isakson]

"extract specific matrices based on their location/indice" Isn't that straight forward?

>> M(:,:,500,1)
ans =
       12476       12481       12486       12491       12496
       12477       12482       12487       12492       12497
       12478       12483       12488       12493       12498
       12479       12484       12489       12494       12499
       12480       12485       12490       12495       12500

"create a vector column(580*1)" row is better if you want to use a for-loop.

"where the code verifies the location" I don't get this

[Amine Ben Ayara]

Sorry for not wording the question correctly. So here is the what I have: 30 sets of 14680 (5*5) matrices (The Large 4D Matrix 5*5*14680*30).let's say that I need to extract multiple 5*5 matrices, where they are stacked up from 1 through 14680 matrices in each set. Since I know where each 5*5 matrix I need is located, I was thinking this can be done with a loop (from i=1:14680) in each set, check the location of the 5*5 matrix, if it is the one I'm looking for (exp matrix number 500) then extract it that 5*5 and store it aside in a new data set. Do the same process in each set (out of the 30 sets) until I crossed check all the locations of the 5*5 matrices that I am looking for.

.

I attached a screenshot of my workspace, where "RowID" is the row vector (508*1) containing the specific location for each 5*5 matrix that I need to extract from The 4D Matrix "MNP130" (5*5*14680 *30).

Answer 1

[per isakson]

Use this code to study for-loop and indexing. Put a break-point at 17;

MNP130 = 1:(5*5*14680*30);  % a 1D double vector
MNP130 = reshape( MNP130, 5,5,14680,30);  % reshape to 4D
for kk = 1 : 30  % loop over all "sets"
    for  jj = reshape( RowID, 1,[] )  % reshape to row vector
        m5x5 = MNP130 (:,:,jj,kk);
        17; % set break-point here
    end
end

Yes, m5x5 is overwritten many times.

.

"done with a loop (from i=1:14680) in each set, check the location of the 5*5 matrix"

RowID = [ 500; 503 ];
for  jj = reshape( RowID, 1,[] )  % reshape to row vector
    m5x5 = MNP130 (:,:,jj,kk);
end

does the same job as

for  jj = 1 : 14680
    if ismember( jj, RowID )
        m5x5 = MNP130 (:,:,jj,kk);
    end
end

with kk from outer loop

.

"extract it that 5*5 and store it aside in a new data set"

"a new data set" in what type of variable, speed and memory speaks for double array. However, 4D may be a bit tricky to get ones head around.

new_data_set = squeeze( MNP130 (:,:,500,:) );
>> whos new_data_set
  Name              Size              Bytes  Class     Attributes
  new_data_set      5x5x30             6000  double

I believe this one-liner is the answer to your question. If so, maybe there is no need to calculate new_data_set in a separate step. MNP130 is not that big, only 88 MB, and the execution time of the one-liner is really short.

.

btw: A matrix is 2D, see Matrix (mathematics)

[Amine Ben Ayara]

Good morning Isakson, Thank you so much for your valuable input, again. I tested this :

for kk=1:30
    for  jj = 1 : 14680
        if ismember( jj, RowID )
            m5x5 = MNP130 (:,:,jj,kk);
        end
    end
end

the output was one 5*5 matrix.

I was supposed to get 580 of 5*5 matrices selected from each set (1 through 30). The outer loop kk=1:30 seems perfect because it checks in each block of 14680 of 5*5 matrices, then the inner loop jj=1:14680 seems correct as well cause it checks through all the 14680 (5*5) matrices). I am assuming that the "if ismember(jj,RowID)" is checking if the (5*5) matrix is located at a specific location matching the elements from the row vector "RowID" correct? and if so, then the matrix-matched is stored in m5*5. Now because the RowID vector has 580 elements, I am expecting to get 580 matrices as final output in each block (1 through 30). The final output should be in the following dimension (5*5*580*30), that's 30 blocks/sets of 580 (5*5) matrices. Please excuse my primitive Matlab language, I am merely a beginner. I would really appreciate any feedback from you!

Kind Regards, Amine.

[per isakson]

• "the output was one 5*5 matrix." as it is expected to be, since m5x5 is overwritten in every iteration of the loop. I've pointed that out twice.
• You didn't read my answer carefully enough! I wrote: Use this code to study for-loop and indexing. Put a break-point at 17

One very useful thing with dynamic languages, Matlab is one, is that the user may execute the code line by line and inspect intermediate values of all the variables. This way the user can learn how the code works.

• First, study Debug a MATLAB Program and Examine Values While Debugging
• Then, return to my answer

[Amine Ben Ayara]

Hello Isa, Thank you for your reply. I did read on Debug and Examining Values as you suggested. I do understand that 17 was used as a breakpoint. I also had an idea already that a given matrix is 2D. I believe that what I lacked was a better explanation of my expected/desired output. Since we are starting with the large data set-4D matrix; (5*5*14680*30), again for the sake of simplicity, at least for me, I think of this as 30 blocks of 14680 of horizontally stacked 5 by 5 matrices. These 5by5 matrices are in order, from 1 to 14680. Now I have a set(column Vector) that has 580 elements (they indicate a specific position of each 5by5 matrix that I need to extract out the 14680 matrices in each block (through 30). The reason I believe that using loop is necessary because, logically, I will need to go through the 14680 of the 5*5 matrices and check where each is located, if it is in a position(row number)==an element from my column vector(580*1) then I extract that 5*5 matrix; this can be done as for i=1:14680,....end; Then an outer loop is necessary so I can repeat the process for the 30 blocks; hence for j=1:30.....do.... end; Each time my iteration works, I find a 5*5 martrix and store it so for the 30 blocks, I will get an output like this (5*5*580*30). Note the output is also 4D. Please let me know if I explained this better this time and I will be extremely thankful for any feedback!Again, I've never taken any coding courses so I'm doing my best to learn.

[per isakson]

• "I did read on Debug and Examining Values as you suggested." Ok, but I suspect you didn't try to step through the code as I proposed. That's because you wrote "I tested this: [...] the output was one 5*5 matrix."
• "I believe that what I lacked was a better explanation of my expected/desired output" Yes, indeed. The question is how you plan to use those new_data_sets. However, I think it's better to stick with a 4D-array.
• "The reason I believe that using loop is necessary [...]" Neither a loop nor if-statements are necessary. MNP130(:,:,RowID,:) creates the "(5*5*580*30)" array.
• "never taken any coding courses" I guess you share that with many Matlab users. I'm one of them. An Algol course half a century ago doesn't count. The MathWorks targets engineers and scientists, who don't want to waste time on computer science. See MATLAB Fundamentals

Make experiment with small data samples. Here is a functions that creates an appropriate sample-array.

>> M = sample4D;
>> subset = M(:,:,[3,4],:);
>> subset
subset(:,:,1,1) =
    13    13
    13    13
subset(:,:,2,1) =
    14    14
    14    14
subset(:,:,1,2) =
    23    23
    23    23
...

where

function  M = sample4D
    M = nan(2,2,5,3);
    for kk = 1 : 3
        for jj = 1 : 5
            M(:,:,jj,kk) = (kk*10+jj)*ones(2,2);
        end
    end
end

The indices, [3,4], of M are [1,2] of subset. To me that's a reason to stick with M

[Amine Ben Ayara]

Good morning Isa, Again, I sincerely thank you for taking the time to help with this. I am going to take your advice and go through the sample code you provided and try to understand and replicate it. Kind Regards, Amine