Custom Modifiable datastore tall array

I have a huge dataset consisting of:

• daily matrix data (NxMx1)
• auxillary parameter

Due to the massive amount, I cannot load all of the data at once. Therefore I want to use the datastore and tall arrays to make the data sort of possible to handle.

The best case would be that the matrix array get added to a 3D-array [NxMxT], with T for the daily increment. This would make the further processing much easier.

Now my questions:

• Is it possible to add data after creating tall arrays?
• Is it possible to save the auxillary parameter with the matricies.
• Can I add data "in between"? (Sometimes the data has to be recovered and must be added after new data is already append)

I looked already into custom datastores, but was not really successful. Also structs where not really helpful due to the lack of mapreduced computation.

Are there any ideas, recommendation or strong opinions which could guide me towards a solution?

Answer 1

[Hari Krishna Ravuri]

“Is it possible to add data after creating tall arrays?”

Yes, it is possible to add data after creating tall arrays. Consider the following sample code which adds data to a tall array -

>> a=[1 2;3 4]                    %Creating an Vector
 
a =
 
     1     2
     3     4
 
>> t = tall(a)                        %Creating a tall array from the in memory vector a
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 6).
 
t =
 
  2×2 tall double matrix
 
     1     2
     3     4
 
>> t=[t;[5 6]]                       %Appending the vector [5 6] after the last row in the tall array.
 
t =
 
  3×2 tall double matrix
 
     1     2
     3     4
     5     6

“Is it possible to save the auxiliary parameters with the matrices”

I’m assuming that the auxiliary parameters are stored in a vector and it is compatible with concatenating to the tall array. In this case, you can just append the auxiliary parameters to the tall array.

“Can I add data in between?”

Yes, you can add data anywhere. Consider the following example

>> t
 
t =
 
  3×2 tall double matrix
 
     1     2
     3     4
     5     6
>> t1=[t(1,:);[00 11];t(2:end,:)]
 
t1 =
 
  4×2 tall double matrix
 
     1     2
     0    11
     3     4
     5     6