Synchronizing several large timetables

Dear community,

while I'm waiting for my synchronization to finish, I'm wondering if there is any faster way perform below synchronization. I had to use tall() to avoid running out of memory.

In my example, I have around 19 timetable with timevector and corrsponding signal. I take my first timetable and put it into variable TT1. Then, I put the second table in TT2 and synchronize TT1 and TT2. The resulting table is then transferred into TT1 and I synchronize the next timetable.

While this certainly works, it will take on the current speed (see screenshot below) approx 15 hours or more. Is this a very inefficient way? Is there a way to speed this up significantly? Any obvious mistakes?

I appreciate your input (This is local parallel pool with 16 workers, Xeon E-2286 2.4 GHz, 32 GB RAM)

TT1=tall(tbl{1,1});


for i=2:sigcount
    
    TT2=tall(tbl{1,i});
    TTSync=synchronize(TT1,TT2,'union','nearest');
    TT1=TTSync;
end
location = 'E:\Matlab\Analysis';

write(location,TT1);

clearvars;

Answer 1

[Seth Furman]

Where are your timetables coming from? Are you reading them from one or more files? If so, it probably makes more sense to create each tall timetable from a datastore rather than an in-memory timetable. See the link below for an example.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

Assuming tblTall is a cell array of tall timetables, we can pass all tall timetables to synchronize at the same time as follows

TTSync = synchronize(tblTall{:,:},'union','nearest');

Please let me know if you are able to try this and, if so, whether you see better performance.

[Lutetium]

Thanks for your answer and sorry for the delay. The timetables come from an in-memory cell array containing several timetables. Your suggestion works for my in-memory cell array but not with the tall array:

"Indexing expressions of the form T{...,...} are not supported for tall arrays."

[Seth Furman]

It sounds like you are creating a tall cell array of non-tall timetables, e.g.

>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
>> tblTall = tall(tblCell)

tblTall =

  1×2 tall cell array

    {1×1 timetable}    {1×1 timetable}

>> tblTall{:,:}
Indexing expressions of the form T{...,...} are not supported for tall arrays.

Instead, what you want is to create a non-tall cell array of tall timetables, e.g.

>> tblTall = cellfun(@tall,tblCell,'UniformOutput',false)

tblTall =

  1×2 cell array

    {1×1 tall}    {1×1 tall}

>> tblTall{:,:}

ans =

  tall timetable

    Time     Var1
    _____    ____

    0 sec     1  


ans =

  tall timetable

    Time     Var1
    _____    ____

    0 sec     2  
    
>> synchronize(tblTall{:,:})

ans =

  1×2 tall timetable

    Time     Var1_1    Var1_2
    _____    ______    ______

    0 sec      1         2   

Please let me know whether you notice any improvement in performance with this workflow.

[Seth Furman]

Please also clarify, does code like the following work on your machine with your timetables or do you get errors about running out of memory?

>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
synchronize(tblCell{:})

ans =

  1×2 timetable

    Time     Var1_1    Var1_2
    _____    ______    ______

    0 sec      1         2