Data Syncrhonization - Error 'Out of Memory'

Hello,

I have data extracted from a drone and I am trying to synchronize the data so that I can have one time span for all data that I want to work with. Unfortunately, i can not share the data as it is research property. When I run the code I am facing this error: (bold text)

Error using timetable/retime

Out of memory.

Error in tryingToSynchronizeData (line 36)

timetables{i} = retime(timetables{i}, 'regular', 'linear', 'TimeStep', dt);

If anyone can help me out or suggest ideas, ill be so grateful.

Tahnk you in advance.

motor_Dt     = datetime(logs.RCOU.TimeS,'ConvertFrom','datenum', 'Format','dd-MMM-yyyy HH:mm:ss.SSSS');
attitudes_Dt = datetime(logs.ATT.TimeS,'ConvertFrom','datenum', 'Format','dd-MMM-yyyy HH:mm:ss.SSSS');
gyr_Dt       = datetime(logs.IMU.TimeS,'ConvertFrom','datenum', 'Format','dd-MMM-yyyy HH:mm:ss.SSSS');

tt_imu = timetable(...
    gyr_Dt, ...
    logs.IMU.AccX, ...
    logs.IMU.AccY, ...
    logs.IMU.AccZ, ...
    logs.IMU.GyrX, ...
    logs.IMU.GyrY, ...
    logs.IMU.GyrZ, ...
    'VariableNames', {'AccX_mDs2',  'AccY_mDs2', 'AccZ_mDs2', 'p_radDs', 'q_radDs', 'r_radDs'});

tt_motor = timetable( ...
    motor_Dt, ...
    (logs.RCOU.C1),...
    (logs.RCOU.C2),...
    (logs.RCOU.C3),...
    (logs.RCOU.C4),...
    (logs.RCOU.C5),...
    (logs.RCOU.C6),...
    'VariableNames',{'M1', 'M2', 'M3', 'M4', 'M5', 'M6'});

tt_att = timetable(...
    attitudes_Dt,...
    logs.ATT.Roll,...
    logs.ATT.Pitch,...
    logs.ATT.Yaw,...
    'VariableNames', {'Roll_deg','Pitch_deg','Yaw_deg'});

timetables  = {tt_imu, tt_motor, tt_att};
dt = seconds(0.01);
for i = 1:numel(timetables)
    timetables{i} = sortrows(timetables{i});
    timetables{i} = retime(timetables{i}, 'regular', 'linear', 'TimeStep', dt);
 end

% Synchronize the timetables
tt_ss = synchronize(timetables{:});

[Siddharth Bhutiya]

The error seems to suggest you are running out of memory. What are the sizes of these timetables and how much memory do you have on you machine?

Also if you are going to synchronize these timetables you probably dont need the retime inside your for loop.

Answer 1

[Shreshth]

Hello Hussein,

I am able to understand that you are encountering an ‘Out of memory’ error when attempting to synchronize data using MATLAB's `retime` function. This can happen if the datasets are very large or if the time resolution you're trying to retime to (0.01 seconds in your case) creates an excessively large number of rows in the timetable.

Since providing the exact code was not possible for you, I can suggest you a few ways in which the memory issue can be resolved.

1. Increase System Memory: If possible, run your script on a machine with more RAM. This might not be an immediate solution, but it could be necessary if your data is very large.
2. Optimize Data Types: Ensure that the data types used in your tables are as memory-efficient as possible. For example, if you're using double precision for data that doesn't require it, consider using single precision or integer types if appropriate.
3. Reduce the Time Range: If you're working with a large time range, consider reducing it to the most relevant period for your analysis to decrease the size of the timetables.
4. Increase the TimeStep: Instead of using a timestep of 0.01 seconds, you could increase this to a larger value to reduce the number of rows that are created during the retime operation.
5. Process in Chunks: If it's not essential to have the entire dataset in memory at once, you could process the data in smaller chunks, retiming each chunk individually and then saving the results to disk before moving on to the next chunk.
6. Use Efficient Functions: MATLAB's retime function can be memory-intensive. You might consider using alternative methods for resampling your data that are more memory-efficient. This could involve writing custom code to handle the resampling in a way that minimizes memory usage.
7. Clear Unnecessary Variables: Make sure to clear variables that are no longer needed from the workspace to free up memory.
8. Use Tall Arrays: If you're working with MATLAB's newer versions, consider using tall arrays, which allow you to work with data that doesn't fit into memory. They're designed for big data processing and can be used with timetables.

To know more about data types and memory allocation , you can take a look at MathWorks documentation on it and decide on data type decisions.

https://www.mathworks.com/help/matlab/matlab_prog/memory-allocation.html

If none of these solutions work, you might need to look into more details about your data and the specific requirements of your analysis to find a workaround that fits within your system's memory constraints.

Thank you,

Shubham Shreshth.