How can I speed up my code to divide cell array

Hi everyone.

I have my cell array name "Finalized_LP_2016" with 1X100 size. Moreover, it contains 366X28 table in each cell. The last column of each cell array (:,28) name "WorkingDay" contain two variables with cell type are '1' and '2' to represent working day and non-working day,respectively.

I try to divide my cell arrays into two new cell array: LP_Work and LP_nonWork that they contain working day and non-working day,respectively.by using this program

if true
clc
clear all
% % % %Read matfile from Load profile database location
load('D:\WhImp2016Excel100Cus\Hourly100cusOver2016_Database\Finalized_LP_2016')
% % Divide data into working and non-working day
[row,col]=size(Finalized_LP_2016);
[row_1,col_1]=size(Finalized_LP_2016{1,1});
for b1=1:col
  for b2=1:row_1
      for b3=1:col_1
           if eq(cell2mat(Finalized_LP_2016{b1}.WorkingDay(b2)),1)
              LP_Work{b1}{b2,b3}= Finalized_LP_2016{b1}(b2,:);
           else
              LP_Work{b1}{b2,b3}={};
           end
      end
  end
end
for b1=1:col
  for b2=1:row_1
      for b3=1:col_1
           if eq(cell2mat(Finalized_LP_2016{b1}.WorkingDay(b2)),2)
              LP_nonWork{b1}{b2,b3}= Finalized_LP_2016{b1}(b2,:);
           else
              LP_nonWork{b1}{b2,b3}={};
           end
      end
  end
end

I face many problem by running this code

1. It takes time to perform this task.

2. It create very large file output when I try to export it in form of file.mat

My question is: How can I improve or fix my program to speed up my program running time to accomplish my target ?

Thank you in advance and help me please T_T

[Stephen23]

"How can I improve or fix my program to speed up my program running time..."

Don't waste time splitting data up. Keep your data together as much as possible, because this makes processing it simpler and usually much more efficient.

[Adam]

As with any question of speed up work

doc profile

is always the best start point. It is simple and easy to use and even if other people give good advice you can still learn a lot about your own program and where it is slow before applying people's suggestions to it.

Answer 1

[Jan]

1. Omit the darn "clear all". It deletes all functions from the RAM. Reloading them from the slow disk wastes a lot of time and has no benefit.

2. eq(cell2mat(Finalized_LP_2016{b1}.WorkingDay(b2)),1) is faster without cell2mat:

if Finalized_LP_2016{b1}.WorkingDay{b2} == 1

3. Is LP_Work{b1} pre-allocated? If not, insert this:

for b1=1:col
  LP_Work{b1} = cell(row_1, col_1);
  for b2=1:row_1
    for b3=1:col_1
       if if Finalized_LP_2016{b1}.WorkingDay{b2} == 1
          LP_Work{b1}{b2,b3}= Finalized_LP_2016{b1}(b2,:);
          % Now the ELSE part is solved by the pre-allocation already.
       end
    end
  end
end

4. Cell arrays are useful, if the cell elements have different dimensions or type. For all other cases, multi-dimensional arrays are much more efficient.: They use less memory, have less overhead for addressing and are processed faster in consequence.

Answer 2

[Guillaume]

I would actually recommend that you don't split your tables into smaller tables. The best answer is probably to advise you to merge all your existing tables into just one table instead. In all likelyhood whatever processing you want to do to each table will be must easier if you just have one table. It is very easy in matlab to compute statistics / apply some calculation to groups of rows using some filtering criteria such as working/non-working day, as long as your data is in just one table. See for example splitapply and rowfun with the 'GroupBy' option.

Also, from your code and description it would appear that the WorkingDay column your tables is a cell array yet only contains the scalar value 1 or 2. It would make your life much easier if that column was a vector instead of a cell array. Even better, would be to make that column a categorical array. Your test would then be simply

if Finalized_LP_2016{b1}.WorkingDay(b2) == 'working_day'

Now, if you really want to split your tables into even more tables (again, not recommended!), then first define a function that would split just one table into two:

function [working, nonworking] = splitbyworkday(t)
   working = t(cell2mat(t.WorkingDay) == 1, :);
   nonworking = t(cell2mat(t.WorkingDay) == 2, :);
end

The splitting code is then simply:

[LP_Work, LP_onWork] = cellfun(@splitbyworkday, Finalized_LP_2016, 'UniformOutput', false);

And that's it! But once again, you're most likely better off not doing any of that.