How can I reduce the execution time when calculating mean values within a 3 times loop?

Deal all, good morning!

I would like to ask your help regarding to a computational problem that I have. I'm trying to calculate 10 minutes values on a daily basis. So my original file is something like this: [year DoY time(minutes) value]

2010 132 150 1.52

2010 132 151 2.5

2010 132 153 3.4

...

2013 365 1440 0.2

So I have to calculate the ten minute mean values for each year and each day of the year. I created a code with with 3 loops in it. The first one for the year, the second one for the day and the third one for the minutes. It looks more or less like this: for y=2010:2013 idy=find(year==y) if ~isempty(idy) for d=1:366 idd=find(doy(idy)==d) if ~isempty(idd) for m=1:10:1440 idm=find(m(idd(idy))==m if ~isempty(idm) mean=[mean; y d m mean(value(idy(idd(idm))))]; end end end end end end

Now, where is the problem?!

It takes days to run and at the end matlab stucks so I have to exit and I can never get to the results...

Any sugestions would be really really APRECIATED!!!

Thank you in advance.

Cheers,

Melina

[Melina Maria]

Sorry, the code looks like ...

for year=2004:2014

    idy=[];
    idy=find(NILU.y==year);
    if ~isempty(idy)
for day=1:366
    iddoy=[];
    iddoy=find(NILU.doy(idy)==day);
    if ~isempty(iddoy)
for lo=1:10:1440
    idmi=[];
    idmi=find(NILU.dt(idy(iddoy))>=lo & NILU.dt(idy(iddoy))<lo+10);
    if~isempty(idmi)
    TENNILU=[TENNILU;year day mean(NILU.dt(idy(iddoy(idmi)))) mean(NILU.sza(idy(iddoy(idmi)))) mean(NILU.tot(idy(iddoy(idmi)))) mean(NILU.par(idy(iddoy(idmi)))) mean(NILU.alg(idy(iddoy(idmi)))) mean(NILU.flmine(idy(iddoy(idmi)))) length(idy(iddoy(idmi)))];
    end
end
    end
end
    end
end

Answer 1

[Titus Edelhofer]

Hi,

I would suggest to preallocate TENNILU and take some of the operations out of the loops:

TENNILU = zeros((2014-2004+1)*366*144, 9);
counter = 0;
for year = ...
  for day=1:366
    iddoy = find(NILU.doy(idy)==day);
    if ~isempty(iddoy)
      idydoy = idy(iddoy);
      for lo=1:10:1440
        idmi=find(NILU.dt(idydoy))>=lo & NILU.dt(idydoy))<lo+10);
        if ~isempty(idmi)
          counter = counter + 1;
          id = idy(iddoy(idmi);
          TENNILU(counter,:) = [year day mean(NILU.dt(id)) mean(NILU.sza(id) ...];
        end
     ...
end

% delete the empty rows at the end
TENNILU(counter+1:end,:) = [];

Does this help to reduce the time?

Titus

[Melina Maria]

Thank you Titus!

I've got your point and I'm gonna try it!

Melina.

P.S. Would it be convinient to set TENnilu=[ ] at the beginning, before the first loop instead of setting TENNILU = zeros((2014-2004+1)*366*144, 9)?

Answer 2

[amanita]

You can also use a parfor somewhere (if it is possible) but you will have to change your code a little bit.

[Melina Maria]

I'm aware of the existence of parfor, but I have never used it before and I don't really know the exact structure of the code that I should use.

Thank you.

[amanita]

You need independent jobs, so if you use parfor in your outer loop you will need to create N TENNILU's, where N is the total number of years (e.g you can create a cell for TENNILU, preallocate TENNILU{year} after the parfor year and set your counter to zero). If you use parfor you will also need to create a parallel pool (parpool)

[Melina Maria]

Thank you!

Answer 3

[Andrei Bobrov]

one way

d = [2010 132 150 1.52
2010 132 151 2.5
2010 132 153 3.4
2012 20 289 4.69
2013 365 1440 0.2];

sdate = addtodate(datenum(d(:,1),1,1),d(:,2) + d(:,3)/1440,'day');
b = datevec(sdate(1));
c = ceil(b(end)*.1)*10;
dte = datenum([[b(1:4),floor(b(5)*.1)*10,0];[c(1:4),ceil(c(5)*.1)*10,0]]);
z = addtodate(sdate(1),(0:10:ceil(diff(dte)*144)*10)','minute');
out = [datevec(z),accumarray(ii,d(:,end),size(z),@mean)];

[Melina Maria]

Thank you Andrei.

taht was a bit complicate for me, since I'm a relatively new user. I'm going to see it extensively though!

Melina.