arrays

I ve got the following issue again with unequal arrays (unequal days and unequal time intervals within the days) and want to add units lets say every minute for every day over many different days, the issue is with keeping unreported times within the day to maintain equal time intervals

    yyyymmdd           hhmmss      unit

    20120103             93501    0

    20120103             93503    0

    20120103             93504    3500

    20120103             93506    0

    20120103             93507    0

    20120103             93560    100

    20120103             93600    0

    20120103             93602    300

    20120103             93603    0

    20120103             93604    200

    20120103             93659    0

    20120103             93801 100

    Output for every day for every minute during the day

    20120103          93500 3600

    20120103          93600  500

    20120103          93700   0 (needed to keep equal time intervals)

    20120103          93800  100

any suggestions will help.

Answer 1

[Andrei Bobrov]

M = dlmread('yourdata.txt');
M(:,2) = fix(M(:,2)*.01)*100;
m = 10^max(ceil(log10(M(:,2))));
k = M(:,1:2)*[m;1];
[a n n] = unique(k);
k1 = setdiff(min(k):100:max(k),k);
k2 = sortrows([a accumarray(n,M(:,3));k1 zeros(numel(k1),1)],1);
k3 = k2(:,1)/m;
out = [fix(k3) rem(k3,1)*m k2(:,2)]

ADD

using without keeping equal time intervals

M = dlmread('yourdata.txt');
m = 10^max(ceil(log10(M(:,2))));
M(:,2) = fix(M(:,2)*.01)*100;
k = M(:,1:2)*[m;1];
[a n n] = unique(k);
out = [fix(a/m) rem(a/m,1)*m accumarray(n,M(:,3))];

[ EDIT ] variant for untervals 5 min (on last comment Tiina)

M = dlmread('yourdata.txt');
m = 10^6;
k = M(:,1:2)*[m;1];
p = datenum(num2str(k),'yyyymmddHHMMSS');
p1 = datevec(p([1,end]));
n = 5;%interval - 5 minutes
p1(:,5) = p1(:,5) - mod(p1(:,5).*[1;-1],n).*[1;-1];
p1(:,end) = 0;
p2 = datenum(p1);
k2 = (p2(1):n/1440:p2(end)).';
[bin,bin] = histc(p,k2);
kout = datestr(k2,'yyyymmddHHMMSS')-'0';
k1c = [kout(:,1:8)*10.^(7:-1:0).' kout(:,9:end)*10.^(5:-1:0).'];
out = [k1c accumarray(bin,M(:,3),size(k2))];

[Tiina]

this is getting me out of memory error at k1. Would there be anyway to define a time grid as one min and sum all in between? (also without running into memory issues 33mil rows) thx

[Andrei Bobrov]

look ADD in my answer.

[Tiina]

yea works without memory issues thx, is it possible to sum to specific number of minutes within the same code instead of one min say every 5 minutes?

[Tiina]

on variant 5 min

it works but it continues adding time and reads the time with an error , for example it goes 9:55:.. 9:60 9:65 ..... 9:95 10:00 10:05 ...

[Andrei Bobrov]

corrected

[Tiina]

there is an issue with datenum, the error is ??? Error using ==> datenum at 182

DATENUM failed.

Caused by:

Error using ==> dtstr2dtnummx

Failed on converting date string to date number.

the argument inside datenum which is reshape ... works well.

[Andrei Bobrov]

small correct in 2 row: m = 10^6;

[Tiina]

Is the error am getting (reported earlier) because am reading data as double and not text?

[Andrei Bobrov]

M must be numeric, as in the our eg

[Tiina]

M is numeric in every column but I am getting an issue reported earlier with line 4, this is the data for M copied from matlab here:

20120103 93501 0

20120103 93503 0

20120103 93504 3500

20120103 93506 0

20120103 93507 0

20120103 93560 100

20120103 93600 0

20120103 93602 300

20120103 93603 0

20120103 93604 200

20120103 93659 0

[Andrei Bobrov]

http://img690.imageshack.us/img690/567/fortiina.png

[Tiina]

I honestly do not know why it is not working here is an image from my output

http://imageshack.us/photo/my-images/15/capturejja.png/

I very much appreciate yur explanations seriously

[Andrei Bobrov]

p = datenum(num2str(k),'yyyymmddHHMMSS');

[Tiina]

still datenum error

[Andrei Bobrov]

ver of MATLAB

[Tiina]

2010b

[Andrei Bobrov]

f = num2str(M(:,1:2))-'0';

f = f(:,~all(f < 0));

f(f<0)=0;

p = datenum(cellfun(@(x)x*10.^(numel(x)-1:-1:0).',mat2cell(f,ones(size(M,1),1),[4 2 2 2 2 2])))

[Andrei Bobrov]

p = datenum(cellfun(@str2num,mat2cell(num2str(M(:,1:2)*[10^6;1]),ones(size(M,1),1),[4 2 2 2 2 2])))

[Tiina]

yes this works on the example here, little issue remaining in the real data, the accumarray is returning an error "first input SUBs must be zero ..." this is perhaps caused by the histc which produces some zeros in the real data , is there a fix around that .. I am voting for you once now althoughu deserve more thanks for your patience