How to interpolate harmonic data series (without sin / cos equation defined)?

Hi, Community

I wanna ask the most crucial in my research and my program. I need to do this to fill data missing in my magnetic data series which re mostly harmonic sinusoidal. Lemme share my magnetic data as plot like this :

I want to interpolate the harmonic signal data missing to be fit with the original data. But i only have data series (without equation, ex. sin eqs or cos eqs) as the only input parameter. In the other equation, like polynomial equation, we can just use polyfit and polyval, but i dont think it would be good if the data series is a harmonic data like that. I already pinned my data (time series with the other variable) here and also, here is my code to read the pinned data file :

[namafile,arah]=uigetfile({'*.txt', 'Text-Files (*.txt)'},'Load Data Magnet Hasil Ekstraksi', 'Multiselect','on');
full_ekstrak = fullfile(arah, namafile);

opts_ekstrak0 = detectImportOptions(full_ekstrak, 'ReadVariableNames',false, 'Delimiter', ' ', 'Whitespace', ' ',...
        'ConsecutiveDelimitersRule', 'join', 'EmptyLineRule', 'skip', 'FileType','text');
T_raw = readtable(full_ekstrak,opts_ekstrak0);
    T_ekstrak0 = standardizeMissing(T_raw,{'0', '0.00', 'N/A'});
    data0 = T_ekstrak0(:,{'Var1','Var2','Var3','Var4','Var5','Var6','Var7','Var8', 'Var9'});
    
    date0 = data0.Var1;
    datefix0 = datetime(date0,'Format', 'yyyy-MM-dd');
    time0 = data0.Var2;
    time0.Format = 'hh:mm:ss';
    data0_deklinasi = data0.Var6; %You can use this as variable (Y Plot)
    data0_komph = data0.Var3; %You can use this as variable (Y Plot)
    data0_kompf = data0.Var4; %You can use this as variable (Y Plot)
    data0_kompfobs = data0.Var5; %You can use this as variable (Y Plot)
    data0_kompd = data0.Var6; %You can use this as variable (Y Plot)
    data0_x = data0.Var7; %You can use this as variable (Y Plot)
    data0_y = data0.Var8; %You can use this as variable (Y Plot)
    data0_z = data0.Var9; %You can use this as variable (Y Plot)
    data0_komph_calc = sqrt((data0_x.^2)+(data0_y.^2)); %You can use this as variable (Y Plot)
    data0_kompf_calc = sqrt((data0_x.^2)+(data0_y.^2)+(data0_z.^2)); %You can use this as variable (Y Plot)
    data0_declin = atan(data0_y./data0_x); %You can use this as variable (Y Plot)
    wak_ekstrak = string(datefix0) +" "+ string(time0); %You can use this as Time Series of X Plot 
    waktu0 = datetime(wak_ekstrak,'InputFormat','yyyy-MM-dd HH:mm:ss', 'Format', 'yyyy-MM-dd HH:mm:ss');
    waktu01 = datetime(wak_ekstrak,'InputFormat','yyyy-MM-dd HH:mm:ss');

So anyone, would you please help me in inding the solution? Iam so grateful, if anyone could help me in this difficult problem, Thank you very much /.\ /.\ /.\

[KSSV]

REad about fillmissing.

[Bjorn Gustavsson]

Why do you need the missing data? What are you going to use it for?

Looking at the data your idea seems suspect to me - you'll have to consider all the rapid variations that are not due to daily variations but to specific geophysical events that are not at all recurring or periodic...

[Tyann Hardyn]

I need it because i want to use it as trend, daily magnetic trend, Sir. The variable i want to calculate is solar quiet / solar regular estimation, Sir. So i need to fill the data as if the data is full filled.

[Tyann Hardyn]

@KSSV Thanks Sir, it works. Iam trying by using movmedian fillmissing... But it appear not too match as the harmonic sinusoidal original data, only like a trend.... I wonder, whats the best function for it....

Answer 1

[Bjorn Gustavsson]

You should not "inpaint" data this way and then use that to extract SQ-curves or any such averaged quantities. You have the data you have and, saddly, should settle for making do with that, properly taking into account that you have missing data. consider the case where there were a major magnetic storm with an onset 10th of April, recovering just about to 15th(?), if you now use some kind of Fourier-series fit to the current data to fill in the missing periods you would insert completely erroneous data into your data-base. To me this would be a major no-no to do.

What you might do for some very narrow uses of averaged parameters would be to do some linear (or non-linear?) combination fits of nearby observatory-data to the data you have available and then use those data to estimate the data during the time-periods where your data-series have gaps.

HTH

[Tyann Hardyn]

The data i share in this link is only a sample, Sir... I just want to know how to fit and to fill the gap of the harmonic data with best function so the fitting curve would shaped-like the original one (harmonic signal) . And yeah, i know about FFT, and in matlab, i just found the function can be done by inputting sin equation (or something like that) while in my case, i just have magnetic data series, Sir...

[Bjorn Gustavsson]

It is not very harmonic though, is it? Look at all your transient spikes.

Didn't you read my reply? The point is that this is a bad idea, you still haven't explained for what reason you want the gaps filled in. If you want to calculate trends in your data you should not introduce additional fake data, you will simply have to work with the data you have, and adapt your methods to handle gaps in the data.

Here's one method you could use to all the missing data assign infinite standard deviation such that all your consecutive tools (weighted least-square-fitting etc) will effectively ignore those time-periods and the actual values you fudge-up doesn't matter. You have the data you have.

Here's another method to extract some average B-field variation over your data:

subplot(3,1,1:2)
imagesc(reshape(Mag(:,13),[],30)') % Stored your data in Mag, plotting the last column, adapt
subplot(3,1,1+2)
plot(median(reshape(Mag(:,13),[],30)','omitnan')) % Sensible average
hold on
% Then lets plot some estimates of spread around that average.
plot(median(reshape(Mag(:,13),[],30)','omitnan')+mad(reshape(Mag(:,13),[],30)'))              
plot(median(reshape(Mag(:,13),[],30)','omitnan')-mad(reshape(Mag(:,13),[],30)'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')-std(reshape(Mag(:,13),[],30)',0,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')+std(reshape(Mag(:,13),[],30)',0,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')+std(reshape(Mag(:,13),[],30)',1,1,'omitnan'))
plot(median(reshape(Mag(:,13),[],30)','omitnan')-std(reshape(Mag(:,13),[],30)',1,1,'omitnan'))

The problem with in-painting data from this type of average is that it will bias your results using this data since it assigns this average data in gaps that should/would have other values (it would shift the slope when the gaps are shifted towards an end of the sample-period.) Surely you can find some information about how this type of problem is handled in the geomagnetic litterature.

HTH