Identify large sudden changes in signal

Hi, I have a time series with large number of missing values. In certain period the non missing values are wrong (winter surface reflectance values due to polar darkness). I need to identify the point where the sudden change occurs and then again when it returns back to normal and change the affected values in this period to NaN. Just for illustration this is the signal when missing values are interpolated with red ellipses identifying the areas where I have the problem.

For every point in my time series this change occurs at a different time, so i am looking for suggestions how to do this automatically. Example of the original data for one point is in dat_example.mat

[Amit]

But the data you have attached has NaN values. Do you have to replace them or remove them?

[Image Analyst]

Can you attach a plot where the missing values are NOT interpolated? Wouldn't that be your original, input signal? If they ARE interpolated already (like what you say you are showing), I'm assuming that would be some attempt at fixing the data already.

Also I'm assuming you want the precipitous drop to be on the left side of the large humps only, and to ignore those on the right side, correct? And is there supposed to be one at 1500?

[Tereza Smejkalova]

I attached the interpolated plot because it shows more clearly what is the problem area but here is plot of the original data

Of course the idea is to remove the wrong values before interpolation. Best example is the jump for example at point 689 where the value appears still correct but the next non missing value 705 is already affected.

The point on 1517 is probably noise in the data but it does not belong to the affected period anymore and should be possible to fix with smoothing.

[Image Analyst]

Can you show what you'd want the interpolated curves to look like if the interpolation was ideal?

[Amit]

Do you want to remove the NaN, interpolate or find where the sudden jumps happening?

[Tereza Smejkalova]

Ok, for that I first need to explain what the data represents. The signal is mean surface reflectance from an area of one lake when the lake is water in summer the reflectance is very low (example for period of 181 - 243 (July, August)) then as the lake starts to freeze in the autumn the reflectance values increase to something around 8000 which roughly represents totally frozen lake. Then in the spring the thaw of the ice begins and the values go down again. What I am trying to do is to identify the start and end of the freeze and thaw season (basically the inflection points in the signal) however the polar darkness creates false inflection points. So basically the process would start with the removal of the affected values, then interpolation of the missing values (I do need value for every day so I cannot remove them) and then finding the inflection points representing start and end of the season. So I need to find the sudden jumps twice the first will be to only in certain time period to remove the affected values and the second is the result of my analysis so I am on thin ice here.

[Image Analyst]

So there is no reflection when there is "polar darkness" and that's why the values, which were increasing, start decreasing again from 320 to 400? But if there were no polar darkness, those values would have headed up towards 8000? So what you want is to ignore that part, and identify that the start was at index 250, and the end was at index 500? Does that capture it? If so, you don't really need to interpolate the values in between 320 and 400 (invent values for them), so much as to ignore them and just find the 250 and 500. Correct?

[afiq mohamad]

you can use 1-d haar transform