Why am I getting gaps in my data when I attempt to interpolate my curve?

I am trying to make finner steps (x) for my data file attached. I am using matlab's interp function. Upon doing so, I get the curve attached with a big gap in the data. Any idea what I am doing wrong?

xInterp=linspace(Dist(1),Dist(end),2500);%also tried less points (500->1000) and it was worse 
yInterp=interp1(Dist,Power,xInterp,'spline');
figure 
plot(xInterp,yInterp)
set(gca, 'YScale', 'log')

Answer 1

[John D'Errico]

As is always the case, look at what you did. Think about it. If necessary, think again.

xy = readtable('Sample Test.txt')

xy = 60x2 table
    Var1      Var2  
    ____    ________

      0     9.06e-07
    0.1     9.65e-07
    0.2     1.15e-06
    0.3     1.66e-06
    0.4     1.88e-06
    0.5     1.93e-06
    0.6     2.57e-06
    0.7     3.22e-06
    0.8      2.8e-06
    0.9     3.05e-06
      1        3e-06
    1.1      3.8e-06
    1.2     4.13e-06
    1.3     5.45e-06
    1.4     8.19e-06
    1.5     1.46e-05

plot(xy.Var1,xy.Var2,'o')

And that looks entirely like data. But is there a problem?

Sadly, you don't tell us what is Dist. So I assume Dist is the first column in your table.

Dist = xy.Var1;
Power = xy.Var2;
xInterp=linspace(Dist(1),Dist(end),2500);%also tried less points (500->1000) and it was worse

First, the NUMBER of points you interpolated at is NOT the problem. The problem lies in a combination of things you did.

1. You used a spline to interpolate.
2. Then you plotted using a log scale.

yInterp=interp1(Dist,Power,xInterp,'spline');
min(yInterp)

ans = -2.4879e-05

Does that give you a hint? What is the log of a negative number? What happens when you plot the y axis using a log scale?

plot(xInterp,yInterp)

semilogy(xInterp,yInterp)

Do you see that MATLAB did not want to plot the logs of negative numbers, since those results will be imaginary.

log(min(yInterp))

ans = -10.6015 + 3.1416i

Ok. Now, why did this happen? You need to understand that when you interpolate using a spline, if the curve has sharp transients in it, the spline will often overshoot. Look carefully at what plot shows. Do you see what look like Gibb's phenomena? Those oscillalations? Far better is to interpolate on curves like this using pchip, which is designed to not do that.

yInterp2 = interp1(Dist,Power,xInterp,'pchip');
min(yInterp2)

ans = 5.6500e-08

plot(xInterp,yInterp2)

Warning: Negative data ignored

semilogy(xInterp,yInterp2)

[SnooptheEngineer]

Thanks for the detailed reponse. So the sharp transitions caused spline to overshoot in the negative direction->yielding negative values and thus matlab ignored them when log plotting them as they were undefined.

Are there other interpolation functions or different approaches you recommend that are better than pchip? I see your other comment about this being lower order and thus less accurate in some scenarios

[John D'Errico]

Its a tradeoff. You cannot always have everything you want. The higher order behavior you get from a spline causes that ringing. Conversely, in order to gain an interpolant that has the better behavior you wish to see in pchip, that algorithm relaxes second derivative continuity. And if you look carefully, you can see second derivative jumps in a curve. At the same time, the difference is not always a problem.

Having said that, there are some alternatives, but not all of them are easily available.

For example, I recall a paper by Gregory and Delbourgo, describing a rational quadratic interpolant, that can be made both twice differentiable as well as locally monotone. So it has the properties of a spline, thus higher order, yet it will not show that ringing behavior. The downside is, my implementation is not owned by me, but by the company I worked for when I wrote an implementation. Regardless, it is not that difficult to write as I recall.

You could also use things called tension splines. Again, I wrote code for them in a different galaxy, long ago, and far, far away. Kind of a pain in the neck to use though as I recall. I much preferred the rational quadratics of Gregory and Delbourgo back then.

A third option is to use my SLM toolbox, as found on the file exchange. It can produce a spline model that will be both twice differentiable, as well as being loally monotone, so it can get rid of the ringing. You would need to download the toolbox, as well as learn to use it. Not a big problem, which I can say as the author, but then I may be biased. ;-)

None of the above are really that great solutions here for you, at least not in my opinion.

Is there yet another alternative? Perhaps. Work in a log space to perform the interpolation.

xy = readtable('Sample Test.txt');
Dist = xy.Var1;
Power = xy.Var2;

logPower = log(Power);
xInterp = linspace(Dist(1),Dist(end),2500);
yInterp = exp(interp1(Dist,logPower,xInterp,'spline'));

Do you see what I did? Because the interpolation is done on log(y), you don't care if log(y) goes negative, because then exponentiating things makes it always positive anyway. Anyway, the bumps and jiggles a spline would produce down there is not an issue, because exp(negative_junk) will be fairly well behaved anyway.

plot(xInterp,yInterp)

As you can see, there are no huge oscillations at the toe of that curve.

semilogy(xInterp,yInterp)

This trick will not always work, but it CAN be an improvement some of the time. Is it really that much better than what pchip did? I'm not terribly sure. But those are the alternatives I can think of, off the top of my head.

In the end, I'd probably just use pchip. Why? Your data has enough noise, bumps, jiggles, etc., in it, that the higher order behavior of a true spline is probably a waste of computational effort.

[SnooptheEngineer]

Hi Jon, I appologize for the delay in responding to you. Things got busy and this project slipped. Thank you for the detailed response and help. I really like your approach of switching the data to log first then interpolate. I cant have cake and eat it too and this seems like the best compromise outside of writting my own interp function :)

Answer 2

[Umar]

Hi Omar,

The 'spline' interpolation method in Matlab can sometimes introduce artifacts or gaps in the data, especially when the data points are not evenly distributed. This can lead to unexpected results in the interpolated curve. To address the gap issue in the interpolated data, we can switch to a different interpolation method that is more suitable for the given data. One alternative method that can provide a smoother interpolation without gaps is the 'pchip' method. Here is the updated code with the interpolation method changed to 'pchip':

xInterp = linspace(Dist(1), Dist(end), 2500);
yInterp = interp1(Dist, Power, xInterp, 'pchip');
figure
plot(xInterp, yInterp)
set(gca, 'YScale', 'log')

I am not sure if you have tried pchip function but it should help resolve your problem. For more information regarding this function, please refer to

https://www.mathworks.com/help/matlab/ref/pchip.html

Let me know if I can provide further assistance.

[John D'Errico]

Note this is not truly a question of the gaps in the data, or the distribution of the data. It is a question of the use of a spline to interpolate data with a rapid transition, thus a region with a very high slope to a region with virtually no slope at all.

But pchip is indeed a viable solution, because pchip is designed to handle those ituations more stably. HOWEVER, note that pchip is a somewhat lower order interpolant. It will not always be as smooth looking if you look very carefully at the result.

Answer 3

[Umar]

Hi Omar,

You mentioned the following in your post about Thanks for the detailed reponse. So the sharp transitions caused spline to overshoot in the negative direction->yielding negative values and thus matlab ignored them when log plotting them as they were undefined. Are there other interpolation functions or different approaches you recommend that are better than pchip? I see your other comment about this being lower order and thus less accurate in some scenarios

First, I would like to tell John that he did a good job to tackle the problem. Since the purpose of this platform is to share knowledge and brainstorming ideas to help out each other. My response to your comments are listed below.

When dealing with data exhibiting rapid transitions, especially those causing overshooting and negative values during interpolation, it is crucial to explore alternative interpolation techniques that can handle such complexities more effectively than PCHIP. While PCHIP is a stable option, its lower order nature may lead to less visually smooth results, prompting the search for more suitable methods.

One alternative interpolation approach worth considering is the Cubic Spline Interpolation. Cubic splines are piecewise-defined polynomials that maintain smoothness and continuity up to the second derivative at the interpolation points. This property makes them particularly useful for capturing rapid transitions while ensuring a higher degree of smoothness compared to PCHIP. By utilizing cubic splines, you can potentially mitigate issues related to overshooting and negative values, providing a more accurate representation of the data.

Another technique to explore is Akima Interpolation, which is known for its ability to handle sharp transitions effectively. Akima interpolation calculates the interpolating polynomial using a weighted average of slopes at neighboring points, allowing for a more robust interpolation process in the presence of rapid changes. This method can offer improved accuracy and stability when dealing with data sets characterized by sudden transitions and high slopes.

Furthermore, Shape-Preserving Piecewise Cubic Interpolation methods such as PCHIPEND or PCHIPLIN can be considered as alternatives to traditional PCHIP. These variations aim to enhance the smoothness and accuracy of interpolation results, especially in regions with steep gradients or rapid changes. By incorporating shape-preserving constraints, these methods can better handle the challenges posed by data exhibiting sharp transitions, ensuring more reliable interpolation outcomes.

Now, John also shared an alternative approach about quadratic interpolant and I do agree that this type of interpolant offers the benefits of being twice differentiable and locally monotone, but there are some drawbacks to be aware of:

Implementing a rational quadratic interpolant can be more complex compared to simpler interpolation methods like linear or cubic splines. The additional parameters and constraints involved in ensuring twice differentiability and local monotonicity can increase the implementation.

The computational cost of evaluating a rational quadratic interpolant may be higher than that of lower-order interpolants. The additional calculations required to maintain the desired properties can lead to increased computational overhead.

Rational quadratic interpolants can be more sensitive to perturbations in the input data compared to lower-order interpolants. Small changes in the data points can have a more significant impact on the interpolant's behavior, potentially leading to unexpected results.

While rational quadratic interpolants offer certain properties like twice differentiability and local monotonicity, they may lack the flexibility to capture complex variations in the data. Higher-order interpolants can sometimes oversmooth the data, leading to a loss of detail or accuracy in the interpolation.

Due to the nature of rational functions, numerical stability issues may arise when using rational quadratic interpolants, especially when dealing with data that contains noise or outliers. Careful consideration and numerical analysis are required to ensure stability in the interpolation process.

As Andrew Singer, professor of electrical and computer engineering once said quoted verbatim, “The art of debugging is figuring out what you really told your program to do rather than what you thought you told it to do.”

So, as humans, not all of us are perfect, we all make mistakes and learn from it. That is quite evident from the bugs found in the software and upgrades to getting these bugs fixed. Again, John and myself are still here to assist you further. Hope, learning from these suggestions and guidance provided, you should be able to accomplish your goal now.

[SnooptheEngineer]

Thank you for your help Umar. I really appreciate the effort both of you have placed in answering my concerns. I am sure, I will have more questions as I dive more into the topic. Nonethless, you have done a great job expalining it to me :)