How can I smooth this data before assigning a spline to it?

I am attempting to fit a cubic spline to rheological data which can be found in the attached excel file. I need to smooth the data so that when plotting the derivative of the function, the curve does not jump up and down. I guess I am tryin to remove noise and smooth into a better looking line. The code and generated figures are below:

clc;
clear all;

%% --Variable Assignment--
osc_strain = xlsread("ag_gel_data.xlsx",1,'A31:A51');
stor_mod = xlsread("ag_gel_data.xlsx",1,'K31:K51');
loss_mod = xlsread("ag_gel_data.xlsx",1,'L31:L51');

tb1 = table(osc_strain,stor_mod,loss_mod);

%% --Fit: Cubic Spline Interpolant--
[xData, yData] = prepareCurveData( osc_strain, stor_mod );

%% --Plotting-- (Figure 1)
lg1 = loglog(tb1,"osc_strain","stor_mod",'LineWidth',1);
hold on;
lg2 = loglog(tb1,"osc_strain","loss_mod",'LineWidth',1);

% Plot fit with data on a log-log scale.
xup = linspace(xData(1) , xData(end),1e4);
yup = interp1(xData,yData,xup,'pchip');
s = plot(xup,yup ,'--');

%% --Figure Stylization--
lg1.LineStyle = "-";
lg1.Color = "magenta";
lg1.Marker = ".";
lg1.MarkerSize = 16;

lg2.LineStyle = "-";
lg2.Color = "magenta";
lg2.Marker = "o";
lg2.MarkerSize = 4;

s.LineStyle = "--";
s.Color = "black";

xlabel('Oscillation Stress, \gamma (%)')
ylabel("G',G'' (Pa)")
legend('Storage Modulus','Loss Modulus','Cubic Spline','Location','NW')

%% --Plotting-- (Figure 2)
figure;
semilogx( xup,gradient(yup,xup),'-.','color',"black")
xlabel('Oscillation Stress, \gamma (%)')
ylabel("Differential Modulus, K (Pa)")

%% --Tabulate Differential Modulus--
K = gradient(yup,xup);
Ktable0 = [xup; K].';
Ktable = Ktable0(K>=0,:);

Figure 1:

The data I would like to smooth are both pink curves.

Figure 2:

Smoothing the pink curves would result in a smoother curve of the derivative which is plotted above.

Any suggestions? Thank you!

Answer 1

[Mathieu NOE]

hello

you could do this

just using basic interpolation (in log scale) and the regular smoothdata (use your own spline smoother if you prefer)

%% --Variable Assignment--
osc_strain = xlsread("ag_gel_data.xlsx",1,'A31:A51');
stor_mod = xlsread("ag_gel_data.xlsx",1,'K31:K51');
loss_mod = xlsread("ag_gel_data.xlsx",1,'L31:L51');

% log scale interpolate and smooth the data
Npoints = 200;
osc_strain2 = logspace(log10(min(osc_strain)),log10(max(osc_strain)),Npoints);

Nsmooth = 40;
method = 'lowess';

stor_mod2 = interp1(log(osc_strain),log(stor_mod),log(osc_strain2));
stor_mod2 = smoothdata(stor_mod2,method,Nsmooth);
stor_mod2 = exp(stor_mod2);

loss_mod2 = interp1(log(osc_strain),log(loss_mod),log(osc_strain2));
loss_mod2 = smoothdata(loss_mod2,method,Nsmooth);
loss_mod2 = exp(loss_mod2);


%% --Plotting-- (Figure 1)
loglog(osc_strain,stor_mod,'*-','LineWidth',2);
hold on
loglog(osc_strain2,stor_mod2,'*-','LineWidth',1);

loglog(osc_strain,loss_mod,'*-','LineWidth',2);
loglog(osc_strain2,loss_mod2,'*-','LineWidth',1);
hold off

[Elias Kerstein]

This is a really great start however, the peak of each curve is very important for determining the maximum stress of the material. Is there a way the smoothed data could be fit to incorporate the peaks?

Additionally, the reason I had fit a cubic spline before is so that I could plot the gradient of the curve. I am unsure if I can plot the gradient the way that you have suggested. Please let me know!

[Mathieu NOE]

well, I don't see a method where you could at the same time have a smoother curve AND have that curve go through the (raw data) peak point - unless you would either shift or stretch (anamorphosis) the curve so that a smoothed version of the curve would indeed go through that peak point, but that would simply mean we are manipulating the data and I don't think this is what you want

at the end , I wonder why we need to interpolate and smooth the curve , as this doesn't seem to be of some need when you compute the gradient (for my own fun I used both diff and gradient functions.

gradient on raw data seems to suffice, or ?

in this version of the code , I reduced a bit the amount of interpolation points (from 200 to 100) and also reduced the amount of smoothing , but , yes , there will always be a small difference between the peaks .

%% --Variable Assignment--
osc_strain = xlsread("ag_gel_data.xlsx",1,'A31:A51');
stor_mod = xlsread("ag_gel_data.xlsx",1,'K31:K51');
loss_mod = xlsread("ag_gel_data.xlsx",1,'L31:L51');

% log scale interpolate and smooth the data
Npoints = 100;
osc_strain2 = logspace(log10(min(osc_strain)),log10(max(osc_strain)),Npoints)';

Nsmooth = 5;
method = 'lowess';

stor_mod2 = interp1(log(osc_strain),log(stor_mod),log(osc_strain2));
stor_mod2 = smoothdata(stor_mod2,method,Nsmooth);
stor_mod2 = exp(stor_mod2);

% loss_mod2 = interp1(log(osc_strain),log(loss_mod),log(osc_strain2));
% loss_mod2 = smoothdata(loss_mod2,method,Nsmooth);
% loss_mod2 = exp(loss_mod2);


%% --Plotting-- (Figure 1)
figure(1);
loglog(osc_strain,stor_mod,'*-','LineWidth',2);
hold on
loglog(osc_strain2,stor_mod2,'*-','LineWidth',1);

% loglog(osc_strain,loss_mod,'*-','LineWidth',2);
% loglog(osc_strain2,loss_mod2,'*-','LineWidth',1);
hold off

%% --Plotting gradient -- (Figure 2)
figure(2);

% gradient computed with diff
gr = diff(stor_mod)./diff(osc_strain);
gr = [gr; NaN];

gr2 = diff(stor_mod2)./diff(osc_strain2);
gr2 = [gr2; NaN];

semilogx(osc_strain,gr,'*-','LineWidth',2);
hold on
semilogx(osc_strain2,gr2,'*-','LineWidth',2);

% compare with "gradient" function
semilogx(osc_strain,gradient(stor_mod)./gradient(osc_strain),'*-','LineWidth',2);
semilogx(osc_strain2,gradient(stor_mod2)./gradient(osc_strain2),'*-','LineWidth',1);

legend('raw data (diff)','interpol. & smoothed data (diff)','raw data (gradient)','interpol. & smoothed data  (gradient)');
hold off

[Elias Kerstein]

Hi, thank you for your answer! I am smoothing the data because my mentor and I are trying to emulate a data analysis method that we found in a literature review, but I agree that the gradient on the raw data is sufficient. It comes down to that smooth curves are easier to explain (and bring up less inquiries) when they are presented in a publication and more often than not, reducing jumps in the raw data down to "just being the data" doesn't quite satisfy reviewers or the principal investigator. I appreciate your work on this! I was able to use your smoothing method then apply my older code to create a cubic spline of the smoothed data, which is what I had set out to do. Here is the code for reference:

%% --Variable Assignment--
osc_strain = xlsread("ag_gel_data.xlsx",1,'A31:A51');
stor_mod = xlsread("ag_gel_data.xlsx",1,'K31:K51');
loss_mod = xlsread("ag_gel_data.xlsx",1,'L31:L51');

%% --Log Scale Interpolate and Smooth Data
Npoints = 200;
osc_strain2 = logspace(log10(min(osc_strain)),log10(max(osc_strain)),Npoints);

Nsmooth = 45;
method = 'lowess';

stor_mod2 = interp1(log(osc_strain),log(stor_mod),log(osc_strain2));
stor_mod2 = smoothdata(stor_mod2,method,Nsmooth);
stor_mod2 = exp(stor_mod2);

loss_mod2 = interp1(log(osc_strain),log(loss_mod),log(osc_strain2));
loss_mod2 = smoothdata(loss_mod2,method,Nsmooth);
loss_mod2 = exp(loss_mod2);

%% --Generating Curve Data for Differentiation
[xData1, yData1] = prepareCurveData( osc_strain2, stor_mod2 );
[xData2, yData2] = prepareCurveData( osc_strain2, loss_mod2 );

% Plot fit with data on a log-log scale.
xup1 = linspace(xData1(1) , xData1(end),1e4);
yup1 = interp1(xData1,yData1,xup1,'pchip');
s11 = plot(xup1,yup1 ,'--');

xup2 = linspace(xData2(1) , xData2(end),1e4);
yup2 = interp1(xData2,yData2,xup2,'pchip');
s22 = plot(xup2,yup2 ,'--');

%% --Plotting-- (Figure 1)
figure(1);
lg1 = loglog(osc_strain,stor_mod,'.-','LineWidth',1);
hold on
s1 = loglog(osc_strain2,stor_mod2,'--','LineWidth',1);
lg2 = loglog(osc_strain,loss_mod,'o-','LineWidth',1);
s2 = loglog(osc_strain2,loss_mod2,'--','LineWidth',1);
hold off

%% --Figure Stylization--
lg1.LineStyle = "-";
lg1.Color = "magenta";
lg1.Marker = ".";
lg1.MarkerSize = 16;

s1.LineStyle = "--";
s1.Color = "black";

lg2.LineStyle = "-";
lg2.Color = "magenta";
lg2.Marker = "o";
lg2.MarkerSize = 4;

s2.LineStyle = "--";
s2.Color = "blue";

xlabel('Oscillation Stress, \gamma (%)')
ylabel("G',G'' (Pa)")
legend('Storage Modulus','Smooth Storage Modulus','Loss Modulus','Smooth Loss Modulus','Location','NW')

%% --Plotting-- (Figure 2)
figure(2);
semilogx( xup1,gradient(yup1,xup1),'-.','color',"black")
hold on
semilogx( xup2,gradient(yup2,xup2),'-.','color',"blue")
hold off

xlabel('Oscillation Stress, \gamma (%)')
ylabel('Differential Modulus, K (Pa)')
legend('Storage Modulus','Loss Modulus','Location','NE')

%% --Tabulate Differential Modulus--
K1 = gradient(yup1,xup1);
Ktable01 = [xup1; K1].';
Ktable1 = Ktable01(K1>=0,:);

K2 = gradient(yup2,xup2);
Ktable02 = [xup2; K2].';
Ktable2 = Ktable02(K2>=0,:);

I'm certain it is bloated, but it works for my purposes as it smooths, applies a cubic spline interpolant, then tabulates the gradient in a table that I can export into a dedicated graphing software. If you have any more comments on the code above please let me know and again, thank you for your help!

[Mathieu NOE]

Glad you have found your way

Cannot really comment your code , as the main difference is related to your "magic sauce" (your function prepareCurveData) that you did not post - but I understand you may not want to share that code

[Elias Kerstein]

prepareCurveData is not a function I have written and I do not have a file in the folder named as such for MATLAB to call. I believe it is a built in function into the software or at least Curve Fitter application. Anyway, thank you for your help!

[Mathieu NOE]

ok, tx for the explanation

all the best