Polyfit returns the wrong correlation coeffcient when I try do a linear regression

Hello :)

I am trying to do a linear regression for a set of data, but the regression coefficient returned by matlab polyfit don't fit my data.

The data is plot just fine but the linear regression function seems very odd...

Please see the attached figure

I tried to plot with polyval (option 1) and with a linear relation (y=ax+b) (option 2) but it does not make any difference... The regression coefficient returned are just no the good ones...

Can anyone help me?

subwFMA         =   w10MeanFMA(521:771, 41:161);            %sub mean wind speed
subssrdFMA      =   ssrdMeanFMA(521:771, 41:161);           %sub mean solar radiation

%OPTION 1
pFMA      =   polyfit(subwFMA(:),subssrdFMA(:),1);          %find regression coefficient a and b for a linear regression

fFMA      =   polyval(pFMA,subwFMA(:));                     %linear regression function

figure(1)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),fFMA,'-');


%OPTION 2
polyfitFMA      =   polyfit(subwFMA(:),subssrdFMA(:),1);    %find correletation coefficient a and b for a linear regression

aFMA            =   polyfitFMA(1,1);                        %coeff a
bFMA            =   polyfitFMA(1,2);                        %coeff b

yssrdFMA        =   aFMA*subwFMA(:) + bFMA;                 %linear regression function

figure(2)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),yssrdFMA,'-');

I also tried to reverse my x and y data in polyfit but it makes things even worse!

Thanks

[John D'Errico]

Um, I think you misunderstand what is a correlation coefficient. You seem to think it is either the linear coefficient or the constant term in the regression, both of which are wrong. Nothing in what you did has returned the correlation coefficient.

https://en.wikipedia.org/wiki/Correlation_coefficient

I think when you use the words correlation coefficient, you think you are referring to the coefficients estimated from a linear regression. That is NOT a correlation coefficient.

As far as what you did wrong here, that is impossible to tell, since we lack your data.

[A LL]

Hi John,

I am sorry for the confucion, what I mean to do is to do a linear regression with my data by computing the regression coefficient (I think this is the correct expression).

When I plot the linear regression obtained from the polyfit function it does not seem to properly fit my data and I do not understand why this is happening because the data are correctly plot on my figure.

Thanks

[John D'Errico]

Again, we don't have your data. And it seems like you fit only a small portion of the data. So how can we possibly know what you did wrong?

Answer 1

[Jon]

I don't think you are plotting your fitted y values

For example where you have

fFMA = polyval(pFMA,subwFMA(:));                            %linear regression function
figure(1)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),yssrdFMA,'-');

I think you want

yssrdfFMA = polyval(pFMA,subwFMA(:));   %linear regression function
figure(1)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),yssrdFMA,'-');

and where you have

figure(2)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),fFMA,'-');

I think you want

figure(2)
plot(subwFMA(:),subssrdFMA(:),'o',subwFMA(:),yssrdFMA,'-');

[A LL]

Hello Jon,

Sorry for the confusion! Yes you are right, I just got confused in copy-pasting my code for the question but his is indeed what I have done. I edited my question.

In all cases, I do not understand why the regression coefficient that I obtain with the polyfit function do not fit my data while the data points are well represented.

Thanks

[Jon]

Looking at your data in your attached .jpg you basically just see a big cloud of raw data points. So, from your data there does not appear to be any functional relationship (linear or otherwise) between windspeed and solar radiation. So fitting the data to a straight line will not give a good fit, because a line does not give a good representation of a cloud of data points with no obvious slope or intercept.

Your problem is not with the how to curve fit the data, your problem is that your raw x and y data values seem to have very little relationship (no correlation)

[A LL]

Ok, I see thank you very much for your response.

I guess I just expected the linear regression to fit between the cloud of points...

(see pink line on the attached figure)

Also, when I computed the correlation coefficient I get -0.1138 which is very weak but still I expected to have some kind of a negative trend that showed up with polyfit by having a negative slope or something.

Thanks again

[Jon]

I agree, I would expect your fit line to go more through the cloud even if there were very little correlation. So I think you also have a programming error.

You said you made some edits to your code since the original one you posted. Can you please provide your current code (please use the code button on the MATLAB answers toolbar to get it nicely formatted). Also if you can provide your input file, as John D'Errico suggested that would be even better.

[A LL]

Ok, thanks again for your help!

I will go over my code one more time and if I can not find the error I will post it here. :)

update: I was able to solve my problem. I had a mistake in my code. Thanks again for your help!