How to get critical values and P values of F statistic and T statistic?

I am trying to develop a multivariable regression model having 2 input vectors and one output vector. there are 10 observations. My objective is to code the entire model along with statistic. I have got F-statistic value of 0.4011. I want to compare it with critical value of F with alpha =0.05, numerator degree of freedom =2 and denominator degree of freedom =7. I also want to generate p value correspond to F=0.4011. Similarly, my t statistic value for a coefficient is 0.7755. How to generate critical t statistic at alpha/2=0.025 and 7 degree of freedom. I also want to generate p value of t=0.7755

Answer 1

[John D'Errico]

Unless your goal is to somehow improve on what is already written, or you simply don't have those existing tools, then you should never be reinventing the wheel. Use existing code, for example, regress is in the stats toolbox. Or you could even use my polyfitn from the file exchange. It will return much of what you need, then you can do the rest.

If your question is really how to generate those statistics, then again, go to the stats toolbox. It will have the tools you need. Yes, you can use the incomplete gamma function, transformed into a Student's t to compute t statistics. But why do so unless absolutely necessary?

So I'm not terribly sure what it is that you really need here. Is your goal to learn how to do these things from scratch?

[Ajay Goyal]

Thank you very much John for your valuable answer. Yes, I understand that there are many regression methods that can be implemented directly in Matlab (regress, stepwiselm,lasso, elastic net etc.). I have already worked in detail on all of them. But still I am keen to code the entire concept behind the models. Could you please help me further in this problem with your valuable knowledge.Thanks

[John D'Errico]

Off to play bridge for now. I'll return this afternoon to add a response. Of course, there is a LOT of ground to cover, but let me see what I can add.

[John D'Errico]

Back...

This can become a small book. No, a large book, given the number of texts to be found on the subject. So I'll just try to point you in the right direction.

A good place to start for any regression estimates is the pivoted QR. Stable. Efficient. Thus if you are solving the problem

A*x = b

then factor A as

[Q,R,E] = qr(A,0)

So we have A factored such that

A(:,E) = Q*R

The good news is that you can use this to solve for the regression estimates, but also you have R. R can be viewed as the transpose of the Cholesky factor of A'*A. That will be needed to compute inv(A'*A), then used for parameter variances. Thus...

% parameter estimates
coef(E) = R\(Q'*y);

% predictions
yhat = A*coef;

But how about the covariance matrix of the parameters? We have this relation:

inv(A'*A) = inv(R'*R)

The nice thing is that Q goes away, but also, there is a nice identity for the inverse of a product of square (non-singular) matrices.

inv(U*V) = inv(V)*inv(U)

So if R has an inverse, then all we need do is compute inv( R ).

inv(R') = inv( R )'

The R inverse can be done efficiently using backslash too, since MATLAB figures out that R is upper triangular, so this is nothing more than a back-substitution, thus much more efficient than a full matrix inverse.

Rinv = R\eye(n,n);

Therefore we have

inv(A'*A) = Rinv'*Rinv

(It has been a while since I've been through some of this. You can read through my polyfitn, and I may have written some of this up in my optimization tips and tricks document too.)

In fact, you don't even need to compute the complete covariance matrix. All you need are the diagonal elements of that matrix. A nice trick can be found for just those diagonal elements. Thus, if Var is a vector containing the desired asymptotic parameter variances, we compute it as:

s = norm(y - yhat);
Var(E) = s^2*sum(Rinv.^2,2)/(n-nt);

Here n is the number of data points, nt is the number of terms in the model to be estimated. So n-nt is the number of degrees of freedom remaining in the model.

So that gives you parameter variance estimates. I won't get into the issues of a poorly conditioned linear system, but then use of those parameter variances would become highly suspect.

Most of your questions were on F and t statistics, but I'll stop this comment here as a reasonable point to stop. I've shown how one can efficiently compute parameter estimates using a pivoted QR, as well, the parameter variance estimates. I'll continue on how one computes those Student's t-statistics in another comment.

[John D'Errico]

Next, given parameter estimates (beta) and parameter standard errors (sigma), the next question is how to compute a t statistic for that parameter. The degrees of freedom are known, so the number of data points (n), minus the number of terms in the model (nt).

We have the ratio beta/sigma as following a Student-t, with n-nt degrees of freedom. If you have the Student-t CDF on hand, this is easy. So tcdf will give it to you from the stats toolbox.

Alternatively, if you start with a p value, and you wish to compute a critical point based on that value of p, then it is an inverse Student-t that you need, so tinv from the stats toolbox.

If you do not have tinv and tcdf, then it must be done the hard way. A quick check of Abramowitz and Stegun finds that we can transform a beta distribution into a Student-t. (A&S is ABSOLUTELY the major source of this information, and still the go-to text after all these years. GET A COPY. It is well worth the minor price for the softcover version, or you can often find hard cover copies for sale.) That tome (in my edition) shows section 26.7.1 gives a transformation for the Student't into an incomplete beta function. Be careful, because as I recall, MATLAB uses a slightly different definition of the beta function, but the transformation is trivial as I recall. (You can find the formula in my polyfitn code.)

Similarly, a call to tinv would be done using the inverse incomplete beta function, betaincinv in MATLAB.

Finally, F dstribution computations can be done using fcdf or finv from the stats toolbox. Or, they can be done by checking section 26.6 in A&S. At a quick glance, 26.6.2 suggests that the F also transforms into a beta.