Multivariate Normal Regression Types

Regressions

Each regression function has a specific operation. This section shows how to use these functions to perform specific types of regressions. To illustrate use of the functions for various regressions, “typical” usage is shown with optional arguments kept to a minimum. For a typical regression, you estimate model parameters and residual covariance matrices with the mle functions and estimate the standard errors of model parameters with the std functions. The regressions “without missing data” essentially ignore samples with any missing values, and the regressions “with missing data” ignore samples with every value missing.

Multivariate Normal Regression

Multivariate normal regression, or MVNR, is the “standard” implementation of the regression functions in Financial Toolbox™ software.

Multivariate Normal Regression Without Missing Data

Estimate parameters using mvnrmle:

[Parameters, Covariance] = mvnrmle(Data, Design);

Estimate standard errors using mvnrstd:

StdParameters = mvnrstd(Data, Design, Covariance);

Multivariate Normal Regression With Missing Data

Estimate parameters using ecmmvnrmle:

[Parameters, Covariance] = ecmmvnrmle(Data, Design);

Estimate standard errors using ecmmvnrstd:

StdParameters = ecmmvnrstd(Data, Design, Covariance);

Least-Squares Regression

Least-squares regression, or LSR, sometimes called ordinary least-squares or multiple linear regression, is the simplest linear regression model. It also enjoys the property that, independent of the underlying distribution, it is a best linear unbiased estimator (BLUE).

Given m = NumSamples observations, the typical least-squares regression model seeks to minimize the objective function

$\sum_{k = 1}^{m} {(Z_{k} - H_{k} b)}^{T} (Z_{k} - H_{k} b),$

which, within the maximum likelihood framework of the multivariate normal regression routine mvnrmle, is equivalent to a single-iteration estimation of just the parameters to obtain Parameters with the initial covariance matrix Covariance held fixed as the identity matrix. In the case of missing data, however, the internal algorithm to handle missing data requires a separate routine ecmlsrmle to do least-squares instead of multivariate normal regression.

Least-Squares Regression Without Missing Data

Estimate parameters using mvnrmle:

[Parameters, Covariance] = mvnrmle(Data, Design, 1);

Estimate standard errors using mvnrstd:

StdParameters = mvnrstd(Data, Design, Covariance);

Least-Squares Regression With Missing Data

Estimate parameters using ecmlsrmle:

[Parameters, Covariance] = ecmlsrmle(Data, Design);

Estimate standard errors using ecmmvnrstd:

StdParameters = ecmmvnrstd(Data, Design, Covariance);

Covariance-Weighted Least Squares

Given m = NUMSAMPLES observations, the typical covariance-weighted least squares, or CWLS, regression model seeks to minimize the objective function

$\sum_{k = 1}^{m} {(Z_{k} - H_{k} b)}^{T} C_{0} (Z_{k} - H_{k} b)$

with fixed covariance C₀.

In most cases, C₀ is a diagonal matrix. The inverse matrix $W = C_{0}^{- 1}$ has diagonal elements that can be considered relative “weights” for each series. Thus, CWLS is a form of weighted least squares with the weights applied across series.

Covariance-Weighted Least Squares Without Missing Data

Estimate parameters using mvnrmle:

[Parameters, Covariance] = mvnrmle(Data, Design, 1, [], [], [], Covar0);

Estimate standard errors using mvnrstd:

StdParameters = mvnrstd(Data, Design, Covariance);

Covariance-Weighted Least Squares With Missing Data

Estimate parameters using ecmlsrmle:

[Parameters, Covariance] = ecmlsrmle(Data, Design, [], [], [], [], Covar0);

Estimate standard errors using ecmmvnrstd:

StdParameters = ecmmvnrstd(Data, Design, Covariance);

Feasible Generalized Least Squares

An ad hoc form of least squares that has surprisingly good properties for misspecified or nonnormal models is known as feasible generalized least squares, or FGLS. The basic procedure is to do least-squares regression and then to do covariance-weighted least-squares regression with the resultant residual covariance from the first regression.

Feasible Generalized Least Squares Without Missing Data

Estimate parameters using mvnrmle:

[Parameters, Covariance] = mvnrmle(Data, Design, 2, 0, 0);

or (to illustrate the FGLS process explicitly)

[Parameters, Covar0] = mvnrmle(Data, Design, 1);
[Parameters, Covariance] = mvnrmle(Data, Design, 1, [], [], [], Covar0);

Estimate standard errors using mvnrstd:

StdParameters = mvnrstd(Data, Design, Covariance);

Feasible Generalized Least Squares With Missing Data

Estimate parameters using ecmlsrmle:

[Parameters, Covar0] = ecmlsrmle(Data, Design);
[Parameters, Covariance] = ecmlsrmle(Data, Design, [], [], [], [], Covar0);

Estimate standard errors using ecmmvnrstd:

StdParameters = ecmmvnrstd(Data, Design, Covariance);

Seemingly Unrelated Regression

Given a multivariate normal regression model in standard form with a Data matrix and a Design array, it is possible to convert the problem into a seemingly unrelated regression (SUR) problem by a simple transformation of the Design array. The main idea of SUR is that instead of having a common parameter vector over all data series, you have a separate parameter vector associated with each separate series or with distinct groups of series that, nevertheless, share a common residual covariance. It is this ability to aggregate and disaggregate series and to perform comparative tests on each design that is the power of SUR.

To make the transformation, use the function convert2sur, which converts a standard-form design array into an equivalent design array to do SUR with a specified mapping of the series into NUMGROUPS groups. The regression functions are used in the usual manner, but with the SUR design array instead of the original design array. Instead of having NUMPARAMS elements, the SUR output parameter vector has NUMGROUPS of stacked parameter estimates, where the first NUMPARAMS elements of Parameters contain parameter estimates associated with the first group of series, the next NUMPARAMS elements of Parameters contain parameter estimates associated with the second group of series, and so on. If the model has only one series, for example, NUMSERIES = 1, then the SUR design array is the same as the original design array since SUR requires two or more series to generate distinct parameter estimates.

Given NUMPARAMS parameters and NUMGROUPS groups with a parameter vector (Parameters) with NUMGROUPS * NUMPARAMS elements from any of the regression routines, the following MATLAB^® code fragment shows how to print a table of SUR parameter estimates with rows that correspond to each parameter and columns that correspond to each group or series:

fprintf(1,'Seemingly Unrelated Regression Parameter
   Estimates\n');
fprintf(1,'   %7s ',' ');
fprintf(1,'  Group(%3d) ',1:NumGroups);
fprintf(1,'\n');
for i = 1:NumParams
	fprintf(1,'   %7d ',i);
	ii = i;
    	for j = 1:NumGroups
    		fprintf(1,'%12g ',Param(ii));
    		ii = ii + NumParams;
    		end
    		fprintf(1,'\n');
end
fprintf(1,'\n');

Seemingly Unrelated Regression Without Missing Data

Form a SUR design using convert2sur:

DesignSUR = convert2sur(Design, Group);

Estimate parameters using mvnrmle:

[Parameters, Covariance] = mvnrmle(Data, DesignSUR);

Estimate standard errors using mvnrstd:

StdParameters = mvnrstd(Data, DesignSUR, Covariance);

Seemingly Unrelated Regression With Missing Data

Form a SUR design using convert2sur:

DesignSUR = convert2sur(Design, Group);

Estimate parameters using ecmmvnrmle:

[Parameters, Covariance] = ecmmvnrmle(Data, DesignSUR);

Estimate standard errors using ecmmvnrstd:

StdParameters = ecmmvnrstd(Data, DesignSUR, Covariance);

Mean and Covariance Parameter Estimation

Without missing data, you can estimate the mean of your Data with the function mean and the covariance with the function cov. Nevertheless, the function ecmnmle does this for you if it detects an absence of missing values. Otherwise, it uses the ECM algorithm to handle missing values.

Estimate parameters using ecmnmle:

[Mean, Covariance] = ecmnmle(Data);

Estimate standard errors using ecmnstd:

StdMean = ecmnstd(Data, Mean, Covariance);