Read older MATFILE in the current version.

Hi, I have very old matlab files. Current Matlab dose not support the old format. I have tried reading it in Python and R. Python does read it but the output data is not correct. Is there a way to read these files and retreive the data. I have attached a test file for the reference. Any help would be great!

[Star Strider]

I can’t open it with load. Try it as a .txt file.

EDIT — (19:06)

To be clear, a .txt file instead of a .mat file.

[Walter Roberson]

Not text files. At the moment I cannot rule out v4 files

[DGM]

Shouldn't v4 files be readable?

How old is plausible given what's known of the file lineage?

[Cris LaPierre]

Do you have a guess at approximately what version of MATLAB this was created in?

[dipak sanap]

I do not know the exact version. But the data was accessable around 2008 last time. Additioanlly, I am able to read the data in Python and R but I get following errors:

1. Python

UserWarning: We do not support byte ordering 'VAX G-float'; returned data may be corrupt

warnings.warn("We do not support byte ordering '%s'; returned "

2. And in R, I get following error.

Error in readMat4Header(con, firstFourBytes = firstFourBytes) :

Looks like a MAT v4 file, but the storage format of numerics (VAX D-float, VAX G-float or Cray) is not supported. Currently only IEEE numeric formats in big or little endian are supported.

In addition: There were 50 or more warnings (use warnings() to see the first 50)

[Walter Roberson]

https://www.mathworks.com/matlabcentral/fileexchange/22675-read-vaxd-and-vaxg-files-in-r2008b-and-later

https://www.mathworks.com/matlabcentral/answers/13901-converting-vax-float-to-matlab-float

[dpb]

Table 1-8, Level 4 MAT-File Matrix Header Format
Field Description
type The type flag contains an integer whose decimal digits encode storage information. If the
integer is represented as MOPT where M is the thousands digit, O is the hundreds digit, P is the
tens digit, and T is the ones digit, then:
M indicates the numeric format of binary numbers on the machine that wrote the file. Use this
table to determine the number to use for your machine:
0 IEEE Little Endian (PC, 386, 486, DEC Risc)
1 IEEE Big Endian (Macintosh, SPARC®, Apollo, SGI, HP
9000/300, other Motorola® systems)
2 VAX D-float
3 VAX G-float
4 Cray
...

Looking at the given file (had to do this locally, couldn't seem to fopen the attachment)...

>> fn='test_read.mat';
>> fid=fopen(fn,'r');
>> type=fread(fid,1,'int4')
>> type =
        3000
>> fid=fclose(fid);

That would indeed, seem to be a MATLAB 4 VAX G-float array....

There's a link to the description of .mat files V4, V5 available at https://www.mathworks.com/matlabcentral/answers/93868-is-there-documentation-on-the-structure-of-mat-files-in-matlab#answer_103219

Answer 1

[DGM]

FWIW, I was able to use Walter's first link to read the file.

myfilename = 'test_read.mat';
fid = fopen(myfilename, 'r', 'ieee-le');

% for sake of simplicity, i'm going to just presume M0PT is always 3000
% always VAX-G, double-precision, numeric
fseek(fid,4,-1);

% get the matrix size
nrows = fread(fid, 1, 'uint32');
ncols = fread(fid, 1, 'uint32');

% i'm going to just ignore the possibility that the matrix is complex-valued
fseek(fid,4,0);

% read the original variable name
namelen = fread(fid, 1, 'uint32');
varname = fread(fid, namelen-1, '*char')
fseek(fid,1,0); % name is followed by a null character

% read the matrix
A = freadVAXG(fid, [nrows ncols], 'double');

% i assume there's only one header in the stream, so just close the file
fclose(fid);

% reshape the data and test it against Steven's recovered version
A = reshape(A,nrows,[]);

load new_test_read.mat

immse(A,E) % they match

Of course, I'm assuming that other files are similar. (real-valued numeric double in VAX-G format, with only a single matrix in the file)

The details of the MAT-file format are specified here:

https://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

See page 1-25 for the V4 format

[dipak sanap]

@DGM, I tried this with function link provided by Walter. I get error with line "Error in testOpen (line 10)

varname = fread(fid, namelen-1, '*char')". Not sure why namelen is empty for me.

[dipak sanap]

@DGM Yes, all files are real-valued numerics and one matrix in it of size 64 by 64.

[DGM]

Is namelen zero or actually empty?

Is this for the same mat-file as before?

I guess if possible, we could also just assume that the variable name is a fixed length.

myfilename = 'test_read.mat';
fid = fopen(myfilename, 'r', 'ieee-le');

% for sake of simplicity, i'm going to just presume M0PT is always 3000
% always VAX-G, double-precision, numeric
fseek(fid,4,-1);

% get the matrix size
nrows = fread(fid, 1, 'uint32');
ncols = fread(fid, 1, 'uint32');

% i'm going to just ignore the possibility that the matrix is complex-valued
% i'm going to assume that the variable name is always a single character
fseek(fid,10,0);

% read the matrix
A = freadVAXG(fid, [nrows ncols], 'double');

% i assume there's only one header in the stream, so just close the file
fclose(fid);

% reshape the data and test it against Steven's recovered version
A = reshape(A,nrows,[]);
load new_test_read.mat

immse(A,E) % they match

That said, i'm not sure why it wouldn't read namelen, so I can't be certain that it would read anything else correctly anyway.

[dipak sanap]

@DGM Yes, its empty and not zero and for the same orignal test_read.mat

[DGM]

I really don't know how fread() could return empty. If i'm missing some sort of platform-dependent behavior or something, maybe someone can tell me what I'm doing wrong.

myfilename = 'test_read.mat';
fid = fopen(myfilename, 'r', 'ieee-le');

% for sake of simplicity, i'm going to just presume M0PT is always 3000
% always VAX-G, double-precision, numeric
fseek(fid,4,-1);

% get the matrix size
nrows = fread(fid, 1, 'uint32')

nrows = 64

ncols = fread(fid, 1, 'uint32')

ncols = 64

% i'm going to just ignore the possibility that the matrix is complex-valued
fseek(fid,4,0);

% read the original variable name
namelen = fread(fid, 1, 'uint32')

namelen = 2

varname = fread(fid, namelen-1, '*char')

varname = 'E'

fseek(fid,1,0); % name is followed by a null character

% read the matrix
A = freadVAXG(fid, [nrows ncols], 'double');

% i assume there's only one header in the stream, so just close the file
fclose(fid);

% reshape the data and test it against Steven's recovered version
A = reshape(A,nrows,[]);
load new_test_read.mat
immse(A,E) % they match

ans = 0

[Walter Roberson]

fread() can return empty when you are at end-of-file.

[DGM]

Is there a way that could happen here?

[dipak sanap]

@DGM, it worked, Idk, I just restarted the Matlab and moved everything to clear directory. Thank you so much! I just have a final question. How can I make it work for files with different size arrays? Like just row or vector or matrix?

[DGM]

As I understand it, the V4 format only supports 2D arrays, and the size of the arrays should be fetched from the header (nrows x ncols). Unless I'm missing something, the given example should already support variations in array sizes (so long as the other assumptions don't change).

Answer 2

[Steven Lord]

I tried loading the original data using the oldest release of MATLAB I can run and was able to read it. It must have been created by a very old release (pre-v4 perhaps?). I resaved it as a newer MAT-file, attached here.

[dipak sanap]

Hi @Steven Lord, thank you very much. I had sent just a test file. This new file works perfectly!!! I am wondering if you could help me read some more files if I send you in a folder. I would really apprecite it.

[dpb]

Is there a TMW service to translate old files that have been orphaned by dropping support for early releases, Steven? That would seem to be a valuable service that a 'bot could handle like the "convert pdf to Excel" services that abound.

Or, I guess presuming the file exchange submissions work, that's not too bad of a route to have to go, other than it seems that it would be a useful amenity for load or whos to let the user know what the deal is instead of just not being able to read it and quitting....Python at least did recognize it as MAT 4 and indicated it was sorry, but at least it gave useful feedback.

Not all that common any more, but when it bites it's bound to hurt a lot and most wouldn't think to try Python, I'd venture nor know to go searching FEX.

Just out of curiosity, how old was "oldest" version?

[Star Strider]

@dpb — I first used it on a PC in 1993 for my first course in linear control systems, and it may have existed before then. (The course was taught at night, after work.) In those days, it came on a series of 3½" (8.9 cm) floppies that had to be loaded sequentially, and the documentation was a library of hardcopy softcover books. I believe earlier mainframe versions existed from about 1980 onward, however I’m not certain of that. I’ve been using it in some capacity ever since.

[dpb]

Oh, yeah, I've had copy since the first Windows 3.1 release; was thinking the other day I probably shouldn't have but I cleaned out and threw out the old 3.5" floppy disc releases a while back...the PC version was released in roughly mid-80s; there's a history at the TMW site but I didn't go refresh my memory of exact dates. I think I purchased the first copy with my first PC when SAIC (thru whom I was consulting at the time) had an employee interest-free paycheck loan program. I had waited until the AT came out and bought the then-new AST Research 286 /287 coprocessor and the whole of 2MB(!) memory and a 10MB harddrive. With a 12" mono monitor, it was something around $4K in, I think, late '87/early '88 time frame. That machine still survives and, I guess the last release I loaded on it would still run, actually, although I've not booted it in years and years and years, now...dunno if it would, but couldn't bring myself to throw it out. :)

What I was really asking Steven about was what was the oldest release he could still run...

[Cris LaPierre]

I found I could open the file in R2007b, and R2008a, but not R2008b.

I couldn't launch anything earlier than that on my computer.

[Steven Lord]

Actually now that I think about it I didn't use the oldest release I believe can run on my machine (which is different from the oldest release whose files I could access.) I went as far back as release R11 (even though that's not remotely close to a supported release on my OS) because I know I can run that release with only a little difficulty.

Since I work on functions that have existed in some cases since Cleve's original MATLAB (predating MathWorks!) sometimes it's useful to perform "archaeology" to figure out when a particular behavior was introduced and that can entail reading or running code from very old releases.

Is there a TMW service to translate old files that have been orphaned by dropping support for early releases, Steven? That would seem to be a valuable service that a 'bot could handle like the "convert pdf to Excel" services that abound.

I don't know if there is such a service, though it's an interesting idea.