Stream Data File using System Objects

I have a very large data file, 10's of GB, from a lab instrument that I want to manipulate in Matlab. I would like to take advantage of using system objects to stream the data through to simulate how my real system will react to incoming signals. My question is to get streaming objects to work for data that is not contained in a audio or video file do I first setup a memory map for my file then advance the file using the memmap function and the step command to push my data through the system objects? I can't seem to find a file reader function in a system object sense for simple data written to a file. Thank you

Answer 1

[Kaustubha Govind]

I don't see a System object to read/write from files either, but I found this demo: Creating New Kinds of System Objects for File Input and Output which shows an example that uses fread/fwrite. It looks like you might need to create your own system object that suits your file format. If the memmap technique is more efficient, then perhaps you could use that instead of fread/fwrite.

[Melanie Chainsman]

New link location is here: Creating New Kinds of System Objects for File Input and Output

Answer 2

[Malcolm Lidierth]

The MAT-file utilities form the FEX have been updated to provided support for fread and for memmapfile and matlab.io.MatFile objects through a consistent API. These can also support custom binary files if you supply a memmapfile object on construction: Example for a MAT-file y=nmatrix('myfile.mat', 'x'); memory maps variable x from file and allows partial i/o using standard MATLAB matrix syntax: thus while y is a scalar object, y(1:10), y(:,:,1:5,:) etc. return the data as appropriate.

See http://sourceforge.net/projects/waterloo/files/

The PDF in the download compares speeds using load, fread and memmapfile in different circumstances. Example:

This compares the speeds for a modestly sized Version 6 MAT-file (so all data could be fitted into the workspace using load). The superclass "remembers" previous calls, so for repeated access to a file in a MATLAB session, the intercept for nmatrix class use would be shifted further towards zero.

Answer 3

[Walter Roberson]

fopen() the file with 'r+' permissions.

Figure out how many bytes you need to seek to go the the chosen place in the middle of the file, by knowing how many objects and the size the object occupies. fseek() to that position in the file.

Any of the methods that you use for serial I/O on a fid would then be read in with methods that look identical to reading from the serial port, with very minor differences (but probably no useful BytesReceivedFcn callback.)

If you would normally have been using digitalio().getdata(), then in the simplest form you would just fread() samples from the file instead. If, though, time stamps and all were recorded in the file, you might need to either decode the data manually or use memmapfile() .

A warning, though: I read a month or two ago that at least one data recording file type uses "short" time stamps most of the time, but upon important changes (e.g., new day), outputs a larger timestamp. It was not at all clear to me in reading that description as to how to know in that program's log which timestamp variety you were going to get; I remember it made a mess of standard memmapfile() usage. Oh yes, and of course that file format always required scanning from the beginning to the end, as you had no idea how many of the longer time-stamps might have occured, and as data might accidentally look like timestamps, you couldn't just read forward from an arbitrary place looking for something that looked like a record boundary...