Main Content

Get Started with MDF Datastore

This example shows you how to use the MDF datastore feature of Vehicle Network Toolbox™ to quickly and efficiently process a data set spread across a collection of multiple MDF-files. This workflow is also valuable when there are too much data to fit into available memory.

Access MDF-Files in a Datastore

Find the collection of MDF-files representing logged information from multiple test sequences. Note that MDF-files to be used by MDF datastore as a set must have the same channel group and channel content structure.

dir("File*.mf4")
File01.mf4  File02.mf4  File03.mf4  File04.mf4  File05.mf4  

Create an MDF Datastore

You create an MDF datastore by selecting a folder location containing a collection of MDF-files. In this case, target all files in the current working directory.

mds = mdfDatastore(pwd)
mds = 
  MDFDatastore with properties:

  DataStore Details
                         Files: {
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  ChannelGroupNumber    AcquisitionName         Comment             ... and 4 more columns    
                                  __________________    _______________    _________________                

                                          1               {0x0 char}       {'Integer Types'}    {0x0 double}
                                          2               {0x0 char}       {'Float Types'  }    {0x0 double}


                      Channels: 
                                  ChannelGroupNumber                 ChannelName                  DisplayName        ... and 10 more columns    
                                  __________________    ______________________________________    ___________                

                                          1             {'Signed_Int16_LE_Offset_32'         }    {0x0 char}     {0x0 double}
                                          1             {'Unsigned_UInt32_LE_Master_Offset_0'}    {0x0 char}     {0x0 double}
                                          2             {'Float_32_LE_Offset_64'             }    {0x0 char}     {0x0 double}

                                ... and 1 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE_Offset_32';
                                'Unsigned_UInt32_LE_Master_Offset_0'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: 'file'
                    Conversion: Numeric

Configure MDF Datastore

Multiple options allow control of what data are read from the MDF-files and how the reads are performed. In this case, the first channel group is used by default. Note that only one channel group may be selected by the datastore at a time. You can also specify certain channels within the selected channel group to read. In this case, all channels are read by default.

mds.SelectedChannelGroupNumber
ans = 1
mds.SelectedChannelNames
ans = 2x1 string
    "Signed_Int16_LE_Offset_32"
    "Unsigned_UInt32_LE_Master_Offset_0"

Preview MDF Datastore

Using the preview function, you can obtain a quick view of the data available in the file set. Preview always returns up to eight data points from the first file in the datastore.

preview(mds)
ans=8×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 
    5 sec                5                                5                 
    6 sec                6                                6                 
    7 sec                7                                7                 

Read All Data in MDF Datastore

You can use the readall function to read the entire data in a single call. This is an efficient way to read from many files when the data set fits into available memory. After running readall, the datastore resets to the beginning of the data set.

data = readall(mds);
data(1:5,:)
ans=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

Read a Subset of Data in MDF Datastore

You can use the read function to obtain a subset of data from the datastore. The size of the subset is determined by the ReadSize property of the MDF datastore object. By default, data from an entire file are read in one call. The power of a datastore comes from reading through multiple files sequentially within the file set. As you read, the datastore automatically bridges from one file to the next until all data from all files are read.

for ii = 1:3
    data = read(mds);
    whos("data")
    data(1:5,:)
end
  Name          Size             Bytes  Class        Attributes

  data      10000x2             141839  timetable              
ans=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

  Name          Size             Bytes  Class        Attributes

  data      10000x2             141839  timetable              
ans=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

  Name          Size             Bytes  Class        Attributes

  data      10000x2             141839  timetable              
ans=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

Reset MDF Datastore

At any time, you can call the reset function to start over at the beginning of the data set.

reset(mds)

Configure Number of Records to Read from MDF Datastore

You can use the ReadSize property to specify how much data to read on each call. ReadSize can be specified as a numeric value to read a fixed number of data points. ReadSize lets you control how much data is loaded into memory when you have a data set larger than available memory. It is recommended to use custom read sizes that are small enough to fit in memory, but still as large as possible to reduce processing overhead and improve performance.

mds.ReadSize = 5
mds = 
  MDFDatastore with properties:

  DataStore Details
                         Files: {
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  ChannelGroupNumber    AcquisitionName         Comment             ... and 4 more columns    
                                  __________________    _______________    _________________                

                                          1               {0x0 char}       {'Integer Types'}    {0x0 double}
                                          2               {0x0 char}       {'Float Types'  }    {0x0 double}


                      Channels: 
                                  ChannelGroupNumber                 ChannelName                  DisplayName        ... and 10 more columns    
                                  __________________    ______________________________________    ___________                

                                          1             {'Signed_Int16_LE_Offset_32'         }    {0x0 char}     {0x0 double}
                                          1             {'Unsigned_UInt32_LE_Master_Offset_0'}    {0x0 char}     {0x0 double}
                                          2             {'Float_32_LE_Offset_64'             }    {0x0 char}     {0x0 double}

                                ... and 1 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE_Offset_32';
                                'Unsigned_UInt32_LE_Master_Offset_0'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: 5
                    Conversion: Numeric

for ii = 1:3
    data = read(mds)
end
data=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

data=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    5 sec                5                                5                 
    6 sec                6                                6                 
    7 sec                7                                7                 
    8 sec                8                                8                 
    9 sec                9                                9                 

data=5×2 timetable
     Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    ______    _________________________    __________________________________

    10 sec               10                                10                
    11 sec               11                                11                
    12 sec               12                                12                
    13 sec               13                                13                
    14 sec               14                                14                

Configure a Time Range to Read from MDF Datastore

You can also specify ReadSize as a duration to read data points by elapsed time. Note that when the read type is changed, the datastore resets to the beginning of the data set.

mds.ReadSize = seconds(5)
mds = 
  MDFDatastore with properties:

  DataStore Details
                         Files: {
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File01.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File02.mf4';
                                '/tmp/Bdoc21b_1757077_208470/tp99e09924/vnt-ex10761765/File03.mf4'
                                 ... and 2 more
                                }
                 ChannelGroups: 
                                  ChannelGroupNumber    AcquisitionName         Comment             ... and 4 more columns    
                                  __________________    _______________    _________________                

                                          1               {0x0 char}       {'Integer Types'}    {0x0 double}
                                          2               {0x0 char}       {'Float Types'  }    {0x0 double}


                      Channels: 
                                  ChannelGroupNumber                 ChannelName                  DisplayName        ... and 10 more columns    
                                  __________________    ______________________________________    ___________                

                                          1             {'Signed_Int16_LE_Offset_32'         }    {0x0 char}     {0x0 double}
                                          1             {'Unsigned_UInt32_LE_Master_Offset_0'}    {0x0 char}     {0x0 double}
                                          2             {'Float_32_LE_Offset_64'             }    {0x0 char}     {0x0 double}

                                ... and 1 more rows

  Options
          SelectedChannelNames: {
                                'Signed_Int16_LE_Offset_32';
                                'Unsigned_UInt32_LE_Master_Offset_0'
                                }
    SelectedChannelGroupNumber: 1
                      ReadSize: 5 sec
                    Conversion: Numeric

for ii = 1:3
    data = read(mds)
end
data=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    0 sec                0                                0                 
    1 sec                1                                1                 
    2 sec                2                                2                 
    3 sec                3                                3                 
    4 sec                4                                4                 

data=5×2 timetable
    Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    _____    _________________________    __________________________________

    5 sec                5                                5                 
    6 sec                6                                6                 
    7 sec                7                                7                 
    8 sec                8                                8                 
    9 sec                9                                9                 

data=5×2 timetable
     Time     Signed_Int16_LE_Offset_32    Unsigned_UInt32_LE_Master_Offset_0
    ______    _________________________    __________________________________

    10 sec               10                                10                
    11 sec               11                                11                
    12 sec               12                                12                
    13 sec               13                                13                
    14 sec               14                                14                

Close MDF-Files

Close access to the MDF-files by clearing the MDF datastore variable from workspace.

clear mds