Virtual Datasets Source remapping or autoupdate

I am using Matlab's low-level hdf5 functions to map multiple chunks of an image stored in .h5 to a VDS. The problem I am having is when the vds file is moved, the mappings remain the same(expected). I was hoping see if there was a way to have the mappings be dependent on the vds file and not static to the folder from which it was mapped. Something along the line of:

Source files:/1/data_folder

VDS: /1/vds.h5

If the folder was changed to /1_new/ the mappings would be reflect based on the new location of the VDS: /1_new/vds.h5

If this isn't possible, is there a way to remap to new locations without creating a new vds or even just convert a VDS to a regular HDF5?

Edit:

Apparently, HDF5 actually has support for relative path prefix where it can be relative to the VDS itself, which might facilitate a VDS that can auto update its source locations. Documentations below. The challenge, however, is that MATLAB does not currently(as of my knowledge) have support for these low level functions, so I have not gotten a chance to try and test. HDF5 has a C library. Does anybody have an idea of how I can utilize this library in MATLAB?

H5Pset_virtual_prefix

H5Pset_efile_prefix()

Answer 1

[Hoang-Le Tran]

When mapping the VDS, it is possible to use relative pathing when using H5P.set_virtual() without using the prefix functions. As long as the provided path does not start with '/' in unix, the relative path of source files will be prepended with the current directory of the VDS file. Documentation here: H5Pset_virtual in the 'source file resolutions' section.

Answer 2

[Ayush Modi]

Hi,

As per my understanding, you are trying to create a dynamic mapping of multiple chunks of data stored in .h5 file to VDS.

However, VDS mappings are static once created; they don't automatically update if the source files are moved to a new location. This means that if you move the source files or the VDS file to a new location, the VDS mappings will still point to the original file locations and will not reflect the new paths.

However, you can convert VDS to a regular HDF5 file. You can read the data through the VDS and write it to a new regular HDF5 file. This way, you lose the virtual mapping, but you create a standalone HDF5 file that contains all the data and does not depend on the source file locations.

Here is an example to demonstrate how you can accomplish this:

% Read the data from the VDS dataset
data = H5D.read(dset_id, 'H5ML_DEFAULT', 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT');
% Creating a new HDF5 file and writing the data
new_file_id = H5F.create('new_regular_file.h5', 'H5F_ACC_TRUNC', 'H5P_DEFAULT', 'H5P_DEFAULT');
dataspace_id = H5S.create_simple(ndims(data), fliplr(size(data)), []);
new_dset_id = H5D.create(new_file_id, '/dataset', 'H5T_NATIVE_DOUBLE', dataspace_id, 'H5P_DEFAULT');
H5D.write(new_dset_id, 'H5ML_DEFAULT', 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', data);

Please refer to the following MathWorks documentation for more information on the following functions:

HF5 - https://www.mathworks.com/help/matlab/ref/fileh5f.html

H5D - https://www.mathworks.com/help/matlab/ref/dataseth5d.html

I hope this resolves the issue you were facing.

[Hoang-Le Tran]

Hey Ayush,

Thanks for chiming in. I just found today that HDF5 actually has support for relative path prefix where it can be relative to the VDS itself, which might facilitate a VDS that can auto update its source locations. Documentations below. The challenge, however, is that MATLAB does not currently(as of my knowledge) have support for these low level functions, so I have not gotten a chance to try and test. Any input on how I could call the whole HDF5 library within MATLAB? Maybe through a java wrapper or C Mex. I have no idea on how MATLAB supports external libraries.

H5Pset_virtual_prefix

H5Pset_efile_prefix()