How to SAVE data to disk in a background thread?

Has anyone figured out a way to run the SAVE command in a background 'thread', so that MATLAB can keep on running code while the file is getting written to the disk?

Answer 1

[Jan]

You can call a MEX-function, which opens a new thread, and the main thread returns to Matlab. The thread running in the background is not allowed to call any Matlab functions, so anything starting with "mx..." and "mex..." must be avoided. When you want to write a binary file, fwrite is fine.

You have to be sure, that the data are not changed during writing. E.g. a pointer to the data of a Matlab array might lead to inconsistencies, when the data are modified or free'd during the writing. But this concerns all multi-threading applications, such that I assume, that you have solved this already.

[Eric Sampson]

Jan, I need to write a MAT file instead of just a binary file. In the MEX funtion's background thread, do you think it would be possible to start up a MATLAB Engine instance and do the SAVE from there?

[Jan]

I cannot imagine that this could be faster than doing this in a single-threaded application. You can create a standalone application, which is able to save MAT files. Perhaps you can forward the values using http://www.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix.

[Eric Sampson]

Hi Jan, I understand that the SAVE operation itself would not be any faster, and that's OK - the situation I'm encountering is that we are writing a lot of data to disk, so each call to save can take 20+ seconds. I'd like to find a way to initiate the save in the background and then immediately return to executing the next line of our code...

Answer 2

[Image Analyst]

I don't have it, but doesn't the Parallel Computing Toolbox have that capability?

[Eric Sampson]

I think you're right IA, looks like I could use the PCT to fire off a batch job on a local worker that runs a function to save the variables to disk. I just would need to justify the toolbox cost :) This seems like basically similar to the 'home grown' approach described above of using a MEX file to spawn a MATLAB Engine process which then does the save. Just a decision of 'build or buy' I guess!