Running MATLAB efficiently from windows command line for faster "unzip and read" process.

I have a pretty large amount of data (~63,000 .zip files). In each zip file, there is a .txt file. These vary in size, but typically, when uncompressed, are around 7-10MB in size, so obviously unzipping them all at once is by no means ideal!

I've been looking into some better ways to unzip: MATLAB's "unzip" function is extremely sluggish. From what I can tell from the tests I've done: python's zipfile.ZipFile.extractall() runs in about 3% of the time of the MATLAB equivalent! So using Python to unzip is the route I want to go down. I would do all my data handling in Python, but I need to merge the data in the zip files with data from an external source which are in .mat files. So my end product, after unzipping and reading these .zip files, is to have ~63,000 .mat files containing only the data I need.

What I wish to do is:

• Unzip using Python
• Manipulate the Data (Calling MATLAB script from Windows command line via Python)
• Save what I wish to keep (using the same MATLAB script)
• Delete unzipped .txt data

Is there a fast and efficient method to call a MATLAB script from the command line? I've tried:

matlab -nosplash -nodesktop -r ScriptToRun.m;quit();

but this opens the window, runs the script then closes the window again. It seems to take a couple of seconds to actually open the MATLAB window before it gets down to running the script.

Any ideas on how I could make this process more efficient?

Note: I'm aware of the ability to call my Python methods using MATLAB, but I'd rather do it the other way about so as to use the multiprocessing/threading modules in Python and make use of all the CPUs on my machine.

Thanks in advance!

[Walter Roberson]

Note: you should call with

matlab -nosplash -nodesktop -r 'ScriptToRun;quit();'

and better yet,

matlab -nosplash -nodesktop -r 'try; ScriptToRun; catch; end;quit();'

Answer 1

[per isakson]

It might be an alternative to run Matlab as an activeX Server.

Answer 2

[Jason Ross]

You could start MATLAB, and have it poll a directory for files to process and output. This would save you the startup costs, since it would already be running and looking for work.

You might also want to take a look at "unzip.m" and see if you could replace, overload, or make an alternate implementation (e.g. myunzip.m") that would call the python unzip instead. Disclaimer: This might get tricky.