MATLAB Answers

Matlab Engine Array Transfer Size Limitation

7 views (last 30 days)
Jonathan
Jonathan on 11 Nov 2015
Commented: Walter Roberson on 11 Nov 2015
I am using engGetVariable() to transfer an mxArray from the Matlab Engine to my C++ program. The documentation says: "The limit for the size of data transferred is 2 GB."
Why is this limitation present? Is there a way to increase it without breaking up the array into separate arrays? I need to transfer large arrays, potentially up to 256GB.
I like the simplicity of the Matlab Engine, so I would rather avoid having to break up, transfer, and re-assemble large arrays.

  3 Comments

James Tursa
James Tursa on 11 Nov 2015
256GB is a lot of memory to copy. Have you considered a mex routine instead, so you can avoid the copy altogether?
Jonathan
Jonathan on 11 Nov 2015
Yes, actually I wrote one. The problem is that I need to solve a large, sparse matrix (Ax=b) on a cluster using Intel Pardiso Sparse Cluster solver.
Unfortunately, integrating Matlab with MPI and using the unsupported Intel MPI compilers seems infeasible.
In fact, I wrote a test program that solves the equation without the MPI code, compiled with GCC using MEX, and it exhibits some weird memory behavior.
So from the above, I concluded that getting this to work in MEX, running on a cluster with MPI is perhaps infeasible.
With regard to transferring 256GB, since I am doing the transfer within a single node (not over a network), it shouldn't take more than a few seconds in theory.
However, are you suggesting that I write my own MEX function that bypasses Matlab Engine to transfer directly using Linux IPC routines?

Sign in to comment.

Answers (1)

Walter Roberson
Walter Roberson on 11 Nov 2015
The 2 Gb limit is for compatibility with 32 bit versions of MATLAB.
You should be considering using memmapfile() or shared memory

  0 Comments

Sign in to comment.

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by