Converting C++ bits packed int** to MATLAB unisgned char mxArray using mex?

Hi,

Could someone help as I'm loosing my mind in determining what's wrong here...I have a C++ array of int values (int** pColors - provided from the Kinect SDK mesh->getColors(&pColors) call - but that shouldn't matter IMHO) which are just triplets of byte sized 0-255 (0xFF) rgb color values (that work and can be read/unpacked fine in the C++) that I would like to now read into and use in MATLAB as a mxArray which will be part of a struct that also holds a number of coresponding vertices that have those coresponding rgb colors each (so the number of vertices equals number of colors i.e. numVertices=numColors)...I have tried preety much anything and almost everyting fails and/or brings down MATLAB (no doubt I try to access parts of memory that aren't reserved).

This is what I have used to create a colors mxArray to store the valid C++ data (I can read and unpack it fine in some C++ code):

int colorsIndSize[2]={1,numColors};
colors = mxCreateNumericArray(3, colorsIndSize, mxUINT8_CLASS, mxREAL);

But after that I had to use this to get a C++ pointer for use in my mex code (mxGetPr instead of mxGetUint8s because I can't use '-R2018a' as I inherited a lot of code and adding that gets a lot of errors):

unsigned char* colorsptr = (unsigned char*)mxGetPr(colors);

So now to traverse a real C++ data and store that in the colors mxArray using the corolsptr pointer (by staying in line with the coresponding vertexIndex) I use:

for (unsigned int t = 0, vertexIndex = 0; t < numVertices / 3; ++t, vertexIndex++)mex
{
    unsigned int color0 = pColors[vertexIndex];
                            
    if(color0>0) mexPrintf("Found one?\n");
                            
    colorsptr[t*3] = ((color0 >> 16) & 0xFF);
    colorsptr[t*3 + 1] = ((color0 >> 8) & 0xFF);
    colorsptr[t*3 + 2] = (color0 & 0xFF);
}                     

But this just won't give me the result (it just hangs, it seems to start as it writes out something mexPrintf once but I never see it finish now)...I have also tried a number of other values for dimension sizes and I just can't figure out what is the problem...can anyone please help?

Maybe I should focus on debug via Visual Studio but I can't attach to the MATLAB process for now for some reason, even though I tried, or maybe I'm just not understanding the pointer arithmetic right and what mxArray the mxCreateNumericArray creates and how to access it's items by adding and multiplication yet, or is that use of unsigned char instead of uint8 which I think are the same size is doing something bad?

TIA

P.S.

At one point as I used various sizes and traversing options I did get an output from the code in question but it was crazy sized like 3 x numVertices x 129 (which I don't get how would I get 129 if this was supposed to be 8 bit colors)...althought that code variant was traversing 3 colors at the time as does some original code that works on C++, but in this case I'd settle for just getting even a simple 1 x (numVertices*3) or 3 x numVertices which has the colors unpacked from the C++ int** array.

Answer 1

[James Tursa]

Some issues:

1) The signature of mxCreateNumericArray according to the doc is:

mxArray *mxCreateNumericArray(mwSize ndim, const mwSize *dims,
    mxClassID classid, mxComplexity ComplexFlag);

So why are you using an int for that second argument when it should be mwSize? This may be a 4-byte vs 8-byte integer size mismatch. Could easily be the cause of a crash.

2) This code:

int colorsIndSize[2]={1,numColors};
colors = mxCreateNumericArray(3, colorsIndSize, mxUINT8_CLASS, mxREAL);

The colorsIndSize array has two elements, but on the very next line you tell mxCreateNumericArray that it has three elements. This will cause mxCreateNumericArray to read off the end of the array into invalid memory. This could also easily cause a crash. You probably meant this instead:

mwSize colorsIndSize[2]={3,numColors}; // changed 1 to 3, and int to mwSize
colors = mxCreateNumericArray(2, colorsIndSize, mxUINT8_CLASS, mxREAL);  // changed 3 to 2

3) I'm confused. You mention more than once that you have a C++ int** array, but I don't see any such thing in your code. What exactly is the definition of pColors?

4) Is numColors equal to numVertices/3? If it isn't then you have another problem.

[Dejan Dimitrijevic]

Hi James,

First thank you, for the prompt reply, I'm gonna look into everything...note now...1) I started from some legacy Kin2 code...unfortunatelly it was written prior to -R2018a (which I have since yesterday mostly rewritten) and now I got this error after using the mwSize and getUint8s instead of the other types and mx methods previosly used:

Requested 1x85296x1414848 (112.4GB) array exceeds maximum array size preference.
Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. 
See array size limit or preference panel for more information.

I'll be going through all of the other answers and changing the code acordingly if it's not yet been rewritten, but I hoped that the above message may be giving some more insight where the code I wrote made a mistake and if it could mean to an experienced mex developer something more than me right now.

As soon as I go over the other answers and I hopefully now setup debugging I'll get back to you if anything more pops up.

Here's the rest as promised

2) Finally, thank you, really so much, I was wondering why, that Kin2 code had 1, when it for vertices had 3, but I see now in matlab documentation 1's become 2 automagically if the *dims has more than 1 element...is that right? Based on that I now managed to get some values successfully read into a MATLAB matrix consistently, but it seems the C++ code either isn't giving regular values as it is when I use it from my C++ code or that mxUint8 I'm now using is cutting it off as I always get 0s...just a side note question, so I understand the mxCreateNumericArray better now, if I have *dims array of three or more elements, do I still get to access those as I would a regular threedimensional array using pointer arithmetic the normal way or would it be something more exotic for MATLAB?

3) The C++ int** I have is straight from the Kinect SDK's INuiFusionColorMesh::getColors(&pColors) docs...as are the vertices via getVertices...and their numbers should be the same if Kinect Fusion worked.

4) numColors is equal numVertices, but each pColor holds three rgb values which I initially just wanted to place into even a single MATLAB row vector which I'd resize then to the value needed afterwards in MATLAB code (just as soon as got the values which C++ Kinect SDK code obviously was returning right to me in C++ code...and which I'll now try to further integrate into MATLAB further...thanks to your helping suggestions).

[James Tursa]

Can you post your current code? Typically when I see outlandish array size allocation errors it is because the requested sizes are still not being passed in correctly.

Generic comments on pointers returned by API functions: For mxArrays the mxGetPr( ) function (and friends) always returns a pointer to the first element of the array, regardless of the number of dimensions. You can always use linear indexing with this pointer to access all of the array elements. The elements are always arranged in "column" order ... i.e. the first index varies fastest, etc.

"... I see now in matlab documentation 1's become 2 automagically if the *dims has more than 1 element ..."

So, this information is simply telling you that all mxArrays have a minimimum of two dimensions, and if you only give a mxCreateEtc function one dimension then the function will automatically create two dimensions with the 2nd dimension set to 1. You do not have this situation in the code you have posted.

[Dejan Dimitrijevic]

Thanks for your help, here's some code, current version...unfinished...one note on it's use first, you'll need KinectV2 SDK and .xef recording (or a KinectV2 to stream in the data to a processing started by process.m file in the found in the zip root - I can also provide the .xef tomorrow to be used with the Kinect Studio and without Kinect sensor itself - but since it's not going to be small for even a 1sec recording that might take a while to find a place to host first).

It also uses the Mex which has to be compiled first (can be done without the SDK as I've included some original Microsoft's dll's in the zip too)....

And what I've noticed today, is that the basically same code when is used outside of the the MATLAB environment but compiled in VS2019 as an command line .exe works fine and that getColors(&pColors) call gets the const int** colors collection fine and that in MATLAB it somehow doesn't, so my current guesses are that maybe some of the biolerplate code I started from the Kin2 project is to blame (that class handle and assert templated code which I didn't bother to look much until now but maybe I should've because even it didn't have the color mesh data code in it) or MATLAB itself can't handle int** right (which I doubt) or that it is returning data from execution only after it releases the memory (but I don't think that also as that is not being printed out whilst KF_getMesh MATLAB function is working but as I said 11111111 00000000 00000000 00000000 over and over again for each vertex/color).

TIA

Answer 2

[Dejan Dimitrijevic]

Just another follow-up, with a small question, how does MATLAB run C++, and treat it's types in comparison...I've now confirmed I get zeros, and never do I get other 0-255 color values...I've done it by manually unpacking and writting all the bits from colors values I got from that getColors SDK call which returns int** pColors...but (even though I'm now sure what's inside of it at the last time of read is basically a bunch of 11111111 00000000 00000000 00000000 - first 24 are thus 0) I've been however able to write some non-zero values when using something else other than UINT8 like when using colors = mxCreateDoubleMatrix(3,numColors,mxREAL); which will give me some random non-zero values in there somehow (although in that case I think I'm just geting jiberish, although I'm guessing 64bit since it's only filled up until 1/8th of the mxArray but the values there are non-repeatable which is even stranger to me and more confusing on how to properly access the mxArray items) as I'm not sure why would using unsigned char* colorsptr = (unsigned char*)mxGetPr(colors); in conjunction with the above mxArray be any different then when used with the UINT8 mxArray code...in which case colors definition above gives me all-0s (which probably is true as I've printed it out to see what I'm reading from the getColors set values...maybe MATLAB isn't able to set that although when running more or less the same C++ code works and it prints out regular color values, unlike in MATLAB when everything else gives me 0, including that mexPrintf of the manually shifted pColor[vertexIndex] values)?

TIA again

P.S. To make it stranger I've been able to read vertices, normals and triangle vertex indices ok though from this Kinect SDK code running via mex in MATLAB though, just not colors, which work outside...

[James Tursa]

Does the definition of pColors look like this?

int *pColors;

[Dejan Dimitrijevic]

Yes, that works, in regular C++, passing in &pColors then, to the getColors SDK method. I've attached some code to the comment above which is still a work in progress so don't judge please for overuse of bool checks and even trying to get it working in multiple calls as the first fails...it's only despair...etc.

Answer 3

[Dejan Dimitrijevic]

Hi James,

Hopefully you're still there now....having now taken out the boilerplate code I had used from the Kin2 MATLAB Kinect toolbox project I really can't tell what's wrong anymore as I just can't get anything else than zero values for colors when the exact same code compiled and run outside of MATLAB works fine...maybe it's truely something to do with MATLAB's inability to assign const int** values by passing a reference or something else external maybe (could it be Microsoft's Kinect v2 SDK has a bug but why would the code work then outside of MATLAB .mex as a standalone .exe please?)

I've included this simpler code example and deleted the one using that boilerplate code which I thought was maybe responsible for the odd behaviour (because that previous version and its call would just as Kin2 Fusion demos only work once and crash MATLAB the second time so I though that a memory leak in it may be causing something to affect the getColors...turns out I was wrong...it still gives me zero values even without that now taken out boilerplate code which was used only to make the mex function call more class->method like).

If you or anyone else has any ideas (I'll try to contact Kin2 framework author too to see why didn't they include color mesh data in their Kinect Fusion demo) I'd appreciate it...I doubt wrapping the getColor() method in C++ code to return anything else would be the only other thing left to try...but I doubt I'll get another result as the 0s returned by getColors initially would just be prepackaged into another object without affecting the bug that returns 0s in the first place?

Best rgds and TIA all,

D.

P.S.

The code also includes an even simpler version of the K3d4,cpp file called K3d4a.cpp (that's compiled using compile_cpp_a.m MATLAB script) which was suppose to be the start of the wrapper code above for me but for it doesn't work because I forgot to to take out something, at line 97, replacing old 4 struct members with just 1 plhs[0] = mxCreateStructArray(2,dims,1,field_names);...

[Guillaume]

"or anyone else has any ideas"

When I saw this question is the list of recent questions, it showed that it had 3 answers with an answer by James Tursa accepted. To me that says, no need to look at it, it's already been looked at by at least 3 people and solved by James, who knows what he's talking about. So, I suspect there won't be many anyone else looking at it.

The only reason I looked at it is because it mentions C++, but I don't see any C++, just plain C.

[Dejan Dimitrijevic]

The original crashing trouble has been resolved by using James's suggested mxCreateNumericArray dims values, however new trouble is (though now seemingly unrelated but I thought it was maybe related somehow last week) that the method from Kinect 2 SDK which was according to Microsoft written in and can be used with C++/C is not returning the same values as it does when it works when compiled as a standalone .exe in VS2017...I didn't want to make a new question in case the two problems may have been related and the troubles above could give some more context, but as this is not the case I have made a new one (if that's needed to be looked at and answered by an MVP)... Additionally if there's a bugzilla I'd like to report a possible MATLAB bug as I currently see no other reason for the code in the .zip provided above to work fine as a standalone .exe but not as .mex (or is that wrong reasoning)?