6 views (last 30 days)

I have large files that contain interleaved complex data. I reach chunks of it at a time for processing. It seams that there is no way to read this efficiently. I can't find out how to do it without a copy that shouldn't really be necessary. Here are some things that I tried (y.fid points to a file opened with fopen(), y.type is a valid type):

% method 1: read in to a vector, interleave with complex

tic

cData = fread(y.fid,numSamp.*2, y.type);

t1 = toc;

tic

cData = complex(cData(1:2:end),cData(2:2:end));

t2 = toc;

The above method is slower than it has to be. I think the indexing really slows down the copy into cData.

% method 2: read into 2xN array, use complex and indexing

tic

cData = fread(y.fid,[2 numSamp], y.type);

t3 = toc;

tic

cData = complex(cData(1,:),cData(2,:)).';

t4 = toc;

The above method is indeed faster. It still requires a copy.

% method 3: read into 2xN array, use complex muliply

tic

cData = fread(y.fid,[2 numSamp], y.type);

t5 = toc;

tic

cData = cData.' * [1 ;1j];

t6 = toc;

The above is the slowest of them all.

I tried memmap, but it's inappropriate for very large files. I looked into MEX files. It requires that you create a new complex array for every read, but you could probably fill the array efficiently. The problem is that this is worse than an fread to an existing array of the correct size because one has to allocate new memory for every call.

Thanks in advance.

James Tursa
on 30 Jun 2020

Edited: James Tursa
on 1 Jul 2020

A mex routine to accomplish this that doesn't use any hacks can be found here:

It uses the MATLAB fopen( ) function in the background, so the syntax is the same. It first reads the interleaved complex data file as a real variable with twice the first dimension, then internally converts this to a complex variable with the requested dimensions using a sleight of hand pointer manipulation and no extra data copy. Uses only official API functions and does not require any mxArrary hacks, so should be OK for any MATLAB version R2018a or later.

There should probably be an enhancement request submitted for this, since I think the MATLAB fopen( ) function should have this capability directly without the need for a mex routine.

EDIT

And I just updated this submission to include a related fwritecomplex mex routine for writing interleaved complex variables. It creates a temporary pseudo shared data copy of the interleaved complex variable as a real and then calls the MATLAB fwrite function. All using official API functions so no mxArray hacks involved.

And, again, I think the MATLAB fwrite( ) function should have this capability directly without resorting to mex routines.

dpb
on 29 Jun 2020

Edited: dpb
on 30 Jun 2020

Well, dunno how much faster/slower it might be; didn't try to time it, but trying something different--

Iffen by "interleaved" you mean the data were written something like

c=complex(rand(10,1),rand(10,1)); % make a dummy array for playing around with

fid=fopen('complex.bin','w');

for i=1:10,fwrite(fid,[real(c(i)) imag(c(i))],'double');end

fid=fclose(fid);

fidr=fopen('complex.bin'); fidi=fopen('complex.bin'); % handle for real, complex parts

fseek(fidi,8,'bof'); % position file pointer for first complex

complex(fread(fidr,inf,'double',8), fread(fidi,inf,'double',8))

fclose all; clear fidr fidi

Reproduced the original c array here...

Alternatively, you could rewind() the file and then read the second.

Whether the i/o would be quicker than the indexing operation I dunno...

Fortran handles it with reading into a complex variable--you might consider mex using Fortran instead of C

James Tursa
on 30 Jun 2020

James Tursa
on 30 Jun 2020

Edited: James Tursa
on 30 Jun 2020

This may not apply to you, but if you have R2018a or later you can just fread( ) into a real variable directly the interleaved data and then use this FEX submission that reinterprets the real variable as a complex variable with half the number of elements using the supplied real2complex mex routine. This returns a shared data copy so it is memory efficient. It uses unofficial hacks to accomplish this because MATLAB does not supply official API functions for this. You need to have a C compiler installed to compile it.

Hopefully MATLAB will eventually supply an official function for this and the reverse function, since it is a natural capability that many users will want.

James Tursa
on 9 Jul 2020

Well, your questions on how this works are certainly legitimate and not out of line, so no apology needed. Particularly in light of the fact that these details are not published in MATLAB documentation, so how could you even know?

But getting back to your point of memory allocation, your original supposition is true ... a new memory allocation is in fact required by the mex routine each time the file is read. My points were that (1) this is also true of m-file code so no disadvantage of the mex routine here, and (2) no extra copy is made by the mex routine when returning plhs[0] back to the caller. The freadcomplex performance gains are purely the result of being able to convert a real variable to a complex variable via pointer manipulation without requiring a data copy.

The mex routine posted in the FEX plays by the rules and allocates new memory each time it is called, because it calls MATLAB fread( ) in the background and that is what MATLAB fread( ) will do. There is a way to avoid that memory allocation each time, but you would have to abandon MATLAB fread( ) and write the code using the C fread( ) function, and you would have to modify a MATLAB variable inplace (reading data directly into its data area) which would violate the rules (you could inadvertently write into other variables sharing data with the one you modified). As long as you didn't care about potential side effects (e.g., you never use those other shared variables downstream in your code) you can get away with this. Or you could hold the varialbe inside the mex routine and return a shared data copy of it (which also violates the official rules). It might be interesting to see what kind of speedup this would get. But given your aversion to unofficial methods, I didn't write that.

Opportunities for recent engineering grads.

Apply TodayFind the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!
## 5 Comments

## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918241

⋮## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918241

## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918346

⋮## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918346

## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918433

⋮## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_918433

## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_919066

⋮## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_919066

## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_919072

⋮## Direct link to this comment

https://la.mathworks.com/matlabcentral/answers/556855-efficient-ways-to-read-and-write-complex-valued-data#comment_919072

Sign in to comment.