Loren's blog on datastore. How should I use counterSize and done?

I am trying to use datastore in Matlab v2018b but it does not have 'partialread'.

I am trying to replicate Loren's code (https://blogs.mathworks.com/loren/2019/05/29/big-data-in-mat-files/ and the section 'Use a portion of all variables') for a large variable whose size exceeds memory.

Unfortunately its written kind of poorly with many variables thrown in and not explained. For example, I cannot figure out how to use 'done' and 'counterSize'.

I tried

[fds_partial, counter, done] = fileDatastore("smallVars*.mat", "ReadFcn", @partialReadFcn, "UniformRead", true);

But I get an error of 'too many output arguments'

I tried

[M, counter, done] = read(fds_partial) %with
fds_partial = fileDatastore("smallVars*.mat", "ReadFcn", @partialReadFcn, "UniformRead", true);

Again I get an error of too many output arguments.

So I end up reading only one block of file and the readfcn function does not proceed beyond one block.

So what gives?

[Image Analyst]

Did you try emailing her?

Answer 1

[Adam Danz]

> I am trying to use datastore in Matlab v2018b but it does not have 'partialread'.

partialreadFcn is a function defined in Loren's blog post. No currently existing release of Matlab has a "partialread" function.

> I cannot figure out how to use 'done' and 'counterSize'.

done appears to be a flag (true|false) indicating whether there is more to read or not.

There is no variable named counteSize.

> But I get an error of 'too many output arguments'

fileDatastore only has 1 output.

doc fileDatastore

Answer 2

[gujax]

I think I figured out some of the answers...

I need later versions of Matlab to get 'partial read' enabled using 'read mode'.

Also though 'done' and 'readcounter' can used in the partialread function, I am still not sure where these two variables get outputed and how I should be using them.

So its kind of useless to go through filedatastore when one can do this simply by re-reading partial files. Not sure how all this helps.