How to optimize imagedatastore speed?

6 views (last 30 days)
Andrew Jamieson
Andrew Jamieson on 14 Feb 2018
Commented: Andrew Jamieson on 19 Feb 2018
I have close to 1 million images I'd like to prepare with imageDatastore (imds) for CNN deep learning training. I would like to create the imds via giving a cell array of the specific image paths.
These Paths are split among 1000s of directories or so. When creating a sub-set as a test, say around 30,000 images, it takes imageDataStore around 30sec or so to initialize. If I try all 1 million, it was taking longer than 2 hours! I did not wait for it to finish. I checked the code profiler to see what the issue was and it turned out to be that it was always checking the dir of each image. (see image)
My alternative was to simply not use imds all together and load 160GB would of images into memory!
Suggestions please. Is there a way to disable this directory checking mechanism?
Thank you.

Accepted Answer

jiro on 16 Feb 2018
I'd like to know a bit more about the folder structure and how you are splitting/choosing your sub-set. The reason I'm asking is that instead of providing a list of image paths using cell arrays, it's much more efficient to provide a list of folders (even better, the top folder) and ask imageDatastore to search through subfolders. Then, in addition you can use splitEachLabel to divide up your image datastore into training and testing set.
For example, I created 1 million images split across 5000 directories (5000 directories with 200 images per directory). Here, each directory corresponds to a category.
To obtain an image datastore for all 1 million images,
imd = imageDatastore('20180216T085544\',...
Elapsed time is 140.757894 seconds.
Certainly, much shorter than 2 hours. Then, to split the datastore into training (70%) and testing (30%),
[imdTrain,imdTest] = splitEachLabel(imd,0.7);
Elapsed time is 26.851960 seconds.
Andrew Jamieson
Andrew Jamieson on 19 Feb 2018
Thanks. The feedback I would have is to perhaps offer a flag/parameter/option for imds to NOT search every directory when given an explicit list of files with the full path already established. In other words, if possible also optimize for a list of files, not just the directory approach.
Also, if we are talking feedback, the ability to define a label for the entire imds in one line would be nice (maybe this is possible and I am confused) but it is an extra step to create a cell array or categorical array matching the list of files, whereas, if I could just do the following:
imdsTrain.Labels = myLabel;
But this is a minor quibble. Thanks again!

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by