Optimum file size in a file datastore

Hello together,

I'm storing a lot of *.mat files that I'm storing in a file datastore. The size of the files is pretty flexible as I convert some text files before. I used to vary the size of these mat files which had a huge impact on the speed of processing my requests with the tall array. However, I would like if there is a "Matlab optimum value" and how to find it out without going through an iterative process.

PS: I'm using v7.3 files and noticed that the processing speed was increasing with increasing file size but that at one point, the size of the files was not growing linearly with the length of my stored variables anymore.

Thank you.

Answer 1

[Eric Martin]

I've just run following test:

• create a timetable (2 variables) of 15 GB and split it into several files
• Test 1: 15 GB in 10 files
• Test 2: 15 GB in 100 files
• Test 3: 15 GB in 1000 files
• Test 4: 15 GB in 10000 files

After that, I use fileDatastore to extract the mean value of the first variables. Results are as follow:

• Test 1: 183 seconds
• Test 2: 95 seconds
• Test 3: 95 seconds
• Test 4: 141 seconds

It is clear that there is an optimum "chunk size" but I still have no clue how to determine it in a non empiric way.