Optimized Iterations and Permutations

I run a good number of simulations that so far I have been doing with deeply nested loops but I figure there has to be some more optimized way to handle this. I have a "base" simulation that takes in a number of parameters and I want to run it for permutations of those parameters. For example here is the format I have been using:

% Variables to iterate
A = [1:10];
B = [5:-1:0];
C = [1,4];
D = [100:0.1:1000];

for Ai = 1:length(A)
    for Bi = 1:length(B)
        for Ci = 1:length(C)
            for Di = 1:length(D)
                % Lines and lines of simulation code
                OutputData(Ai,Bi,Ci,Di) = MySim(A(Ai),B(Bi),C(Ci),D(Di));             
            end
        end
    end
end

I generaly have >10 of these variables to iterate over so managing them and dealing with the output data becomes a hassle. Is there a better way to do this type of analysis?

Answer 1

[Walter Roberson]

WIth just 4 levels, you already have more than 1 million elements. If you have an average as low as 4 elements for each of the remaining 6 parameters, you would be looking at roughly 4 billion function evaluations, and just storing the outputs would take on the order of 32 gigabytes.

This is going to take time to do for non-trivial function MySIm() .

You might see in other postings that people often use ndgrid() to build arrays of all of the combinations, and then use arrayfun() to execute the function over each combination in turn. I estimate that the memory to store all those combinations would be on the order of 350 gigabytes in your situation -- not feasible.

I would therefore suggest that you use what has sometimes been called an "odometer" pattern. I posted generalized code for that at https://www.mathworks.com/matlabcentral/answers/623358-get-a-combination-of-unique-paths-for-given-pair-of-numbers#comment_1082638

odometer pattern execution is not the fastest of executions (significant amounts of overhead in iterating through all of the possibilities), but has the advantage of taking very little memory, and of clearly eventually finishing.

Oh yes, it is crucial that you pre-allocate the output arrays.