Alternative to Nested Parpool

I have some code in which I need to call a function multiple times, by iterating through a for loop. The function itself runs in parallel, and I would like to use parallelization on both the outer loop, and inside the function. I'm running this on an HPC, so I have plenty of cores available, but each function call will use a significant portion of memory, so that I cannot use all of the cores on the outer loop. All the answers I've seen say that matlab doesn't support nested for loops. I can submit each element of the outer for loop as its own job in the HPC, but I was wondering if there was a better option.

[Walter Roberson]

Are you running directly on the HPC, or are you using DCS (Distributed Computing Server) ?

[Kevin Johnston]

Directly on the HPC.

Answer 1

[Matt J]

Instead of doing this

     parfor i = 1:10
        MyFun(i)
     end

    function MyFun(i)

      parfor j = 1:5
          func2(i,j);
      end
    end

you should try to reconfigure MyFun so that it can do a specified subset of iterations. Then you can re-implement with a single parfor loop as follows:

     parfor k = 1:50

        [j,i]=sub2ind([5,10], k);

        MyFun(i,j);

     end

    function MyFun(i,Jsubset)

      for j=Jsubset
       func2(i,j);
      end

    end