FFT slowdown even after workspace reset

I'm experiencing behavior with the fft() function that is causing me to have to restart Matlab between executions of a long script that is both processing and memory intensive and requires, among other things, millions of fft's on the CPU and GPU. If I run bench() prior to running the script, my computer (i9-13950HX w/64GB of ram, running Windows 11, Matlab R2024a) clocks in very fast. After I run my script, all performance metrics are basically identical except for fft() which clocks >10x slower than before.

No matter what I do to the workspace (clear all, clear classes, clear functions, close all hidden force, clc, reset(gpuDevice), etc.), or the fft planner I cannot bring the performance of fft() back to what it was before execution of the script.

Am I overlooking anything that could reset the performance of the fft short of restarting Matlab itself? I would like to let the computer loop over a bunch of datasets but right now the slowdown in the fft is making this very inefficient. I am currently considering calling the Matlab engine from Python so that I can restart it between script calls to prevent this. I am running Matlab on 2024a and may be able to update to 2024b but cannot upgrade past 2024b.

[dpb]

I believe probably only Mathworks can address this. Contact official support request at <Product Support Page>

[Paul]

@Timothy

If the culprit is identified, would you mind posting back here as others may be interested in the findings.

[Walter Roberson]

I note that the i9-13950HX has 8 performance cores and 16 efficiency cores. I wonder if later in the run, most of the computation is being shunted to the efficiency cores?

[Timothy]

@dpb, thanks I might give that a shot.

@Paul, I found the culprit function, which calls another function which splits a bunch of large complex double precision N x M arrays into three dimensional 32 x M x (N/32) arrays, computes the FFT in the column dimension, multiplies with the conjugate of another 32 x M x (N/32) array, inverse Fourier transforms & normalizes to create a bunch of cross-correlations. However, it seems that if I isolate this sub function and run it a bunch of times by itself it doesn't affect the performance of the fft() function. Still, if I delete the sub function from the larger function I also have no reduction in performance, so I know that it is tied to it somehow. If I can get more specific or create a simple toy function that creates the fft performance loss I'm seeing I will post it here.

@Walter Roberson: Maybe this is happening, I don't know how to monitor what cores are used. Matlab is the main process on my computer however, and this slowdown can be created in under 10 minutes of operations by iterating the function described above in a loop. If I call the bench in each iteration of the loop and store the FFT time I can watch it slow down each iteration starting with the 4th (I get about 25 iterations in 10 minutes, by which time the FFT score has gone from ~0.15 seconds to ~1 second, and keeps slowing down the more times I call the function).

[Paul]

@Timothy

fft used to have a memory leak, but that was fixed. Maybe a similar, yet different, issue has reared up since then. Matlab 2020a/b fft function memory leak - MATLAB Answers - MATLAB Central

Also, this thread How can I solve memory leak in fft? - MATLAB Answers - MATLAB Central, which isn't really about a memory leak, discusses memory management with fft and seems like it might be on point based on the problem description. Maybe the

fftw(wisdom,[])

command is worth a try. Though it sounds like all of the FFTs are 32-point, so maybe this isn't the issue.

[Walter Roberson]

@Paul I think you mean

fftw('wisdom',[])

[dpb]

The doc for <fftw> is may be a little confusing for that case -- it does show the form as @Paul used, but uses wisdom as a place holder for either 'swisdom' or 'dwisdom'. 'wisdom' alone isn't documented but doesn't error on local system...but it does need to be a character string (or a variable that would contain the string).

[Paul]

Yes, as dpb suggested I was using wisdom as a variable that would have the value ‘dwisdom’ or ‘swisdom’ as appropriate.

[dpb]

Was going to comment that was good spelunking @Paul to find the thread and refer to fftw; something like that was what I had in mind that would get reset on restart but wasn't affected by normal memory clearing, etc., ...back in days of yore before MATLAB and had to use the libraries directly in FORTRAN (before Fortran days, too) I knew about fftw but it had completely slipped my mind in the ensing 40 years. Of course, that also predated having multi-cores, GPUs, parallel computing TBs so one had far more direct knowledge of what was going on inside.

[Timothy]

@Paul, @dpb, @Walter Roberson I'm sorry I wasn't specific enough in my original post, I have definitely tried reseting the wisdom in fftw for both single and double precision and it did not affect anything.