Error using pdist2mex Error in kmeans>distfun

10 views (last 30 days)
Hi, To represent our data (3233477*256) with Bag of visual word (BOW) which use KMeans clustering to extract visual words when we choose K=5000 this problem show:
Error using pdist2mex
Requested 3233477x5000 (120.5GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a
long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in kmeans>distfun (line 747)
D = pdist2mex(X,C,'sqe',[],[],[]);
Error in kmeans/loopBody (line 445)
D = distfun(X, C, distance, 0, rep, reps);
Error in internal.stats.parallel.smartForReduce (line 136)
reduce = loopbody(iter, S);
Error in kmeans (line 335)
ClusterBest = internal.stats.parallel.smartForReduce(...
Error in BOWHistogram (line 12)
[idx,c,sumd,D2] = kmeans(double(Tab_Feature_Data),NumClust);
What can I do to fix the error? Please advise me
  2 Comments
Tripoli Settou
Tripoli Settou on 18 Mar 2018
How can i do that i must extract the visual word from all data...how can i split it and get the visual word of all data?

Sign in to comment.

Accepted Answer

Bernhard Suhm
Bernhard Suhm on 25 Mar 2018
You could try converting your large input data into a tall array (maybe as simple as t = tall(double(Tab_Feature_Data)), and then pass that tall array to kmeans. Though watch there are limitations which options of kmeans are available with tall arrays, see https://www.mathworks.com/help/stats/tall-array-support-usage-notes-and-limitations.html
  4 Comments
Tripoli Settou
Tripoli Settou on 26 Mar 2018
So u advise me to use the Parallel Computing Toolbox to leverage multiple cores? right? also, I need the 4th output variable (D2) in next work
[idx,c,sumd,D2] = kmeans(double(Tab_Features),NumClust);
and can I use the Parallel Computing Toolbox without tall version KMeans i mean KMeans with Parallel Computing Toolbox?? it's work??

Sign in to comment.

More Answers (1)

carpcarp carpcarp
carpcarp carpcarp on 2 Apr 2021
我也遇到过这个问题,你可以看看你是不是命名了一个kmeans的m文件,如果有,并且是自己添加的,先把它删除(或者先发到qq等可以保存代码的应用后删除),然后你再去跑你的程序就可以成功了。
  2 Comments
Walter Roberson
Walter Roberson on 2 Apr 2021
(User points out that there can be problems if you accidentally have your own kmeans.m instead of using MATLAB's)

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by