Increase mdce heap space?

Hello, I´ve been experiencing problems with mdce. I have a CentOS machine with 4 processors (4x20 cores) and 320GB of RAM, currently 83 workers running. mdce-service.log keeps saying (several times):

 Group-10:out:PeerRmiDispatcher: Error in dispatch while sending return message: 
 java.lang.OutOfMemoryError: Java heap space

So I think I should increase the java heap space for mdce, but searching for -Xmx I found several parts on the MDCE where this value is mostly hardcoded. What is the right way of doing it?

BTW only 100GB/320GB of RAM is being used starting all of the workers.

Answer 1

[Kyle Allison]

The right way to increase the Java heap size for MDCE is to adjust the JOB_MANAGER_MAXIMUM_MEMORY setting in the mdce_def file, which is located at

• matlabroot/toolbox/distcomp/bin/mdce_def.sh (on Linux or Macintosh operating systems)
• matlabroot\toolbox\distcomp\bin\mdce_def.bat (on Windows operating systems)

By default, it is set to 1024MB:

# JOB_MANAGER_MAXIMUM_MEMORY: The maximum heap size of the job manager java
# process.
JOB_MANAGER_MAXIMUM_MEMORY="1024m"

[Juan Mucarquer]

After increasing the memory to 4096m and restart mdce, still got the same error several times:

 Group-10:out:PeerRmiDispatcher: Error in dispatch while sending return message: 
 java.lang.OutOfMemoryError: Java heap space

should I keep increasing? is normal for mdce to consume too much memory for the administration of each worker or this is because I so many workers (~100 workers)? I assume that are the mdce/mjs processes that are out of memory, not the workers itself.

[Kyle Allison]

In most cases, 1024MB is sufficient for the job manager heap memory and 4096MB is plenty (even for a job manager with 100 workers or more). I would recommend not increasing the heap memory above 4096MB in an attempt to fix the issue and instead contact support with more details about how the cluster is being used to cause the OutOfMemoryError.

Answer 2

[Rakesh]

The java.lang.OutOfMemoryError: Java heap space message indicates to us one or more JVM processes have exhausted the heap memory allocated to them. This may happen even when there is more free space too. Because, based on the configuration set the JVM will react. It does not see the system's total memory. Better monitor which MDCE component is facing this error and check for its current JVM settings. Please make sure if the application uses hardcoded -Xmx value. If that is the case please modify the startup configuration. As you seem to have more unused physical memory, allocating a larger heap to the affected JVM is a reasonable first step. Once you make such a change, monitor heap utilization and garbage collection activity to verify that the additional memory resolves the problem and that the application is not experiencing an underlying memory management issue.

If you wish to understand this better and get proper usage steps on how to set the heap size to your application you can refer to this blog, Sizing Your Heap Correctly: Understanding -Xms and -Xmx.