Resolve MATLAB Pod Issues

The MATLAB^® pod comprises four Docker^® containers, one of which is MATLAB Runtime.

When the services are running successfully and have no issues, then the status of pods is Running.

However, if the services are having issues, pods can still show the status Running. This can be because, for example, the MATLAB Docker image was built incorrectly, or there are issues connecting with the network license manager server.

Run the following command to display pod status, replacing <namespace-name> with the namespace you used for MATLAB Online Server™:

kubectl get pods --namespace <namespace-name>

This command outputs information similar to that shown in the following example:

 NAME                                               READY    STATUS    RESTARTS      AGE 
<namespace-name>-core-ui-core-ui-5bd9b7c88b-fqbl4     1/1     Running      0          111s 
<namespace-name>-authnz-authnz-7c7999589c-x74d9       1/1     Running      0          111s 
<namespace-name>-gateway-669dd7865f-t6l74             1/1     Running      0          110s 
<namespace-name>-license-548f7b6655-8skdt             1/1     Running      0          108s 
<namespace-name>-matlab-pool-8c679f76-c4h4n           2/2     Running      0          107s 
<namespace-name>-matlab-pool-8c679f76-cckkp           2/2     Running      0          107s 
<namespace-name>-matlab-pool-ui-579fc8ff49-2bz25      1/1     Running      0          107s 
<namespace-name>-ingress-nginx-controller-kcwwd       1/1     Running      0          104s
<namespace-name>-resource-resource-894447df7-r9ngb    1/1     Running      0          103s

Problems and Resolutions

Check the possible problems and solutions in this section to help diagnose and resolve common issues.

License Checkout Issues

Although a license checkout error is a failing scenario, the MATLAB pod status still shows as Running because a prewarmed pool of MATLAB instances might run successfully but fail to check out a license when a user signs in.

To determine if a license checkout error is the issue, perform the following assessment:

Run the following command to get the MATLAB container logs, replacing <matlab-pod-name> with the name of your MATLAB Online Server MATLAB pod and <namespace-name> with the namespace you used for MATLAB Online Server:
```
kubectl logs <matlab-pod-name> --namespace <namespace-name> -c matlab
```
Check for License Manager Error -<n>: License Checkout Failed in the logs. If this error has occurred, then it is highly likely that MATLAB is having issues checking out the license.
Logs can be lengthy. You can use a text-searching tool to get the information you need faster. For example, you can search the logs for a license checkout error:
```
kubectl logs <matlab-pod-name> --namespace <namespace-name> -c matlab | grep 'License Checkout'
```
You can also search for a license manager error:
```
kubectl logs <matlab-pod-name> --namespace <namespace-name> -c matlab | grep 'License Manager Error'
```
If you see License Manager Error -<n>: License Checkout Failed in the logs, check the specific error message, and then consult the following table.

Error Message Resolution Steps

Invalid license file syntax Check to make sure that the license file syntax is appropriate (that is, check the network license manager logs). Then, making sure that you use the correct license file, try again.

Cannot connect to license server

Try to connect to the license server locally.
Check to see if the network firewall rules for the license server are set correctly to allow connection from the MATLAB Online Server cluster.
Network policy on a MATLAB pod is enabled by default, which allows it to talk to the network license manager only on ports 27000 and 27001 by default (controlled by license.dat). Make sure your network license manager server is listening on those ports. If your server is listening on other ports, you need to change the following configuration in the matlab-pool override and then restart the pod:
networkPolicy: enabled: true additionalAllowedPorts: - port: <flexlm_port_here> protocol:

Incorrect MATLAB Image

The MATLAB Image is built from a local MATLAB install and can sometime have issues if it was built from an incorrect MATLAB location or version that is not supported by MATLAB Online Server.

To determine if an incorrect MATLAB image is the problem, perform the following assessment.

Run the following command to display a pod status list, replacing <namespace-name> with the namespace you used for MATLAB Online Server:
```
kubectl get pods --namespace <namespace-name>
```

Examine the pod status information, as shown in this example:

NAME                                               READY   STATUS    RESTARTS   AGE 
<namespace-name>-authnz-authnz-569df55f78-pqklk      1/1     Running   0          47h 
<namespace-name>-core-ui-core-ui-84c59ddd45-fwf4t    1/1     Running   0          47h 
<namespace-name>-gateway-84dbf7c8d-6gjcz             1/1     Running   0          47h 
<namespace-name>-license-85ffb6c595-q6x4f            1/1     Running   0          47h 
<namespace-name>-matlab-pool-apparmor-loader-jzqsg   1/1     Running   0          40s 
<namespace-name>-matlab-pool-f479b5b96-59xmj         3/4     Running   0          40s 
<namespace-name>-matlab-pool-helpsearch-546cb5fd8-jkx4p  1/1     Running   0          40s 
<namespace-name>-matlab-pool-ui-7b6d688f98-rfdz5	1/1     Running   0          40s 
<namespace-name>-ingress-nginx-controller-5c6b5f775f-7ls87  1/1     Running   0          27h
<namespace-name>-resource-resource-8554f87bbf-g4ssq   1/1     Running   0          47h

If you see MATLAB pods in a 3/4 Running state, as shown in the example log, run the following command to learn why. Replace <matlab-pod-name> with your MATLAB Online Server MATLAB pod name and <namespace-name> with the namespace you used for MATLAB Online Server:

kubectl logs <matlab-pod-name> --namespace <namespace-name> -c matlab

This command outputs a log similar to this example:

Starting MATLAB 
+ exec /MATLAB/bin/matlab -prewarm -c '"27000@flexlm.mwcloudtest.com"' -nodesktop -softwareopengl -nosplash 
MATLAB is selecting SOFTWARE OPENGL rendering. 
...
 
< M A T L A B (R) > 
Copyright 1984-2024 The MathWorks, Inc. 
...
 
Warning: Unrecognized command line option: prewarm.  
  
To get started, type doc. 
For product information, visit www.mathworks.com.

The logs show that MATLAB was not able to start successfully. To verify that this is what happened, you can run the following command to look at the logs of resource-proxy, which is another container in the MATLAB pod that proxies requests to MATLAB.

Issue this command, replacing <matlab-pod-name> with your MATLAB Online Server MATLAB pod name and <namespace-name> with the namespace you used for MATLAB Online Server:

kubectl logs <matlab-pod-name> --namespace <namespace-name> -c resource-proxy

Examine the logs output by this command. If you see the following messages, the backend (in this case, MATLAB) is not ready and is not functioning properly.

YYYY-MM-DD 19:06:51,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:06:51,507 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:06:54,498 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:04,499 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:11,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:11,507 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:14,500 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:24,500 WARNING matlabonlineserver.http.client Jetty request failed: Connection refused 
YYYY-MM-DD 19:07:31,037 INFO matlabonlineserver.resource_proxy Backend is not ready 
YYYY-MM-DD 19:07:31,507 INFO matlabonlineserver.resource_proxy Backend is not ready

In the example log, the message "Backend is not ready" indicates that MATLAB is not yet ready.

For resolution, check to see if you are building the MATLAB image from the correct MATLAB install location.

Storage Mount Errors

When all services and MATLAB are in a Running state, but you are encountering issues using MATLAB Online™, it is possible that there was a problem with mounting the storage on MATLAB at the time of user sign on.

To determine if storage mounting on MATLAB is the problem, perform the following assessment:

Run the following command to generate pod logs, replacing <matlab-pod-name> with the name of your MATLAB Online Server MATLAB pod and <namespace-name> with the namespace you used for MATLAB Online Server.
```
kubectl logs <matlab-pod-name> --namespace <namespace-name> -c resource-proxy
```

Examine the output from this command:

YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Setting up the profile: nfsHome 
YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Setting up the mount: home 
YYYY-MM-DD 20:44:39,124 INFO matlabonlineserver.storage Starting NFS mount 
YYYY-MM-DD 20:44:39,125 INFO matlabonlineserver.storage Executing: mount -o rw,
relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,
acdirmin=600,acdirmax=600,hard,nocto,noacl,proto=tcp,timeo=600,retrans=2,mountproto=tcp,
local_lock=none,nolock nfs:/exports/home/oin /mounts/homeoin 
YYYY-MM-DD 20:44:49,167 INFO matlabonlineserver.storage mount exit value= 137, mount result= , mount error=

If you see the mount error message, as displayed in the last line of the example shown, check that the ports allowed by the network policy for MATLAB Pool include ports for your NFS server setup. For more details on network policies, see Configure Network Policies.

MATLAB Pod Not in Running State

If none of the possible issues described in this section help you to resolve the problem, try the following:

See Resolve MATLAB Pod Issues.
Contact MathWorks Support.
1. Run the following kubectl commands, replacing <license-pod-name> with the name of the pod that is stuck and <namespace-name> with the namespace you used for MATLAB Online Server:
```
kubectl describe pod <license-pod-name> --namespace <namespace-name>
kubectl logs license-pod-name --namespace <namespace-name>
```
2. Keep the output nearby to discuss with a support technician.