Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

IN THIS PAGE

Troubleshooting Upgrade-Related Errors

 

This topic describes the possible errors that you might encounter while you are upgrading Contrail Service Orchestrator (CSO).

It also suggests how to resolve those errors.

Salt Synchronization Error

Problem:

Description: The upgrade or revert status is displayed as Going to sync salt... for a considerable time while upgrading CSO to CSO Release 4.0.0 or reverting to the previously installed release.

The Salt Master on the installer VM might be unable to reach all Salt Minions on the other VMs and a salt timeout exception might occur.

Solution

Based on the output of the salt ‘*’ test.ping command, you must restart either the Salt Master or the Salt Minion.

To resolve the error:

  1. Open another instance of installer VM.
  2. Run the salt ‘*’ test.ping command, to check whether the Salt Master on the installer VM is able to reach other VMs.
    root@host:~/# salt ‘*’ test.ping

    • Restart the Salt Master if the following error occurs:

      Salt request timed out. The master is not responding. If this error persists after verifying the master is up,worker_threads may need to be increased

      root@host:~/# service salt-master restart

    • If there are no errors, review the output.

      root@host:~/# salt ‘*’ test.ping

      You must log in to the VM and restart the Salt Minion whether the status of the VM is False.

      root@host:~/csp-central-infravm.8V1O2D.central# service salt-minion restart
  3. Rerun the salt ‘*’ test.ping command to verify whether the status for all VMs is True.

Cache Clearance Error

Problem:

Description: The following error might occur while upgrading CSO to CSO Release 4.0.0:

Could not free cache on host server ServerName

Solution

You must clear the cache on the host server.

To resolve the error:

  1. Log in to the host server through SSH.
  2. To clear the cache, run the following command:
    root@host:~/Contrail_Service_Orchestration_4.0.0# free && sync && echo 3 > /proc/sys/vm/drop_caches && free

    The following output is displayed:

The cache is cleared on the host server.

Kube-system Pod Error

Problem:

Description: The following error might occur while upgrading CSO to CSO Release 4.0.0:

One or more kube-system pods are not running

Solution

Check the status of the kube-system pod, and restart kube-proxy if required.

To resolve the error:

  1. Log in to the central or regional microservices VM through SSH.
  2. Run the following command to view the status of the kube-system pod:
    root@host:~/# kubectl get pods –namespace=kube-system

    The following output is displayed:

    Check the status of kube-proxy. You must restart kube-proxy if the status is Error, Crashloopback, or MatchNodeSelector.

  3. Run the following command to restart kube-proxy
    root@host:~/# Kubectl apply –f /etc/kubernetes/manifests/kube-proxy.yaml

The kube-system pod-related error is resolved.

Kubernetes Node Error

Problem:

Description: The following error might occur while upgrading CSO to CSO Release 4.0.0:

One or more nodes down

Solution

Check the status of kube-master or kube-minion and restart the nodes, if required.

To resolve the issue:

  1. Log in to the central or regional microservices VM through SSH.
  2. Run the following command to check the status of each node:
    root@host:~/# kubectl get nodes

    Identify the node that is in the Not Ready status. You must restart the node if the status is Not Ready.

  3. Restart the node if the status is Not Ready by logging in to the node through SSH and running the following command:
    root@host:~/# service kubelet restart
  4. Rerun the following command to check the status of the node that you restarted.
    root@host:~/# kubectl get nodes

The Kubernetes node-related error is resolved.

Related Documentation