Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Upgrade Instructions


Instructions for upgrading your Contrail Cloud to the specified release.

Upgrade from Contrail Cloud Release 13.1 to 13.2

This is an in place upgrade as defined by Red Hat. You will have to upgrade role-by-role and host-by-host to complete this upgrade. You must follow a reboot process following the upgrade.

There are no changes in the configuration YAML files between Contrail Cloud 13.1 and 13.2. Therefore, You don't need configuration changes between Contrail Cloud 13.1 to 13.2. If configuration changes must be made for any reason, they must be applied to your existing Contrail Cloud 13.1 deployment before upgrading to Version 13.2. As a best practice, it is always good to review your configuration files to make sure they adhere to a proper schema and the needs of your deployment environment.

Before You Upgrade

Take these initial steps before starting your Contrail Cloud Upgrade. This will help eliminate possible errors that might occur during the upgrade process and will help ensure expected results. The sections below are a prerequisite to the upgrade of your Contrail Cloud.

Review Your Configuration Files

At this point you want to review your current setup to ensure all configuration settings are accurate and reflect a desired deployment for your Contrail Cloud environment.

  • Review all the YAML files in the /var/lib/contrail_cloud/config directory and ensure all values match your expected results.

Verify Undercloud/Overcloud Health and Service Operations

It is vital that you always check the health of your cloud and the services running in your cloud before attempting any deployment or upgrade activities. You must ensure that the undercloud/overcloud is fully functional, healthy, and that all services are active. Any problems in your cloud health may cause errors during upgrading. Incorrect settings and configurations will carry over to the Contrail Cloud 13.2 deployment.

  1. Check the health of the undercloud, overcloud and the nodes running on them. To verify the health of your cloud and the services, see Node Reboot and Health Check and refer to the “Verify Quorum and Node Health section” in the document.

Back Up Your Undercloud and Overcloud

Make sure to back up your undercloud and overcloud before running the update script. For complete instructions to back up your cloud, see BACK UP AND RESTORE THE DIRECTOR UNDERCLOUD, Backing up the overcloud control plane services, and Backing up Contrail Databases in JSON Format.

Pause and Shutdown Business Services

You must pause or shutdown external business services at this time to ensure a smooth upgrade while preventing possible data loss or workload errors. These business services can include the scope of anything outside of the Contrail Cloud deployment but interacts with Contrail Cloud as a whole. The steps to complete the tasks below are dependent on the specific business service/VM that is running. Please consult the documentation for the specific service you need to pause/shutdown.

  • Quiesce all external API requests, for example, Horizon.

  • Gracefully shutdown any vulnerable workloads.

  • You will want to consider migrating your services/VMs to a different cloud that is outside of the upgrade environment.

Start the Upgrade from Contrail Cloud Release 13.1 to 13.2

Time to upgrade to Contrail Cloud Release 13.2. Contrail Cloud 13.2 will deliver updated containers, RHEL image and kernel version that are associated with Release 13.2.

The procedure below will guide you through the update. There is a small disruption in service during the update. However, the update preserves existing overcloud configurations. For example: images, projects, networks, volumes, virtual machines, and so on.

Retrieve Adjusted Keys and Install

Follow these steps to start your upgrade:

  1. Send an e-mail message to to request Contrail Cloud 13.2. Provide the following information:
    • Include your current activation key in the email request. Your Contrail Cloud activation key will be adjusted to Version 13.2.

    • Specify the time and date you would like to upgrade your Contrail Cloud version. The Contrail Cloud team will prepare the activation for your maintenance window.

  2. Refresh your Contrail Cloud subscription on the jump host server by running the from the jump host with the arguments:
  3. Ensure that all overcloud nodes have valid subscription-manager registrations.

Upgrade to Contrail Cloud 13.2

The following procedure and scripts will upgrade your Contrail Cloud to Version 13.2.

As the “contrail” user (su - contrail from root), execute the following scripts on the jump host to perform the update:

  1. Upgrade the jump host and the undercloud VM.
  2. Prepare the overcloud for upgrade:
  3. Perform the overcloud upgrade.

    There are two different methods that can be used to complete this step. The different methods are listed below (choose one):

    • Default method. All nodes will upgrade in one run.

      • All roles are upgraded one by one.

      • Within each role the nodes are upgraded one by one.

    • Targeted method. You have the ability to target roles and even nodes to control the upgrade sequence.

      • Ability to set the desired upgrade targets in the configs/site.yml file.

      • Typical to upgrade all control plane roles together with this method.

      • Computes can be upgraded in small targeted groups.

    If you encounter failures while running the script, see If an Upgrade Fails in the sections below.


    The overcloud upgrade script has a hard timeout of 4 hours, which may not be sufficient for complex deployments. Consider using targeted updates to allow for incremental role upgrades which can complete within that timeframe.

    Default Method

    To upgrade all the nodes using the default method, run the script below. This will upgrade all nodes in one run and require no additional steps. The update will apply to all roles one at a time and one node at a time within each role.

    Targeted Method

    The procedure below allows you to target specific roles and nodes during the upgrade. This approach allows for control and predictability of the upgrade and subsequent compute node reboots. This method is desirable if you want to update the control plane roles at one time, and then target specific compute resources to be updated as workloads are migrated.

    To complete a targeted update, just edit your /config/site.yml for each targeted group and rerun the update script each time a change is made. This process can be rerun multiple times if necessary. You can use the name of a specific node, or the name of a specific role to upgrade. Just remember to change your /config/site.yml with each update. Per-node control allows for planning around node reboot. It may be desirable to reboot compute nodes as they are updated to avoid disruption later. For how to reboot your compute nodes, see Node Reboot and Health Check and refer to the node reboot section.


    Compute nodes may automatically reboot as part of the upgrade process.

    • You need to configure your /config/site.yml to reflect the nodes you want upgraded. The upgrade will be performed in the order you configure in the site.yml file. To start, you might configure it to look like this:
    • Run the update script after you have set your variables:
    • You can now edit your /config/site.yml to target the specific nodes you want to update (e.g. computes). Replace the role names with the node names you want to update. Below is an example targeting specific compute nodes to be upgraded:

      Run the update script after all variables have been set:

    You need to create a flag file to mark as compete once all overcloud nodes have been upgraded. The flag file is required before running the next upgrade script. Run the following command:

  4. Converge the overcloud upgrade. The script below will update Ceph and converges the overcloud heat stack. Note, the overcloud[‘deployment_timeout’] value in the config/site.yml can be increased to avoid timeouts in the Ceph upgrade.

Move on to the next sections to upgrade your AppFormix and Contrail Command for Contrail Cloud 13.2.

Upgrade AppFormix

Upgrade to the latest version of AppFormix for use with Contrail Cloud 13.2.

  1. As the “contrail” user (su - contrail from root), execute the following script on the jump host to perform the update:
  2. Verify the status of AppFormix.

    Run the following command to view the status AppFormix:

    This will return a 200 on success. Any other code returned should be considered a failure. The API output also contains the AppFormix version. This is helpful to verify the correct version has been installed. See the sample below:

Upgrade Contrail Command

Upgrade to the latest version of Contrail Command for use with Contrail Cloud 13.2.

  1. As the “contrail” user (su - contrail from root), execute the following script on the jump host to perform the update:
  2. Login to the Contrail Command web UI to verify that it was successfully installed. You access Contrail Command by entering https://<jumphost>:9091 in your browser.

    Review the /var/lib/contrail_cloud/config/vault-data.yml for Contrail Command authentication details.

If an Upgrade Fails

If at any point your upgrade fails you will need to troubleshoot. Follow these basic steps for failure analysis:

  • Review the failure output and take screenshots. The screenshots will help others review your failure.

  • Review your configuration files. There could be mistakes in your YAML configuration files. Some common configuration errors include (but not limited to): NIC setup, role assignment, and networking related errors.

  • Gather information to help troubleshoot the problem. One common troubleshooting step is to retrieve the log from a failed node. You do this by ssh to the node and check /var/log/messages. Use the following sequence of CLI commands:

    1. Log in to the jump host as the root user.
    2. su - contrail
    3. ssh undercloud
    4. source stackrc
    5. Run openstack stack failures list overcloud to identify any stack failures to help identify which roles are having issues.

      Nothing will return in the CLI if there are no failures to report.

    6. nova list
    7. ssh <address>. Use the list generated in step 6 to identify the node you need to ssh to.
    8. sudo vi /var/log/messages from within the selected node.
  • You must bring all services back to health for the failure to be considered corrected.

    Restore the Pacemaker cluster that was stopped as a result of the failed step in the upgrade procedure (pcs cluster start on the controller nodes that have it stopped) to bring the cluster back to healthy state.

    Re-run the failed script only when the failure has been corrected and Pacemaker has been started with the cluster healthy again. Move forward with the upgrade procedure only after the failed playbook runs successfully.

You can safely move on to reboot your nodes if you received no failures during the upgrade process.

Remove Duplicate vRouters

It is possible that duplicate instances of the vRouter might occur during the upgrade process, and it is necessary to remove these duplicates. Access the GUI at this point to identify and remove any duplicate vRouters before continuing with the upgrade process.

Reboot Your Nodes

Contrail Cloud 13.2 introduces a new RHEL image and kernel. You need to reboot the nodes as described in, Node Reboot and Health Check.

Upgrade from Contrail Cloud Release 13.02 to 13.1

Contrail Cloud 13.1 does not support upgrade from earlier releases. You must redeploy using adjusted activation keys and retrieve new software packages from the Contrail Cloud Satellite.

  1. Send a request to regarding the adjustment of your Contrail Cloud keys to Version 13.1.
  2. Redeploy Contrail Cloud using the adjusted activation keys.

    For more information, see Deploying Contrail Cloud.

Upgrade from Contrail Cloud Release 13.0.1 to 13.0.2

Upgrade to Contrail Cloud Release 13.0.2 to apply the updated containers that are delivered with Contrail Networking 5.0.2. This update restarts each instance of overcloud roles, one-by-one, so there is a small disruption in service during the update. However, the update preserves existing overcloud configurations. For example: images, projects, networks, volumes, virtual machines, and so on.

To update Contrail Cloud to 13.0.2:

  1. Ensure that the overcloud is fully functional and that all services are active.
  2. Review the config/site.yml.
    1. Remove any overcloud.registry configuration

    2. Validate that the control host storage allocations use defined storage pools. If the defaults were not used then it might be necessary to adjust the control-host configuration.

  3. Review the config/overcloud-nics.yml, config/control-host-nodes.yml, and config/appformix-nodes.yml to rename all instances of ControlInterfaceDefaultRoute to ControlPlaneDefaultRoute.
  4. Send an e-mail message to to coordinate the deployment activation key from Contrail Cloud 13.0.1 to Contrail Cloud 13.0.2. An update script is then provided.
  5. Download the script to /var/lib/contrail_cloud/scripts/ on the jumphost. Make this file executable:
  6. As the “Contrail” user, execute the following script on the jumphost to perform the update:/var/lib/contrail_cloud/scripts/

Workaround for DPDK Compute Nodes

The update script does not update the contrail-vrouter-agent-dpdk container on the DPDK compute nodes.

Use the instructions below to update the Contrail Cloud 13.0.2 DPDK compute nodes:

  1. For each DPDK compute node, update /etc/sysconfig/network-scripts/network-functions-vrouter-dpdk-env to the following:
  2. Restart the vhost0 interface for the changes to take effect.

Workaround for Kernel vRouter Compute Nodes

The update script does not update the contrail-vrouter-kernel-init container on the kernel compute nodes.

Use the instructions below to update the Contrail Cloud 13.0.2 kernel vRouter compute nodes:

  1. For each kernel vRouter compute node, pull the latest Docker image:
  2. Find the docker image ID:
  3. Run the init container:
  4. Restart the vRouter agent and vhost0 interface:
  5. Reboot to apply the updates: