Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?


Contrail In-Service Software Upgrade from Releases 3.2 and 4.1 to 5.0.x using Ansible Deployer

Contrail In-Service Software Upgrade (ISSU) Overview

If your installed version is Contrail Release 3.2 or higher, you can perform an in-service software upgrade (ISSU) to upgrade to Contrail Release 5.0.x using the Ansible deployer. In performing the ISSU, the Contrail controller cluster is upgraded side-by-side with a parallel setup, and the compute nodes are upgraded in place.


We recommend that you take snapshots of your current system before you proceed with the upgrade process.

The procedure for performing the ISSU using the Contrail Ansible deployer is similar to previous ISSU upgrade procedures.


This Contrail ansible deployer ISSU procedure does not include steps for upgrading OpenStack. If an OpenStack version upgrade is required, it should be performed using applicable OpenStack procedures.

In summary, the ISSU process consists of the following parts, in sequence:

  1. Deploy the new cluster.

  2. Synchronize the new and old clusters.

  3. Upgrade the compute nodes.

  4. Finalize the synchronization and complete the upgrades.


The following prerequisites are required to use the Contrail ansible deployer ISSU procedure:

  • A previous version of Contrail installed, no earlier than Release 3.2.

  • There are OpenStack controller and compute nodes, and Contrail nodes.

  • OpenStack needs to have been installed from packages.

  • Contrail and OpenStack should be installed on different nodes.


Upgrade for compute nodes with Ubuntu 14.04 is not supported. Compute nodes need to be upgraded to Ubuntu 16.04 first.

Preparing the Contrail System for the Ansible Deployer ISSU Procedure

In summary, these are the general steps for the system preparation phase of the Contrail ansible deployer ISSU procedure:

  1. Deploy the 5.0.x version of Contrail using the Contrail ansible deployer, but make sure to include only the following Contrail controller services:

    • Config

    • Control

    • Analytics

    • Databases

    • Any additional support services like rmq, kafka, and zookeeper. (The vrouter service will be deployed later on the old compute nodes.)


    You must provide keystone authorization information for setup.

  2. After deployment is finished, you can log into the Contrail web interface to verify that it works.

The detailed steps for deploying the new cloud using the ansible deployer are as follows:

  1. To deploy the new cloud, download contrail-ansible-deployer-release-tag.tgz onto your provisioning host from Juniper Networks.

  2. The new cloud file config/instances.yaml appears as follows, with actual values in place of the variables as shown in the example:

  3. Finally, run the ansible playbooks to deploy the new cloud.

After successful completion of these commands, the new cloud should be up and alive.

Provisioning Control Nodes and Performing Synchronization Steps

In summary, these are the general steps for the node provisioning and synchronization phase of the Contrail ansible deployer ISSU procedure:

  1. Provision new control nodes in the old cluster and old control nodes in the new cluster.

  2. Stop the following containers in the new cluster on all nodes:

    • contrail-device-manager

    • contrail-schema-transformer

    • contrail-svcmonitor

  3. Switch the new cloud into maintenance mode to prevent provisioning computes in the new cluster.

  4. Prepare the config file for the ISSU.

  5. Run the pre-sync script from the ISSU package.

  6. Run the run-sync script from the ISSU package in background mode.

The detailed steps to provision the control nodes and perform the synchronization are as follows:

  1. Pair the old control nodes in the new cluster. It is recommended to run it from any config-api container.

  2. Run the following command for each old control node, substituting actual values where indicated:

  3. Pair the new control nodes in the old cluster with similar commands (the specific syntax depends on the deployment method of the old cluster), again substituting actual values where indicated.

  4. Stop all the containers for contrail-device-manager, contrail-schema-transformer, and contrail-svcmonitor in the new cluster on all controller nodes.

These next steps should be performed from any new controller. Then the configuration prepared for ISSU runs. (For now, only manual preparation is available.)


In various deployments, old cassandra may use port 9160 or 9161. You can learn the configuration details for the old services on any old controller node, in the file /etc/contrail-contrail-api.conf.

The configuration appears as follows and can be stored locally:

  1. Detect the config-api image ID.

  2. Run the pre-synchronization.

  3. Run the run-synchronization.

  4. Check the logs of the run-sync process. To do this, open the run-sync container.

  5. Stop and remove the run-sync process after all compute nodes are upgraded.

Transferring the Compute Nodes into the New Cluster

In summary, these are the general steps for the node transfer phase of the Contrail ansible deployer ISSU procedure:

  1. Select the compute node(s) for transferring into the new cluster.

  2. Move all workloads from the node(s) to other compute nodes. You also have the option to terminate workloads as appropriate.

  3. For Contrail Release 3.x, remove Contrail from the node(s) as follows:

    • Stop the vrouter-agent service.

    • Remove the vhost0 interface.

    • Switch the physical interface down, then up.

    • Remove the vrouter.ko module from the kernel.

  4. For Contrail Release 4.x, remove Contrail from the node(s) as follows:

    • Stop the agent container.

    • Restore the physical interface.

  5. Add the required node(s) to instances.yml with the roles vrouter and openstack_legacy_compute.

  6. Run the Contrail ansible deployer to deploy the new vrouter and to configure the old compute service.

  7. All new compute nodes will have:

    • The collector setting pointed to the new Contrail cluster

    • The Control/DNS nodes pointed to the new Contrail cluster

    • The config-api setting in vnc_api_lib.ini pointed to the new Contrail cluster

  8. (Optional) Run a test workload on transferred nodes to ensure the new vrouter-agent works correctly.

Follow these steps to rollback a compute node, if needed:

  1. Move the workload from the compute node.

  2. Stop the Contrail Release 5.0.x containers.

  3. Ensure the network configuration has been successfully reverted.

  4. Deploy the previous version of Contrail using the deployment method for that version.

The detailed steps for transferring compute nodes into the new cluster are as follows:


After moving workload from the chosen compute nodes, you should remove the previous version of contrail-agent. For example, for Ubuntu 16.04 and vrouter-agent installed directly on the host, these would be the steps to remove the previous contrail-agent:

  1. The new instance should be added to instances.yaml with two roles: vrouter and openstack_compute_legacy. To avoid reprovisioning the compute node, set the maintenance mode to TRUE. For example:

  2. Run the ansible playbooks.

  3. The contrail-status for the compute node appears as follows:

  4. Restart contrail-control on all new controller nodes after the upgrade is complete:

  5. Check status of new compute nodes by running contrail-status on them. All components should be active now. You can also check the status of the new instance by creating AZ/aggregates with the new compute nodes and run some test workloads to ensure it operates correctly.

Finalizing the Contrail Ansible Deployer ISSU Process

Finalize the Contrail ansible deployer ISSU as follows:

  1. Stop the issu-run-sync container.

  2. Run the post synchronization commands.

  3. Disengage maintenance mode and start all previously stopped containers. To do this, set the entry MAINTENANCE_MODE in instances.yaml to FALSE, then run the following command from the deployment node:

  4. Clean up and remove the old Contrail controllers. Use the script called from the config-api container with the config issu.conf. Replace the credential variables and API server IP with appropriate values as indicated.

  5. Run the following commands from any controller node.


    All *host_info parameters should contain the list of new hosts.

  6. Servers can be cleaned up if there are no other services present.

  7. All configurations for the neutron-api must be edited to have the parameter api_server_ip point to the list of new config-api IP addresses. Locate ContrailPlugin.ini (or other file that contains this parameter) and change the IP addresses to the list of new config-api IP addresses.

  8. The heat configuration needs the same changes. Locate the parameter [clients_contrail]/api_server and change it to point to the list of the new config-api IP addresses.