Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?


Upgrade Procedure for RHOSP-based Contrail 3.2.x to Contrail 4.1

This section presents the steps to upgrade an OSP-based Contrail deployment from Contrail version 3.2.x to Contrail version 4.1.


Ensure that you have a cloud up and running with RHOSP10 Z4 and Contrail 3.2 before you proceed with the upgrade procedure.

This procedure has been validated with the following Contrail, Red Hat, and OpenStack versions.

Contrail Version

Red Hat Version

OpenStack Version


RHEL 7.3

RHOSP10 (packages dated Apr. 15, 2017)


RHEL 7.4

RHOSP10 (packages dated Feb. 2, 2018)


RHEL 7.4

RHOSP10 (packages dated Feb. 27, 2018)


RHEL 7.5

RHOSP10 (packages dated Jun. 4, 2018)

RHOSP11 (packages dated Jun. 4, 2018)


RHEL 7.5

RHOSP10 (packages dated Oct 29, 2018)


For Contrail Release 4.1.1, you must ensure that the OpenJDK version is java-1.8.0-openjdk- This is because of a compatibility issue that the Cassandra 3.0 package has with the latest version of Open JDK provided in RHEL 7.5.

You must add the correct version of OpenJDK to the Contrail repository and remove the older version of the package from the contrail controllers and analytics database.

To upgrade to OpenStack Platform release 10, you must first apply the patch for updating puppet-tripleo. This is because of a bug in the OpenStack Platform release OSP10 z8. For more information, see This bug has been resolved in build puppet-tripleo-5.6.8-7.el7ost; see For more information about this bug, see

Upgrade the Undercloud

Upgrade the undercloud to the most current RHOSP10 version.

  1. Log in to the undercloud as the stack user.

    su – stack

  2. Update the Contrail repositories.

    sudo rm –rf /etc/yum.repos.d/*contrail*

    curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.repo

  3. Stop the main OpenStack platform services.

    sudo systemctl stop 'openstack-*' 'neutron-*' httpd

  4. Update the python-tripleoclient package and its dependencies to ensure you have the most current scripts for the minor version update.

    $ sudo yum update python-tripleoclient

  5. Upgrade the undercloud.

    $ openstack undercloud upgrade

  6. Reboot the node.

    $ sudo reboot

  7. Wait until the node reboots, then check the status of all services.


    It can take as much as 10 minutes or more for the openstack-nova-compute to become active after a reboot.

    $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

  8. Verify the version of RHEL after the undercloud upgrade.

  9. Check the versions of the RHOSP images to verify that the new images are available.

    In the following example, the new images are rhosp-director-images-ipa-10.0-20180103.3.el7ost.noarch and rhosp-director-images-10.0-20180103.3.el7ost.noarch.

  10. Verify the existence of the overcloud and its nodes.

    $ openstack stack list

    $ ironic node-list

  11. Review the power status, provision state, and maintenance, ensuring:

    • Power state is set to Power on

    • Provision state is set to Active

    • Maintenance is set to False

  12. Verify that all OpenStack servers are Active.

    $ openstack server list

    Figure 1: Server ListServer List

Update Red Hat Director Image Archives

The undercloud update process might download new image archives from the rhosp-director images and the rhosp-director-ipa packages. You’ll want to update your existing system with any new image archives.

  1. Check the yum log to determine if new image archives are available.

    $ sudo grep "rhosp-director-images" /var/log/yum.log

  2. If new image archives are available, replace your current images with the new images. Before deploying any new images, remove any existing images from the images undercloud on the stack user’s home (/home/stack/images).

    $ rm -rf ~/images/*

  3. Extract the new image archives.

  4. Install the Contrail package on the overcloud image by using the virt-customize command.

  5. Import the new image archives into the undercloud and configure nodes to use the new images.

    $ openstack overcloud image upload --update-existing --image-path /home/stack/images/ $ openstack baremetal configure boot

  6. Verify that the images are uploaded.

    $ glance image-list

  7. Show the details of the new image that has been created. The new image will be used to add a new node in the overcloud.

    $ glance image-show overcloud-full

  8. Observe the contrail-status on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.

    [stack@undercloud ~]$ for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done

Prepare Repositories on all Nodes

  1. Delete existing repositories on all overcloud nodes. Be sure to verify each deletion.

    for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode 'find /etc/yum.repos.d/ ! -name 'contrail-install.repo' -type f -exec sudo rm -f {} +' ; done

  2. Add new repositories on all overcloud nodes. Be sure to verify each addition.

    for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode ' curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.rep' ; done

Upgrade the Operating System on Contrail Nodes

  1. Define a list ($iplist) that contains all Contrail nodes.

    Iplist=” @IPcontrailController1 @IPContrailController2 …”

  2. Upgrade the operating system for all nodes in the iplist.

    for ipnode in $iplist; do echo -e "\n\n\t******upgrade node : $ipnode ******" && ssh heat-admin@$ipnode 'sudo yum update -y --disablerepo=*contrail* --skip-broken && exit' ; done

Prepare the Contrail Packages

Check the undercloud Contrail packages versions for contrail-tripleo-puppet, puppet-contrail, and contrail-tripleo-puppet. The newest versions of those packages must be installed before proceeding with the overcloud upgrade. See the following example, with current packages versions.

Upgrade the Contrail Heat Templates

You must copy the new Contrail Heat templates package to the undercloud node, while retaining a copy of the Heat templates that were used for the existing deployment.

  1. Make a copy of all of the Heat templates that were used for deployment and save the copies, because the existing files will be overwritten by the new versions. The templates to copy are of the form contrail-services.yaml, contrail-net.yaml, and so on.

  2. Copy the new contrail-tripleo-heat templates to the undercloud node.

Modify the Yum Update Script for TripleO Puppet

Before starting the upgrade, a few Puppet commands must be added to the yum_update script, located at:


  1. Update the following Puppet commands in the Refer to the following patch for details regarding the exact placement of the commands patch:

    yum install -y contrail-tripleo-puppet puppet-contrail

    rsync -a /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/

  2. Ensure that the new version of the packages puppet-contrail and contrail-tripleo-puppet have been installed on the Contrail nodes, and that contrail-tripleo-puppet has been copied to openstack-puppet.

    rm -rf /usr/share/openstack-puppet/modules/tripleo/contrail-tripleo-puppet

    cp -R /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo

  3. Update the fields *contrail version and *contrail repo in contrail-services.yaml.

    Filename: \templates/environments/contrail/contrail-services.yaml.

    Add the following parameters:

    ContrailVersion: 4

    ContrailRepo : <location of the contrail-41 repo>

Update the Overcloud Deployment Plan

  1. Update the current plan by rerunning the command used for cloud deployment and adding the suffix - -update-plan-only.


  2. Make a copy of the existing deploy script to the The is the script used to update the overcloud plan, and it references the same templates that were used to deploy the stack. All files used for the overcloud update should be identical to the files used for deployment, with the exception of the contrail-services file that was updated with the latest contrail-version and contrail-repo.


  3. Update the deployment plan.


Upgrade Cautions


The steps to perform the overcloud upgrade are service disrupting, and should only be performed within a maintenance window.

Potential Packages Failures

Read the following before proceeding with the overcloud upgrade. The upgrade may fail due to packages conflicts in Contrail nodes for analytics, analytics database, and Contrail controllers. Some observed failures due to packages conflicts are detailed in this section.

OpenStack Undercloud Upgrade Failure

OpenStack undercloud upgrade may fail with the following error message:

For more information about this error, see

Contrail-api Service Not Restarted During Upgrade

Contrail-api remains in the initializing state during upgrade. Modify the puppet file as described in


This issue has been resolved in Contrail Release 4.1.2.

Contrail-api Service Down After Upgrade

You can resolve this issue by removing the admin_token entry from the /etc/contrail/contrail-keystone-auth.conf file. For more information, see


This issue has been resolved in Contrail Release 4.1.2.

Overcloud Update Failure

Upgrade from OpenStack Platform release 10 GA image to the latest version hangs because of a systemd bug. For more information, see


To upgrade from Contrail Release 3.2.x to 4.1.1:

  1. Run the following command before you start the update:

  2. Run the overcloud update.

  3. Run the following command:

  4. Restart the node.

Analytics Database Failure

You may encounter a Cassandra package version conflict issue.

Error message: cassandra22 conflicts with cassandra-3.10-0contrail0.el7.centos.noarch

Solution: Remove the cassandra22 package.

sudo rpm -e --nodeps cassandra22-2.2.8-1.noarch

Analytics Node Failure

Error message: file /usr/lib/python2.7/site-packages/redis/ from install of python-redis-2.10.3-3.el7ost.noarch conflicts with file from package redis-py-0.1-2contrail.el7.noarch

Solution: Remove the redis-py package.

sudo rpm -e python-redis-2.10.3-3.el7ost.noarch

Contrail Controller Failure

Error message: cassandra22 conflicts with cassandra-3.10-0contrail0.el7.centos.noarch

Solution: Remove the cassandra22 package.

sudo rpm -e --nodeps cassandra22-2.2.8-1.noarch


You can delete redis-py from the analytics node, and delete cassandra22 from the analytics database and the Contrail controllers node.

Contrail vRouter Package Conflict

Error message: /etc/init.d/contrail-vrouter-nodemgr conflicts between attempted installs of contrail-openstack-vrouter- and contrail-vrouter-agent-

Solution: Remove the contrail-openstack-vrouter.

yum remove contrail-openstack-vrouter

Analytics Node snmp-lib Version Conflict

Error message: duplicate versions of net-snmp-libs -> net-snmp-libs-5.7.2-28.el7_4.1.x86_64 is a duplicate with 1:net-snmp-libs-5.7.2-28.el7.i686


rpm -e --nodeps net-snmp-libs

Upgrade the Overcloud


The steps in this section are service disrupting, and should only be performed within a maintenance window.

  1. Update the overcloud stack.

  2. Verify the overcloud stack status, the contrail-status, and the contrail-version after the upgrade.

    Overcloud Stack Status

    Contrail Stack Status

    Contrail Version Check

Contrail Service Recovery After Upgrade

Upon completing the upgrade, you might have some Contrail services reporting status down on some nodes. this section provides recovery suggestions for different services.

Contrail Controller Services

In a recent upgrade, the following services were down in the Contrail controller nodes.

  • contrail-control — failed

  • contrail-dns — failed

To determine the root cause of these failures, check the Contrail logs in /var/log/contrail/ .

In this example, it was the IFMap and Discovery services causing the failures, because those services are no longer used in the Contrail 4.x releases.


  1. Remove the IFMap and Discover configuration sections from:

    1. /etc/contrail/contrail-control.conf

    2. /etc/contrail/contrail-dns.conf

  2. Restart the services contrail-controller and contrail-dns.

    1. service contrail-control restart

    2. service contrail-dns restart

  3. Verify that the services contrail-controller and contrail-dns are up and running after the restart.

    1. contrail-status

Contrail Compute Services

On the Contrail compute node, the contrail-vrouter-agent service might be down after the upgrade. The vrouter agent is dependent on the kernel module. The RHEL7.3 has been upgraded to RHEL7.4, and the node needs to b rebooted to ensure the correct kernel module gets loaded.

  1. Reboot the node.

  2. After reboot, restart the vrouter service..

    service supervisor-vrouter restart

  3. Verify that all Contrail services are up and running.