Upgrade Procedure for RHOSP-based Contrail 4.1.3 to Contrail 4.1.4

Prerequisites

Ensure you have a cloud up and running with RHOSP10 and Contrail 4.1.3 before you proceed with the upgrade procedure.

This procedure has been validated with the following Contrail, Red Hat, and OpenStack versions.

Post-Installation

After the installation, you’ll have a cloud networking running RHOSP10 and Contrail 4.1.4. The Red Hat Enterprise Linux (RHEL) kernel version updates to 7.7 during this procedure.

Table 2 summarizes the post-installation software versions.

Contrail version R4.1.4 supports net-snmp package version 5.7.2-43 to support SNMP. The net-snmp packages come from Red Hat, with the exception of the net-snmp-python-5.7.2-43.el7.x86_64.rpm package which is provided in the Contrail repository.

Table 3 summarizes the net-snmp depend packages and their associated repository locations.

Acquire the Software

To download the software images for this procedure:

1. Go to the Juniper Networks Support site for Contrail.

2. Select OS as Contrail and Version as 4.1.4. Download the images that apply to your environment.

Upgrade the Undercloud

1. Log in to the undercloud as the stack user.

   $ su – stack

2. Update the Contrail repositories.

   • Backup the Contrail 4.1.3 packages to a repository with a different name. In this example, the packages are moved to a repository named contrail-R4-1-3.

   • Create a new repository directory to store the Contrail 4.1.4 packages:

3. Copy the downloaded file—in the provided sample, the file is contrail-install-packages_4.1.4.0-63-newton.tgz—to the Contrail repository created in Step 2.

   Note:

   This step assumes that you've already downloaded the Contrail software. See Acquire the Software.

4. Untar the downloaded tgz file.

5. Create a repository in the new directory:

   If the createrepo command is not available, download the createrepo package from Red Hat (Red Hat subscription required).

6. (Clusters deployed using Swift Puppet files only) If your Contrail 4.1 cluster was deployed using Swift Puppet, perform these steps:

   1. Remove overcloud artifacts from the undercloud:

   2. Delete the deployments-artifacts.yaml file if the file is present.

   3. Clean the repositories and confirm that all repositories are available.

7. Stop the main OpenStack platform services.

   $ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

8. Update the python-tripleoclient package and its dependencies to ensure you have the most current scripts for the minor version update.

   $ sudo yum update python-tripleoclient

9. Upgrade the undercloud.

   $ openstack undercloud upgrade

10. Reboot the node.

    $ sudo reboot

    Wait for the node to reboot. The reboot process can take 10 or more minutes to complete.

11. Ensure the undercloud has the latest Contrail R4.1.4 contrail packages:

12. Ensure the undercloud has the latest RHOSP images:

13. Review the ironic node-list output to confirm the following statuses for each package::

    • Power state is power on.

    • Provision State is active.

    • Maintenance is False.

    Note:

    This output presentation has been modified for readability. The UUID and Instance UUID fields were removed as part of this modification.

14. Verify that all OpenStack servers are in the Active state.

    Note:

    This output presentation has been modified for readability. The ID, Image Name, and Networks fields were removed as part of this modification.

15. If new image archives are available, replace your current images with the new images.

    Before uploading the new images onto the undercloud node, move any existing images from the images directory on the stack user’s home directory (/home/stack/images).

16. Extract the new image archives.

17. Import the new image archives into the undercloud and configure the nodes to use the new images.

18. Verify that the images are uploaded:

19. Observe the contrail-status on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.

20. Ensure that all overcloud node contrail repository pointers are properly pointing to the contrail repository.

    Contrail Analytics Example:

Update Red Hat Director Image Archives

The undercloud update process might download new image archives from the rhosp-director images and the rhosp-director-ipa packages. You will have to update your existing system with any new image archives.

1. Check the yum log to determine if new image archives are available.

   $ sudo grep "rhosp-director-images" /var/log/yum.log

2. If new image archives are available, replace your current images with the new images. Before deploying any new images, remove any existing images from the images undercloud on the stack user’s home (/home/stack/images).

   $ rm -rf ~/images/*

3. Extract the new image archives.

4. Install the Contrail package on the overcloud image by using the virt-customize command.

5. Import the new image archives into the undercloud and configure nodes to use the new images.

   $ openstack overcloud image upload -update-existing -image-path /home/stack/images/

6. Verify that the images are uploaded.

   $ openstack image list

   

7. Show the details of the new image that has been created. The new image will be used to add a new node in the overcloud.

   $ openstack image show overcloud-full

8. Verify contrail-status on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.

   [stack@undercloud ~]$ for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done

Upgrade the Operating System on Contrail Nodes

To upgrade the operating system on Contrail nodes:

1. Define a list ($iplist) that contains all Contrail nodes. Run the following command on undercloud VM as a stack user.

   Iplist=” @IPcontrailController1 @IPContrailController2 …”

   CAUTION:

   Attach the new satellite subscription key on all overcloud nodes before upgrading the overcloud packages. Satellite must be synced with filter end date 9th Dec 2019. Make sure to clear cache by typing sudo yum clean all.

2. Upgrade the operating system for all nodes in the iplist.

   Run the following command on undercloud VM as a stack user:

   sudo for ipnode in $iplist; do echo -e "\n\n\t******upgrade node : $ipnode ******" && ssh heat-admin@$ipnode 'sudo yum update -y --disablerepo=*contrail* --skip-broken && exit' ; done

3. (Compute nodes only) Reboot overcloud contrail compute nodes. After the reboot, stop the supervisor-vrouter service.

   This step needs to be performed before installing contrail packages on the compute VM.

   Compute services may go down after rebooting with the latest kernel. These services return later in this procedure during the openstack overcloud deploy process.

   Reboot Procedure:

   Post-Reboot:

   Stop the supervisor-vrouter service:

Prepare the Contrail Packages

Check the undercloud Contrail packages versions for contrail-tripleo-puppet, puppet-contrail, and contrail-tripleo-heat-templates.

Upgrade the Contrail Heat Templates

You must copy the new Contrail Heat templates package to the undercloud node, while retaining a copy of the Heat templates that were used for the existing deployment.

1. Make a copy of all of the Heat templates that were used for deployment and save the copies, because the existing files will be overwritten by the new versions. The templates to copy are of the form contrail-services.yaml, contrail-net.yaml, and so on.

   Note:

   Red Hat does not support changing IP address of the existing cluster as a part of upgrade. Do not change IP address of the cluster while creating new tripleo-heat-templates

2. Copy the new contrail-tripleo-heat templates to the undercloud node.

   Note:

   The directory /home/stack/tripleo-heat-templates is user defined, it can be User Defined-directory>/openstack-tripleo-heat-templates

Modify the Yum Update Script for TripleO Puppet

Following Puppet commands must be added to the yum_update script before starting the upgrade. The script is located at:

/home/stack/tripleo-heat-templates/extraconfig/tasks/yum_update.sh

1. Update the following Puppet commands in the yum_update.sh after the line “echo -n "false" > $heat_outputs_path.update_managed_packages”.

   Refer to the following patch for details regarding the exact placement of the commands patch: https://github.com/Juniper/contrail-tripleo-heat-templates/blob/stable/newton/environments/contrail/yum_updates.patch

   yum install -y contrail-tripleo-puppet puppet-contrail

   rsync -a /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/

2. Update the fields *contrail version and *contrail repo in contrail-services.yaml.

   Default parameter for contrailVersion is 4.

   Filename:/home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml.

   Add the following parameters:

   ContrailVersion: 4

   ContrailRepo : <location of the contrail-41 repo>

   Note:

   /home/stack/tripleo-heat-templatesdirectory is user defined and it can be directory name under stack user.

Update the Overcloud Deployment Plan

1. Update the current plan by re-running the command used for cloud deployment and adding the suffix - -update-plan-only.

   Example:

2. Make a copy of the existing deploy script to the update-stack.sh. The update-stack.sh is the script used to update the overcloud plan, and it references the same templates that were used to deploy the stack. All files used for the overcloud update should be identical to the files used for deployment, except contrail-services file that was updated with the latest contrail-version and contrail-repo.

   cp deploy.sh update-stack.sh

3. Update the deployment plan.

   ./update-stack.sh

   Example:

Upgrade the Overcloud

1. Update the overcloud stack.

2. Verify the overcloud stack status, the contrail-status, and the contrail-version after the upgrade.

   Overcloud Stack Status

   Note:

   The openstack stack list output presentation has been modified for readability. The ID field was removed as part of this modification.

   Contrail Stack Status

   Contrail Version Check

Upgrade Cautions

The upgrade procedure may fail due to packages conflicts in Contrail analytics nodes. Some observed failures due to packages conflicts are detailed in this section. Continue with the deployment after applying the recommended solution.

• Analytics Node snmp-lib Version Conflict
• Services Need Manual Restart After Upgrade