Upgrade Procedure for RHOSP-based Contrail 4.1.2 to Contrail 4.1.3

Prerequisite

Before upgrading to Contrail Release 4.1.3, you must update the net-snmp package to the net-snmp #37 version. The following net-snmp packages must be available in the upgrade repository and are installed automatically on Contrail Analytics nodes during the upgrade process:

• net-snmp-5.7.2-37.el7.x86_64.rpm

• net-snmp-agent-libs-5.7.2-37.el7.x86_64.rpm

• net-snmp-libs-5.7.2-37.el7.x86_64.rpm

• net-snmp-utils-5.7.2-37.el7.x86_64.rpm

Ensure you have a cloud up and running with RHOSP10 and Contrail 4.1.2 before you proceed with the upgrade procedure.

This procedure has been validated with the following Contrail, Red Hat, and OpenStack versions.

Upgrade the Undercloud

Upgrade the undercloud to the most current RHOSP10 version.

1. Log in to the undercloud as the stack user.

   $ su – stack

2. Update the Contrail repositories.

   $ sudo rm –rf /etc/yum.repos.d/*contrail*

   $ curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.repo

3. Stop the main OpenStack platform services.

   $ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

4. Update the python-tripleoclient package and its dependencies to ensure you have the most current scripts for the minor version update.

   $ sudo yum update python-tripleoclient

5. Upgrade the undercloud.

   $ openstack undercloud upgrade

6. Reboot the node.

   $ sudo reboot

7. Wait until the node reboots, then check the status of all services.

   Note:

   It can take as much as 10 minutes or more for the openstack-nova-compute to become active after a reboot.

   $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

8. Verify the version of RHEL after the undercloud upgrade.

   Note:

   Contrail does not support undercloud Red Hat version running with RHEL-7.6 as part of Contrail 4.1.3 release.

9. Verify the existence of the overcloud and its nodes.

   $ openstack stack list

   $ ironic node-list

10. Verify that all OpenStack servers are Active.

    $ openstack server list

    Figure 1: Server List

Update Red Hat Director Image Archives

The undercloud update process might download new image archives from the rhosp-director images and the rhosp-director-ipa packages. You will have to update your existing system with any new image archives.

1. Check the yum log to determine if new image archives are available.

   $ sudo grep "rhosp-director-images" /var/log/yum.log

2. If new image archives are available, replace your current images with the new images. Before deploying any new images, remove any existing images from the images undercloud on the stack user’s home (/home/stack/images).

   $ rm -rf ~/images/*

3. Extract the new image archives.

4. Install the Contrail package on the overcloud image by using the virt-customize command.

5. Import the new image archives into the undercloud and configure nodes to use the new images.

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

6. Verify that the images are uploaded.

   $ openstack image list

   

7. Show the details of the new image that has been created. The new image will be used to add a new node in the overcloud.

   $ openstack image show overcloud-full

   

8. Verify contrail-status on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.

   [stack@undercloud ~]$ for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done

Prepare Repositories on all Nodes

1. Delete existing repositories on all overcloud nodes. Verify each deletion.

   sudo for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode 'find /etc/yum.repos.d/ ! -name 'contrail-install.repo' -type f -exec sudo rm -f {} +' ; done

2. Add new repositories on all overcloud nodes. Verify each addition.

   sudo for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode ' curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.rep' ; done

Upgrade the Operating System on Contrail Nodes

1. Define a list ($iplist) that contains all Contrail nodes. Run the following command on undercloud VM as stack user.

   Iplist=” @IPcontrailController1 @IPContrailController2 …”

   CAUTION:

   Attach the new satellite subscription key on all overcloud nodes before upgrading the overcloud packages. Satellite must be synced with filter end date 29th Oct 2018. Make sure to clear cache by typing sudo yum clean all.

2. Upgrade the operating system for all nodes in the iplist. Run the following command on undercloud VM as stack user

   sudo for ipnode in $iplist; do echo -e "\n\n\t******upgrade node : $ipnode ******" && ssh heat-admin@$ipnode 'sudo yum update -y --disablerepo=*contrail* --skip-broken && exit' ; done

3. Reboot overcloud contrail compute nodes, if there is any change in the kernel version. This needs to be done before installing contrail packages on compute VM.

   Supported kernel versions: 3.10.0-862.11.6.el7.x86_64 and 3.10.0-957.el7.x86_64 ------------------------------- [root@overcloud-novacompute-2 ~]# modinfo vrouter filename: /lib/modules/3.10.0-862.11.6.el7.x86_64/extra/net/vrouter/vrouter.ko version: 4.1.3.0 license: GPL retpoline: Y rhelversion: 7.5

Prepare the Contrail Packages

To prepare the Contrail packages for the installation from a local repository:

1. Navigate to the Contrail repository and perform the following tasks:

   • Delete the existing Contrail repositories.

     All existing repositories in the undercloud and overcloud will be deleted during these steps.

   • Access the Contrail update package.

   • Copy the SNMP packages into the repository:

     • net-snmp-5.7.2-37.el7.x86_64.rpm

     • net-snmp-agent-libs-5.7.2-37.el7.x86_64.rpm

     • net-snmp-libs-5.7.2-37.el7.x86_64.rpm

     • net-snmp-utils-5.7.2-37.el7.x86_64.rpm

     In the provided example, all 4 of these files are in the /mnt/net-snmp/ directory and all files from the directory are copied into the repository.

   • Unsubscribe every node with all registered satellite server repositories.

   • Delete all repositories on undercloud and overcloud nodes, and replace these deleted repositories with a Contrail repository.

   • Clean the yum cache, verify the repository list, and check for yum updates.

   A sample procedure:

2. Check the undercloud Contrail packages versions for contrail-tripleo-puppet, puppet-contrail, and contrail-tripleo-heat-templates. The newest versions of those packages must be installed before proceeding with the overcloud upgrade. See the following example, with current package versions.

Upgrade the Contrail Heat Templates

You must copy the new Contrail Heat templates package to the undercloud node, while retaining a copy of the Heat templates that were used for the existing deployment.

1. Make a copy of all of the Heat templates that were used for deployment and save the copies, because the existing files will be overwritten by the new versions. The templates to copy are of the form contrail-services.yaml, contrail-net.yaml, and so on.

   Note:

   Red Hat does not support changing IP address of the existing cluster as a part of upgrade. Do not change IP address of the cluster while creating new tripleo-heat-templates

2. Copy the new contrail-tripleo-heat templates to the undercloud node.

   Note:

   The directory /home/stack/tripleo-heat-templates is user defined, it can be User Defined-directory>/openstack-tripleo-heat-templates

Modify the Yum Update Script for TripleO Puppet

Following Puppet commands must be added to the yum_update script before starting the upgrade. The script is located at:

/home/stack/tripleo-heat-templates/extraconfig/tasks/yum_update.sh

1. Update the following Puppet commands in the yum_update.sh after the line “echo -n "false" > $heat_outputs_path.update_managed_packages”.

   Refer to the following patch for details regarding the exact placement of the commands patch: https://github.com/Juniper/contrail-tripleo-heat-templates/blob/stable/newton/environments/contrail/yum_updates.patch

   yum install -y contrail-tripleo-puppet puppet-contrail

   rsync -a /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/

2. Update the fields *contrail version and *contrail repo in contrail-services.yaml.

   Default parameter for contrailVersion is 4.

   Filename:/home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml.

   Add the following parameters:

   ContrailVersion: 4

   ContrailRepo : <location of the contrail-41 repo>

   Note:

   /home/stack/tripleo-heat-templatesdirectory is user defined and it can be directory name under stack user.

Update the Overcloud Deployment Plan

1. Make a copy of the existing deploy script to the update-stack.sh file by re-running the command used for cloud deployment and adding the suffix - -update-plan-only.

   Example:

2. If you are using a local repository for the update and the environment-rhel-registration.yaml and rhel-registration-resource-registry.yaml files are present, delete these lines from the deploy script:

3. Prepare the YAML files for the update:

   • Verify each .yaml template referenced in the update-stack.sh file contains the original settings that match the files that were backed up.

   • In the contrail-net.yaml file, adapt all referenced templates from heat_template_version: newton to heat_template_version: 2015-04-30. Keep all other original installation settings in this file.

4. Update the deployment plan.

   ./update-stack.sh

   Example

Upgrade the Overcloud

1. Update the overcloud stack.

2. Verify the overcloud stack status, the contrail-status, and the contrail-version after the upgrade.

   Overcloud Stack Status

   Contrail Stack Status

   Contrail Version Check