ON THIS PAGE
Upgrade Procedure for RHOSP-based Contrail 4.1.1 to Contrail 4.1.2
This section presents the steps to upgrade an OSP-based Contrail deployment from Contrail version 4.1.1 to Contrail version 4.1.2.
Prerequisite
Ensure you have a cloud up and running with RHOSP10 Z4 and Contrail 4.1.1 before you proceed with the upgrade procedure.
This procedure has been validated with the following Contrail, Red Hat, and OpenStack versions.
Contrail Version |
Red Hat Version |
OpenStack Version |
---|---|---|
3.2.3 |
RHEL 7.3 |
RHOSP10 (packages dated Apr. 15, 2017) |
3.2.6 |
RHEL 7.4 |
RHOSP10 (packages dated Feb. 2, 2018) |
4.1 |
RHEL 7.4 |
RHOSP10 (packages dated Feb. 27, 2018) |
4.1.1 |
RHEL 7.5 |
RHOSP10 (packages dated Jun. 4, 2018) RHOSP11 (packages dated Jun. 4, 2018) |
4.1.2 |
RHEL 7.5 |
RHOSP10 (packages dated Oct 29, 2018) |
4.1.3 |
RHEL 7.5 |
RHOSP10 (packages dated Oct 29, 2018) |
Set the Red Hat Satellite filter end date to October 29, 2018 before proceeding with the upgrade.
Upgrade the Undercloud
Upgrade the undercloud to the most current RHOSP10 version.
Log in to the undercloud as the stack user.
$ su – stack
Update the Contrail repositories.
$ sudo rm –rf /etc/yum.repos.d/*contrail*
$ curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.repo
Stop the main OpenStack platform services.
$ sudo systemctl stop 'openstack-*' 'neutron-*' httpd
Update the
python-tripleoclient
package and its dependencies to ensure you have the most current scripts for the minor version update.$ sudo yum update python-tripleoclient
Upgrade the undercloud.
$ openstack undercloud upgrade
Reboot the node.
$ sudo reboot
Wait until the node reboots, then check the status of all services.
Note:It can take as much as 10 minutes or more for the
openstack-nova-compute
to become active after a reboot.$ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"
Verify the version of RHEL after the undercloud upgrade.
Note:Contrail does not support undercloud Red Hat version running with RHEL-7.6 as part of Contrail 4.1.2 release.
[root@undercloud ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) [root@undercloud ~]#
Verify the existence of the overcloud and its nodes.
$ openstack stack list
$ ironic node-list
Verify that all OpenStack servers are Active.
$ openstack server list
Figure 1: Server List
Update Red Hat Director Image Archives
The undercloud update process might download new image archives from the rhosp-director images and the rhosp-director-ipa packages. You will have to update your existing system with any new image archives.
Check the yum log to determine if new image archives are available.
$ sudo grep "rhosp-director-images" /var/log/yum.log
[stack@undercloud]$ sudo grep "rhosp-director-images" /var/log/yum.log Oct 26 15:09:20 Installed: rhosp-director-images-ipa-10.0-20180821.1.el7ost.noarch Oct 26 15:10:10 Installed: rhosp-director-images-10.0-20180821.1.el7ost.noarch
If new image archives are available, replace your current images with the new images. Before deploying any new images, remove any existing images from the images undercloud on the stack user’s home (/home/stack/images).
$ rm -rf ~/images/*
Extract the new image archives.
mkdir images cd images for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done
Install the Contrail package on the overcloud image by using the virt-customize command.
export LIBGUESTFS_BACKEND=direct /usr/bin/virt-customize -a /home/stack/images/overcloud-full.qcow2 \ --copy-in /etc/yum.repos.d/mylocalrepo.repo:/etc/yum.repos.d \ --run-command 'yum -y install puppet-tripleo contrail-tripleo-puppet puppet-contrail‘\ --run-command ‘ cp -r /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/ ‘ \ --run-command 'rm -fr /var/cache/yum/*' \ --run-command 'yum clean all' \ --selinux-relabel
Import the new image archives into the undercloud and configure nodes to use the new images.
$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
Verify that the images are uploaded.
$ openstack image list
Show the details of the new image that has been created. The new image will be used to add a new node in the overcloud.
$ openstack image show overcloud-full
Verify
contrail-status
on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.[stack@undercloud ~]$ for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done
Prepare Repositories on all Nodes
Delete existing repositories on all overcloud nodes. Verify each deletion.
sudo for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode 'find /etc/yum.repos.d/ ! -name 'contrail-install.repo' -type f -exec sudo rm -f {} +' ; done
Add new repositories on all overcloud nodes. Verify each addition.
sudo for ipnode in $(nova list | sed '4,$ !d;$d'| awk -F 'ctlplane=' '{print $2}' | tr -d '|'); do echo cleaning yum repolist on $ipnode && ssh heat-admin@$ipnode ' curl http://newrepo.contrail41-dev.repo -o /etc/yum.repos.d/localrepo.rep' ; done
Upgrade the Operating System on Contrail Nodes
Define a list ($iplist) that contains all Contrail nodes. Run the following command on undercloud VM as stack user.
Iplist=” @IPcontrailController1 @IPContrailController2 …”
CAUTION:Attach the new satellite subscription key on all overcloud nodes before upgrading the overcloud packages. Satellite must be synced with filter end date 29th Oct 2018. Make sure to clear cache by typing sudo yum clean all.
Upgrade the operating system for all nodes in the iplist. Run the following command on undercloud VM as stack user
sudo for ipnode in $iplist; do echo -e "\n\n\t******upgrade node : $ipnode ******" && ssh heat-admin@$ipnode 'sudo yum update -y --disablerepo=*contrail* --skip-broken && exit' ; done
Reboot overcloud contrail compute nodes, if there is any change in the kernel version. This needs to be done before installing contrail packages on compute VM.
Supported kernel versions: 3.10.0-862.11.6.el7.x86_64 and 3.10.0-957.el7.x86_64 ------------------------------- [root@overcloud-novacompute-2 ~]# modinfo vrouter filename: /lib/modules/3.10.0-862.11.6.el7.x86_64/extra/net/vrouter/vrouter.ko version: 4.1.2.0 license: GPL retpoline: Y rhelversion: 7.5
Prepare the Contrail Packages
Check the undercloud Contrail packages versions for contrail-tripleo-puppet
, puppet-contrail
, and contrail-tripleo-heat-templates
.
The newest versions of those packages must be installed before proceeding
with the overcloud upgrade. See the following example, with current
packages versions.
[stack@undercloud~]$ rpm -qa | grep contrail puppet-contrail-4.1.2.0-NN.el7.noarch contrail-tripleo-heat-templates-4.1.2.0-NN.el7.noarch contrail-tripleo-puppet-4.1.2.0-NN.el7.noarch
Upgrade the Contrail Heat Templates
You must copy the new Contrail Heat templates package to the undercloud node, while retaining a copy of the Heat templates that were used for the existing deployment.
Make a copy of all of the Heat templates that were used for deployment and save the copies, because the existing files will be overwritten by the new versions. The templates to copy are of the form contrail-services.yaml, contrail-net.yaml, and so on.
Note:Red Hat does not support changing IP address of the existing cluster as a part of upgrade. Do not change IP address of the cluster while creating new tripleo-heat-templates
Copy the new contrail-tripleo-heat templates to the undercloud node.
sudo cp -r /usr/share/contrail-tripleo-heat-templates/environments/contrail /home/stack/tripleo-heat-templates/environments/ sudo cp -r /usr/share/contrail-tripleo-heat-templates/puppet/services/network/* /home/stack/tripleo-heat-templates/puppet/services/network
Note:The directory /home/stack/tripleo-heat-templates is user defined, it can be User Defined-directory>/openstack-tripleo-heat-templates
Modify the Yum Update Script for TripleO Puppet
Following Puppet commands must be added to the yum_update
script before starting the upgrade. The
script is located at:
/home/stack/tripleo-heat-templates/extraconfig/tasks/yum_update.sh
Update the following Puppet commands in the
yum_update.sh
after the line“echo -n "false" > $heat_outputs_path.update_managed_packages”
.Refer to the following patch for details regarding the exact placement of the commands patch: https://github.com/Juniper/contrail-tripleo-heat-templates/blob/stable/newton/environments/contrail/yum_updates.patch
yum install -y contrail-tripleo-puppet puppet-contrail
rsync -a /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/
Update the fields
*contrail version
and*contrail repo
incontrail-services.yaml
.Default parameter for
contrailVersion
is 4.Filename:/home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml.
Add the following parameters:
ContrailVersion: 4
ContrailRepo : <location of the contrail-41 repo>
Note:/home/stack/tripleo-heat-templatesdirectory is user defined and it can be directory name under stack user.
Update the Overcloud Deployment Plan
Update the current plan by re-running the command used for cloud deployment and adding the suffix
- -update-plan-only
.<openstack overcloud deploy> –update-plan-only
Example
openstack overcloud deploy --update-plan-only --templates /home/stack/tripleo-heat-templates/ \ --roles-file /home/stack/tripleo-heat-templates/environments/contrail/roles_data.yaml \ -e /home/stack/tripleo-heat-templates/environments/puppet-pacemaker.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/network-isolation.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/contrail-net.yaml \ -e /home/stack/tripleo-heat-templates/environments/ips-from-pool-all.yaml \ -e /home/stack/tripleo-heat-templates/environments/network-management.yaml \ -e /home/stack/tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml \ -e /home/stack/tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml \ --libvirt-type qemu
Make a copy of the existing deploy script to the
update-stack.sh
. Theupdate-stack.sh
is the script used to update the overcloud plan, and it references the same templates that were used to deploy the stack. All files used for the overcloud update should be identical to the files used for deployment, exceptcontrail-services
file that was updated with the latestcontrail-version
andcontrail-repo
.cp deploy.sh update-stack.sh
Update the deployment plan.
./update-stack.sh
Example
[stack@undercloud-R4-1-1-upg-R4-1-2 ~]$ ./update-stack.sh Removing the current plan files Uploading new plan files Started Mistral Workflow. Execution ID: 6c8fb5b7-6eda-4d92-8245-f7ac46bb369d Plan updated Deploying templates in the directory /tmp/tripleoclient-CdyN2I/tripleo-heat-templates Overcloud Endpoint: http://10.87.67.232:5000/v2.0 Overcloud Deployed [stack@undercloud-R4-1-1-upg-R4-1-2 ~]$
Upgrade Cautions
The steps to perform the overcloud upgrade are service disrupting, and should only be performed within a maintenance window.
The upgrade procedure may fail due to packages conflicts in Contrail analytics nodes. Some observed failures due to packages conflicts are detailed in this section. Continue with the deployment after applying the recommended solution.
Analytics Node snmp-lib Version Conflict
Error message: Protected multilib versions: 1:net-snmp-libs-5.7.2-37.el7.x86_64
!= 1:net-snmp-libs-5.7.2-33.el7_5.2.i686
Solution:
rpm -e --nodeps net-snmp-libs
Upgrade the Overcloud
The steps in this section are service disrupting, and should only be performed within a maintenance window.
Update the overcloud stack.
$ openstack overcloud update stack -i overcloud on_breakpoint: [u'overcloud-contrailanalyticsdatabase-0'] Breakpoint reached, continue? Regexp or Enter=proceed (will clear 4386bdc7-5087-4a4d-865c-0b0181ce9345), no=cancel update, C-c=quit interactive mode: IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS COMPLETE update finished with status COMPLETE
Verify the overcloud
stack status
, thecontrail-status
, and thecontrail-version
after the upgrade.Overcloud Stack Status
[stack@undercloud]# heat stack-list +--------------------------------------+------------+-----------------+----------------------+----------------------+ | ID | Stack Name | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+-----------------+----------------------+----------------------+ | e56c5512-3e32-4940-a05a-e194f48ce67a | overcloud | UPDATE_COMPLETE | 2018-02-16T20:32:17Z | 2018-02-21T01:12:08Z | +--------------------------------------+------------+-----------------+----------------------+----------------------+ [stack@undercloud ~]$
Contrail Stack Status
sudo for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done
Contrail Version Check
for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-version; done