Upgrade Procedure for Ubuntu-based Contrail 4.1.3 to Contrail 4.1.4 Using Juju with Netronome SmartNIC
This section presents the steps to upgrade from an Ubuntu-16.04-based Contrail deployment from Contrail version 4.1.3 to Contrail version 4.1.4 using Juju charms.
Prerequisites
These instructions assume that these requirements for installing Contrail Release 4.1.3 are already present in your environment:
MaaS Server—MaaS version 2.3 is installed on this server. This procedure was tested using MaaS version 2.3.5.
Juju Controller—Juju version 2.3 is installed, and the Juju controller is bootstrapped on a VM or a bare metal server. This procedure was tested using Juju version 2.3.7-xenial-amd64.
A repository to get Netronome, patched Openstack packages, and Contrail vRouter packages is operational.
A Contrail Controller using Ubuntu 16.04 xenial is operational.
A Contrail cluster with one or more compute nodes using Agilio SmartNICs.
Acquire the Software
To acquire the Contrail 4.1.4 software images to perform this procedure:
Go to the Juniper Networks Support site for Contrail.
Select OS as Contrail and Version as 4.1.4 from the drop-down menus.
Download the contrail-cloud-docker_4.1.4.0-63-ocata_xenial.tgz file.
Extract the following images from the contrail-cloud-docker_4.1.4.0-63-ocata_xenial.tgz file:
Contrail Analytics package: contrail-analytics-ubuntu16.04-4.1.4.0-63.tar.gz.
Contrail Analytics Database package: contrail-analyticsdb-ubuntu16.04-4.1.4.0-63.tar.gz .
Contrail Controller package: contrail-controller-ubuntu16.04-4.1.4.0-63.tar.gz
The images need to be extracted because the Contrail Analytics, Contrail Analytics Database, and Contrail Controller packages must be upgraded individually to perform this upgrade.
Attach Contrail Packages using Juju
The Contrail Controller, Contrail Analytics, and Contrail Analytics DB packages need to be attached using Juju to perform this upgrade.
To attach these software packages into Juju:
Attach the Contrail Controller, Contrail Analytics, & Contrail Analytics DB packages into Juju:
juju attach contrail-analytics contrail-analytics=/home/jenkins/docker/contrail-analytics-ubuntu16.04-4.1.4.0-63.tar.gz juju attach contrail-controller contrail-controller=/home/jenkins/docker/contrail-controller-ubuntu16.04-4.1.4.0-63.tar.gz juju attach contrail-analyticsdb contrail-analyticsdb=/home/jenkins/docker/contrail-analyticsdb-ubuntu16.04-4.1.4.0-63.tar.gz
Check status of the software image attachments into Juju using the
juju status
command.Wait for the
juju status
command output to indicate that the upgrade is successful. The output in thejuju status
should indicate that all processes are Active and all machine states are started.
Upgrade the Contrail Clusters
This section provides the steps to update the Contrail clusters for this upgrade.
It includes the following sections:
- Upgrade the Contrail Controllers
- Upgrade Contrail Analytics Nodes
- Upgrade Analytics Database Nodes
- Updating the Neutron Plugin and the vRouter Agent
Upgrade the Contrail Controllers
The Contrail controllers must be upgraded one by one to complete this procedure.
To upgrade the Contrail controllers:
SSH into the Contrail controller server and decommission the Contrail controller from the Cassandra cluster:
sudo docker exec -it contrail-controller /usr/bin/nodetool decommission
Remove the Contrail Controller container:
sudo docker rm -f contrail-controller
Update the hooks to the Contrail Controller from the Juju Controller:
juju run --application contrail-controller hooks/update-status
Wait for the Contrail status for all packages on the upgrading node to change to active. This step can take up to 10 minutes.
Enter the contrail-status command to check status. All packages in the Contrail Control section of the output must move to the active state before proceeding.
Check Juju status by entering the
juju status
command.All Contrail components in this output should be in the active state.
After each controller update, check the controllers to make sure the databases are consistent across all controllers:
Enter the nodetool describecluster command. Confirm that the schema version output is identical on all 3 controllers.
Enter the echo stat | nc localhost 2181 command. The node count output should be identical on all 3 controllers.
Ensure that the contrail-status output is active for all components in all 3 controllers.
If your upgrade is not successful after 15 minutes, retry steps 1 through 5.
If you need to decommission a node that is not upgrading successfully, use the nodetool removenode node-ID command.
Repeat steps 1 through 6 for all other Contrail controller nodes.
Upgrade Contrail Analytics Nodes
To upgrade the Contrail Analytics nodes:
SSH into the first Contrail Analytics node and remove the Contrail Analytics container:
sudo docker rm -f contrail-analytics
Confirm Juju status using the
juju status
command.The output in the
juju status
should indicate that all processes are Active and all machine states are started.From the MaaS server, update hooks to the Contrail Analytics controller:
juju run --application contrail-analytics/0 hooks/update-status
Wait for the Contrail status for all packages on the upgrading node to change to active. This step can take up to 10 minutes.
Enter the contrail-status command to check status. All packages in the Contrail Analytics section of the output must move to the active state before proceeding.
Repeat steps 1 through 4 for all other Contrail Analytics nodes.
Upgrade Analytics Database Nodes
To upgrade the Contrail Analytics database nodes:
SSH into a Contrail analytics database server and decommission the node from the Cassandra cluster:
sudo docker exec -it contrail-analyticsdb /usr/bin/nodetool decommission
Remove the AnalyticsDB container:
sudo docker rm -f contrail-analyticsdb
From the Juju controller, update the hooks to the Contrail Analytics DB controller:
juju run --application contrail-analyticsdb hooks/update-status
Wait for the Contrail status for all packages on the upgrading node to change to active. This step can take up to 10 minutes.
Enter the contrail-status command to check status. All packages in the Contrail Database section of the output must move to the active state before proceeding.
Check Juju status by entering the
juju status
command.All Contrail components in this output should be in the active state.
After each analytics database node update, check the nodes to ensure the databases are consistent inside the contrail analytics database containers:
Enter the nodetool describecluster command. Confirm that the schema version output is identical on all 3 nodes.
Enter the echo stat | nc localhost 2181 command. The node count output should be identical on all 3 nodes.
Ensure that the contrail-status output is active for all components in all 3 contrail analytics db nodes.
If your upgrade is not successful after 15 minutes, retry steps 1 through 5.
If you need to decommission a node that is not upgrading successfully, use the nodetool removenode node-ID command.
Repeat steps 1 through 6 for all other Contrail Analytics database nodes.
Updating the Neutron Plugin and the vRouter Agent
The process for updating the neutron plugin and the vRouter agent is different for compute nodes than it is for other nodes.
This section covers both procedures and includes these sections:
Updating the Neutron Plugin and the vRouter Agent on Non-Compute Nodes
Use this procedure to update the Neutron Plugin and the vRouter agent on all non-compute nodes in your environment:
This procedure assumes that the APT Get repository was created during the previous installation, and that the latest Contrail packages can be placed into the repository.
SSH into the Neutron API plugin unit.
From the Neutron API plugin unit, get the latest APT Get update:
sudo apt-get update
Upgrade APT GET:
sudo apt-get upgrade
Note:This step shows how to upgrade APT get for all packages. You can also manually update the neutron-plugin-contrail and python-contrail packages to complete this step, if you’d rather not perform the complete upgrade. This procedure does not provide the steps to manually update these packages.
Restart the Neutron service:
sudo systemctl restart neutron-server.service
Updating the Neutron Plugin and the vRouter Agent on Compute Nodes
Use this procedure to update the Neutron Plugin and the vRouter agent on all compute devices in your environment:
This procedure assumes that the APT Get repository was created during the previous installation, and that the latest Contrail packages can be placed into the repository.
SSH into the Neutron API plugin unit.
From the Neutron API plugin unit, get the latest APT Get update:
sudo apt-get update
Upgrade APT GET:
sudo apt-get upgrade
Note:This step shows how to upgrade APT get for all packages. You can also manually update the following packages to complete this step:
contrail-lib
contrail-nodemgr
contrail-setup
contrail-utils
contrail-vrouter-agent
contrail-vrouter-common
contrail-vrouter-dkms
contrail-vrouter-init
contrail-vrouter-utils
python-contrail
python-contrail-vrouter-api
python-opencontrail-vrouter-netns
This procedure does not provide the steps to manually update these packages.
Upgrade the vRouter agent and, if using Netronome SmartNICs, the netronome plugin.
If you are performing this procedure on a compute node without a Netronome SmartNIC:
Note:The network connection over the vhost is down while this procedure is performed. Traffic will be lost.
Stop the Contrail vRouter agent:
sudo systemctl stop contrail-vrouter-agent
Remove the Contrail vRouter module:
sudo rmmod vrouter
Insert the vRouter module:
sudo insmod /lib/modules/4.4.0-116-generic/updates/dkms/vrouter.ko
Activate the vhost:
sudo ifup vhost0
Restart the Contrail vRouter agent:
sudo systemctl start contrail-vrouter-agent
If you are performing this procedure on a compute node with a Netronome SmartNIC:
Note:The network connection over the vhost is down while this procedure is performed. Traffic will be lost.
Stop the Contrail vRouter agent:
sudo systemctl stop contrail-vrouter-agent
Stop the Virtio forwarder module:
sudo systemctl stop virtio-forwarder
Stop the vRouter control module:
sudo /opt/netronome/bin/ns-vrouter-ctl stop
Restart the Virtio forwarder module:
sudo systemctl start virtio-forwarder
Restart the Contrail vRouter agent:
sudo /opt/netronome/bin/ns-vrouter-ctl start
Activate the vhost:
sudo ifup vhost0
Restart the Contrail vRouter agent:
sudo systemctl start contrail-vrouter-agent
Verify Contrail status:
sudo contrail-status
All packages in the Contrail vRouter section of the output should be in the active state. This step can take several minutes.