Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Upgrading Contrail Networking Release 5.x or Release 190x with RHOSP13 to Contrail Networking Release till 1909 with RHOSP13

 

This document provides steps to upgrade Contrail Networking from Release 5.x or Release 190x with RHOSP13 till Release 1909 with RHOSP13.

Additionally, the procedure is also applicable to upgrade Contrail Networking Release 5.0 with RHOSP13 to Release 5.1 or Release 19xx with RHOSP13.

Before you begin:

  • Access ContrailImageTag located at README Access to Contrail Registry 19XX  .

  • Enable RHEL subscription for the overcloud nodes.

  • Enable SSH migration for the Compute nodes if you do not have CEPH or alike storage.

    Upgrading the compute nodes requires workload migrations and CEPH or alike storage allows VM migration.

  • Take backup of the Contrail configuration database.

Procedure:

  1. Get Contrail TripleO Heat Templates (Stable/Queens branch) from https://github.com/Juniper/contrail-tripleo-heat-templates.
  2. Prepare docker registry with Contrail Networking R1908 images. It can be undercloud or a separate node.
  3. Deploy Contrail Controller In-Service Software Upgrade (ISSU) node.
    1. Prepare new bare-metal server node and create flavor contrail-controller-issu for the ISSU node.

      The hardware requirements for ISSU node is the same as for the Contrail Controller Node.
    2. Prepare the parameters in the yaml file, ~/tripleo-heat-templates/environments/contrail/contrail-issu.yaml :
      • ContrailIssuSshKey—Generate and set the ssh keys (You require SSH access between ISSU and Contrail Controller nodes for ISSU).

      • ContrailIssuImageTag—Set the new docker images tag for the upgrade procedure.

      • ContrailControllerIssuCount—Set the required number of ISSU nodes. The value can be 1 or 3 and is dependent on various cluster requirements including cluster size, expected upgrade duration, etc.

    3. Update ServiceNetMap parameter in the ~/tripleo-heat-templates/environments/contrail/contrail-services.yaml file.

      ContrailIssuControlNetwork—Set the same value as ContrailControlNetwork. The default value is tenant.

    4. Run deploy command with all the parameters used for deployment and the new environment file.
      openstack deploy ...\

      -e ~/tripleo-heat-templates/environments/contrail/contrail-issu.yam
    5. Check the status of Contrail service on the ISSU node.

      All services must be active .

      sudo contrail-status

  4. Prepare for the upgrade procedure.
    1. Update the parameter ContrailImageTag to the new version.
      vi ~/tripleo-heat-templates/environments/contrail/contrail-services.yaml
    2. Run prepare command with all the options from the ISSU node.
      --stack overcloud \ …

      -e ~/tripleo-heat-templates/environments/contrail/contrail-issu.yam
  5. Run In-Service Software Upgrade (ISSU).
    1. Make SSH connection to ISSU node.Note

      If you have 3 ISSU nodes deployed, you must perform SSH operations and run scripts on the same node for the entire procedure.

    2. Locate ISSU directory.

      cd /etc/contrail/issu

    3. Pair ISSU node with the old cluster.

      ./issu_node_pair.sh

    4. Check the status of Contrail service on the ISSU node.

      sudo contrail-status

      All services must be active except config_devicemgr, config_schema and config_svcmonitor.

    5. Run ISSU sync container.

      ./issu_node_sync.sh

    6. Check ISSU container logs.
      sudo docker logs issu-run-sync
      sudo docker exec issu-run-sync cat /var/log/contrail/issu_contrail_run_sync.log
    7. Restart contrail_control_control container on all the ISSU nodes.
      openstack server list --name issu -c Networks -f value | cut -d'=' -f2 | xargs -i ssh heat-admin@{} sudo docker restart contrail_control_control
  6. Upgrade the Compute nodes.

    Perform these steps on all the Compute Nodes.

    1. Select the Compute node for upgrade and migrate workload from it.
    2. Verify the migrated instance has active state.
      openstack server show instance_52
    3. Upgrade the selected Compute Nodes.

      You can use comma seperated list for the various Compute nodes.

      nodes=overcloud-novacompute-0

      openstack overcloud upgrade run --nodes $nodes --playbook upgrade_steps_playbook.yaml

      openstack overcloud upgrade run --nodes $nodes --playbook deploy_steps_playbook.yaml
    4. Re-initialize vhost0 interfaces on all the DPDK mode compute nodes.

      Make SSH connection to the upgraded Compute nodes and run the following commands:

      ifdown vhost0

      Ifup vhost0
    5. Check the status of Contrail service on the upgraded Compute nodes.

      sudo contrail-status

      The status must be active.

  7. Upgrade Contrail Plugins including Neutron, Heat, etc. on OpenStack controllers and connect them to the ISSU node.
    nodes=overcloud-controller-0

    openstack overcloud upgrade run --nodes $nodes --playbook upgrade_steps_playbook.yaml

    openstack overcloud upgrade run --nodes $nodes --playbook deploy_steps_playbook.yaml
  8. Disconnect the ISSU node from the Contrail control plane.
    1. Make SSH connection to ISSU node.
    2. Run the following commands:
      ./issu_node_sync_post.sh

      ./issu_node_pair.sh del
    3. Check the status of Contrail service on the ISSU node.

      sudo contrail-status

      The status must be active or backup.

  9. Upgrade the Contrail control plane node.
    1. Make SSH connection to the ISSU node.
    2. Run the following commands:
      cd /etc/contrail/issu/

      nodes=overcloud-contrailcontroller-0,overcloud-contrailcontroller-1,overcloud-contrailcontroller-2

      openstack overcloud upgrade run --nodes $nodes --playbook upgrade_steps_playbook.yaml

      openstack overcloud upgrade run --nodes $nodes --playbook deploy_steps_playbook.yaml

      openstack overcloud upgrade run --nodes $nodes --playbook post_upgrade_steps_playbook.yaml
    3. Check the status of Contrail service on the Contrail control plane node.

      sudo contrail-status

      The status must be active or backup.

  10. Connect the ISSU node to the upgraded Contrail control plane node.
    1. Make SSH connection to the ISSU node.
    2. Pair the ISSU node with upgraded Contrail control plane.

      cd /etc/contrail/issu

      ./issu_node_pair.sh add pair_with_new

    3. Sync data with new Contrail control plane.

      issu_config=issu_revert.conf ./issu_node_sync.sh

    4. Restart control container on the upgraded nodes.

      openstack server list --name "overcloud-contrailcontroller-" -c Networks -f value | cut -d'=' -f2 | xargs -i ssh heat-admin@{} sudo docker restart contrail_control_control

  11. Run the following commands on the Compute nodes and OpenStack controllers.
    nodes=overcloud-novacompute-0,overcloud-novacompute-1

    openstack overcloud upgrade run --nodes $nodes --playbook post_upgrade_steps_playbook.yaml

    nodes=overcloud-controller-0 openstack overcloud upgrade run --nodes $nodes --playbook post_upgrade_steps_playbook.yaml
  12. Disconnect ISSU and upgraded Contrail control plane.
    1. Make SSH connection to ISSU node.
    2. Un-pair ISSU node with the old Contrail cluster.

      issu_config=issu_revert.conf ./issu_node_sync_post.sh

      ./issu_node_pair.sh del pair_with_new

  13. Reconnect the OpenStack nodes and Compute nodes to the upgraded control plane.

    Run the command with all the parameters from deploy.

    openstack overcloud upgrade converge \

    --stack overcloud \

    ...

    -e ~/tripleo-heat-templates/environments/contrail/contrail-issu.yaml
  14. Check the status of Contrail service on all the upgrades nodes.

    sudo contrail-status

    The status must be active.

  15. Remove the ISSU node from the cluster.

    set ContrailControllerIssuCount: 0

    Run stack deploy command with all the parameters.

    openstack overcloud deploy \



    -e ~/tripleo-heat-templates/environments/contrail/contrail-issu.yaml