Guidance for Migrating to CentOS 7 for NorthStar 6.0.0 and Later

Example Scenario

For example purposes, these instructions assume you are migrating from CentOS 6.10 to CentOS 7.7, and your network configuration includes:

• Three NorthStar application servers in a cluster

• Three analytics servers in a cluster

• Three collector nodes

Your actual operating system version and network topology might be different, but the principles still apply.

We recommend backing up your operating system files and directories so you have a reference since some of the files differ between CentOS 6.x and CentOS 7.x. Back up these operating system files and directories, and save them to an external or network drive:

1. /etc/selinux/config

2. /etc/sysconfig/

3. /etc/hosts

4. /etc/ntp.conf

5. /etc/resolv.conf

6. /etc/ssh/

7. /root/.ssh/

Back up these NorthStar files and directories, and save them to an external or network drive:

1. /opt/pcs/db/sys/npatpw

2. /opt/northstar/data/northstar.cfg

3. /opt/northstar/data/*.json

4. /opt/northstar/data/junosvm.conf

5. /opt/northstar/northstar.env

6. /opt/northstar/thirdparty/netconfd/templates

7. /opt/northstar/saved_models (if used for saving NorthStar Planner projects)

The Basic Work Flow

For any node, whether it is a NorthStar application node, an analytics node, or a collector node, the work flow to upgrade your operating system while preserving your clusters and data is essentially the same:

1. Power down one standby node in the cluster setup.

2. Boot that node from the operating system minimal ISO.

   CentOS 7.7 minimal ISO is available here:

   1. http://mirrors.mit.edu/centos/7.7.1908/isos/x86_64/

   2. http://mirrors.tripadvisor.com/centos/7.7.1908/isos/x86_64/

3. Install the operating system on the node.

4. Run yum -y update to address any critical or security updates.

5. Install recommended packages:

6. Install the NorthStar 5.1.0 application on this same node, setting it up as a standalone host.

   Note:

   For NorthStar application nodes, you will need a new license because the interface names change from ethx to ensx when you upgrade the operating system. You will not need a new license for analytics or collector nodes.

7. For NorthStar application nodes, launch the web UI on the host https://northstar_ip_address:8443 to ensure the license is working and you can log in successfully.

8. You can check the status of the NorthStar processes by running the supervisorctl status command.

In this procedure, we have you start with upgrading the operating system on your analytics cluster, then your NorthStar application cluster, and your collector cluster last. However, this order is not a strict requirement. When all nodes in all clusters are running the upgraded operating system and NorthStar 5.1.0, you then upgrade to NorthStar 6.0.0 or later.

Upgrade the Operating System on Your Analytics Nodes

For analytics nodes, Elasticsearch will self-form the cluster and distribute the data per the replication policy. Therefore, there is no need to first delete the node from Elasticsearch history. To migrate your analytics cluster, use the following procedure:

1. Install CentOS 7.7 on a standby analytics node, including the previously stated recommended packages.

2. Install NorthStar-Bundle-5.1.0-20191210_220522_bb37a329b_64.x86_64.rpm on the node where you have the freshly installed operating system.

3. Copy the SSH keys from the existing active node in the analytics cluster and all application nodes to the new analytics node:

4. Working from an existing node in the cluster, add the new analytics node into the cluster:

   1. From net_setup.py, select Analytics Data Collector Setting (G) for external standalone/cluster analytics server setup.

   2. Select Add new Collector node to existing cluster (E).

      You can use the previous node’s ID and other setup information.

Once this process is completed for the first node, repeat the steps for the remaining analytics cluster nodes. Once the process is complete on all three nodes, your analytics cluster will be up and running with CentOS 7.7 and NorthStar 5.1.0.

The following are useful Elasticsearch (REST API) commands you can use before, during and after upgrading your operating system. Run these from an existing node in the analytics cluster.

1. curl -X GET "localhost:9200/_cluster/health?pretty"

2. curl -X GET "localhost:9200/_cat/nodes?v"

3. curl -X GET "localhost:9200/_cat/indices"

4. curl -X GET "localhost:9200/_cat/shards"

Use the following command to check that all nodes in your analytics cluster are up:

Upgrade the Operating System on Your NorthStar Application Nodes

Use the following procedure to upgrade your operating system on the NorthStar application nodes:

1. Install CentOS 7.7 on one of the NorthStar application standby nodes (server or VM), including the recommended packages listed previously.

2. Install the NorthStar 5.1.0 application software (NorthStar-Bundle-5.1.0-20191210_220522_bb37a329b_64.x86_64.rpm). It is important to provide the installation script with the same database password that is on the existing nodes. If necessary, you can reset the database passwords on the existing nodes for consistency before adding the node into the cluster.

   1. Install /opt/pcs/db/sys/nptapw and chown pcs.pcs /opt/pcs/db/sys/npatpw

      Copy your npatpw file to the location /opt/pcs/db/sys/npatpw. Then run the chown pcs:pcs /opt/pcs/db/sys/npatpw command.

   2. Update /opt/northstar/netconfd/templates.

3. Copy the SSH keys from the existing active node in the NorthStar cluster and all application nodes.

4. From an existing node in the cluster, delete the knowledge of the CentOS 6.x node from the cluster, then add it back as a new node:

   1. The example below shows identifying the node that needs to be deleted (the one that is down), removing the node from Cassandra, and then observing the output of status commands as the new node is added back into the cluster. UN = up normal, DN = down normal, UJ = up joining. The goal is to replace all nodes and see them return to UN status.

   2. It is important that you resynchronize all your SSH keys once you have rebuilt each node, which includes updating the SSH key on your JunosVM.

   3. After the SSH keys are updated on each JunosVM, back up any changes made to the JunosVM by using the net_setup.py script and selecting Option D > Option 1.

   4. From the net_setup.py main menu, select HA Setup (E).

      Select Add a new node to existing cluster (J), using the existing node data in the script, and allow HA deployment to complete.

   5. Monitor failover to ensure that it completes properly:

      1. Check the output of the supervisorctl status command on the current active node to ensure all processes come up.

      2. Check the cluster status using the following command:

      3. On the node with the VIP (the active node), test failover using the following command:

      4. On the restored node promoting to VIP, use the following command to observe the failover process:

      5. Test the failover process between the three nodes. Optionally, you can add host priority using the net_setup.py script option E (HA Settings).

      6. Run the following command to determine which nodes are currently standby nodes. They should be the two with the higher priority numbers:

      7. Check the NorthStar web UI again for each node while it is the active node, to make sure the data is synchronized properly between the three nodes.

      8. At this point, you should have a fully-functioning NorthStar 5.1.0 three-node cluster running on the CentOS 7.7 operating system.

Upgrade the Operating System on Your Collector Nodes

Collector nodes operate independently, but are tied to the application VIP. They can be deleted or installed back in independently. Proceed one node at a time with reinstallation.

All three collectors are currently running CentOS 6.10 with NorthStar 5.1.0 (NorthStar-Bundle-5.1.0-20191210_220522_bb37a329b_64.x86_64.rpm).

If you have not already done so, back up the NorthStar files and directories listed previously, and save them to an external or network drive.

1. Install the CentOS 7.7 operating system minimal installation on any one of the collector nodes.

2. Install the following recommended packages: net-tools, bridge-utils, wget, ntp, telnet, ksh, java-1.8.0-openjdk-headless.

3. Bring the system back online with the same IP address. Download the NorthStar 5.1.0 package and install it.

4. Run the collector install script.

5. Repeat this process on the remaining collector nodes, one at a time.

Special Notes for Nested JunosVM Nodes

The following additional procedure applies to migrating a nested JunosVM setup:

1. Copy the configuration here: /opt/northstar/data/junosvm/junosvm.conf.

2. Use the net_setup.py script to assign the JunosVM IP address back to the JunosVM.

3. Copy your backup of junosvm.conf into /opt/northstar/data/junosvm/junosvm.conf.

4. Restart the JunosVM:

5. Observe the JunosVM boot process using this command:

Upgrade all Nodes to NorthStar 6.0.0 or Later

Now that your network and configuration are upgraded to CentOS 7.7, you can proceed with upgrading NorthStar to 6.0.0 or later.

• Analytics Node Upgrade to NorthStar 6.0.0 or Later
• NorthStar Application Node Upgrade to NorthStar 6.0.0 or Later
• Collector Node Upgrade to NorthStar 6.0.0 or Later