Upgrading the Host OS on the Cluster Server
Prerequisites
Ensure all Kubernetes pods are running normally. Issue the kubectl get pods command from the control machine or the master node and verify that all Kubernetes pods are READY and have a STATUS of Running.
The server that you are upgrading has access to the Internet to download the new host OS release.
You have the common secret that you specified when you ran the proNX Optical Director installation script for the current installation.
Note If you have forgotten the common secret, you can retrieve it by issuing the following commands from the master node:
Type kubectl get secret common.secret -o yaml to retrieve the encoded password.
Type echo '<encoded-password>' | base64 --decode to see the encoded password in clear text.
Use this procedure to upgrade the host operating system on each server in the cluster. Once you start, you must upgrade the host operating system on all three servers. The typical procedure is to upgrade the regular nodes first and the master node last.
Although this procedure does not cause any application downtime, it is recommended that you not use the proNX Optical Director to make any configuration changes while this upgrade is taking place.
This procedure shows an upgrade from Atomic Host 7.1805 to 7.1808 as an example. The same steps apply to upgrading from and to other releases.
- Log in to one of the servers. For example:
# ssh 10.228.4.51
- Show the Atomic Host version you are currently running.
For example:
# atomic host status State: idle Deployments: ● ostree://centos-atomic-host:centos-atomic-host/7/x86_64/standard Version: 7.1805 (2018-06-11 11:03:09) Commit: 6275caab694c56515c57f5e5ef923d999e16e8fb1241992d155ff03cfb528261 GPGSignature: Valid signature by 64E3E7558572B59A319452AAF17E745691BA8335 Unlocked: hotfix ostree://centos-atomic-host:centos-atomic-host/7/x86_64/standard Version: 7.1805 (2018-06-11 11:03:09) Commit: 6275caab694c56515c57f5e5ef923d999e16e8fb1241992d155ff03cfb528261 GPGSignature: Valid signature by 64E3E7558572B59A319452AAF17E745691BA8335 [root@server1 ~]# - Upgrade the host OS. For example:
# atomic host deploy 7.1808 Resolving version '7.1808' 2 metadata, 0 content objects fetched; 18 KiB transferred in 1 seconds 768 metadata, 4542 content objects fetched; 334286 KiB transferred in 161 seconds Copying /etc changes: 26 modified, 13 removed, 211 added Transaction complete; bootconfig swap: yes deployment count change: 0 Upgraded: NetworkManager 1:1.10.2-14.el7_5 -> 1:1.10.2-16.el7_5 NetworkManager-libnm 1:1.10.2-14.el7_5 -> 1:1.10.2-16.el7_5 atomic 1:1.22.1-3.git2fd0860.el7 -> 1:1.22.1-22.git5a342e3.el7 atomic-registries 1:1.22.1-3.git2fd0860.el7 -> 1:1.22.1-22.git5a342e3.el7 audit-libs 2.8.1-3.el7 -> 2.8.1-3.el7_5.1 audit-libs-python 2.8.1-3.el7 -> 2.8.1-3.el7_5.1 <trimmed> Run "systemctl reboot" to start a reboot
- Reboot the server.
# systemctl reboot
- After the server has rebooted, log back in to the server
to verify that the server is running the new version of the OS. For
example:
# atomic host status State: idle; auto updates disabled Deployments: ● ostree://centos-atomic-host:centos-atomic-host/7/x86_64/standard Version: 7.1808 (2018-09-12 16:35:46) Commit: b62513420243def54db7d6c36a79d8f0eda94c5ac410c1f64ee6e0b44e5dca99 GPGSignature: Valid signature by 64E3E7558572B59A319452AAF17E745691BA8335 ostree://centos-atomic-host:centos-atomic-host/7/x86_64/standard Version: 7.1805 (2018-06-11 11:03:09) Commit: 6275caab694c56515c57f5e5ef923d999e16e8fb1241992d155ff03cfb528261 GPGSignature: Valid signature by 64E3E7558572B59A319452AAF17E745691BA8335 Unlocked: hotfix - From the control machine, verify that the Kubernetes pods
are READY and have a STATUS of Running and the databases have been
rebalanced. You should perform this step from the control machine
because the kubectl utility has not yet been reinstalled on the node
with the upgraded host operating system.
Check the application database pods. For example:
# kubectl get pods | grep joc-db NAME READY STATUS RESTARTS AGE joc-db-blue-99052229 1/1 Running 1 6h joc-db-green-1848954212-hq0vn 1/1 Running 1 6h joc-db-red-29840614 1/1 Running 1 6h
Check to make sure each database has been rebalanced. Rebalancing can take an hour or longer. If the rebalance is complete, you will see a status of notRunning. For example:
# kubectl exec -it joc-db-green-1848954212-hq0vn -- couchbase-cli rebalance-status -c localhost -u Administrator -p <common-secret> { "status": "notRunning", "msg": "Rebalance is not running", "details": {} }where <common-secret> is the common secret you specified when you installed the current version of the proNX Optical Director software.
Do not proceed until you have confirmed that the Kubernetes pods are READY and have a STATUS of Running and the databases have been rebalanced.
- Repeat this procedure on the next server until all the servers are upgraded.