Upgrading Software on a Virtual Chassis Fabric Using Nonstop Software Upgrade
Nonstop software upgrade (NSSU) enables you to upgrade the software running on all member switches in a Virtual Chassis Fabric (VCF) with minimal network traffic disruption during the upgrade.
For an overview of the benefits of NSSU and how NSSU works, see Understanding Nonstop Software Upgrade on a Virtual Chassis Fabric.
Preparing the Switch for the New Software Installation
Before you begin software installation using NSSU:
Ensure that the VCF is configured correctly to support NSSU:
Verify that you connected the VCF members in a spine-and-leaf topology with each leaf device connected to all of the configured spine devices. A spine-and-leaf topology prevents the Virtual Chassis from splitting during an NSSU.
Note:NSSU is supported in a QFX5100 VCF with up to 20 members.
NSSU is not supported in a QFX5110 VCF.
Ensure the VCF is preprovisioned and you explicitly assigned the Routing Engine role or linecard role to the member switches acting in each of those roles.
Make sure the preprovisioned configuration has only two members in the Routing Engine role. The NSSU process checks the member configuration, displays a warning message if it detects that you configured more than two switches in the Routing Engine role, and stops the upgrade.
In a two-member Virtual Chassis, configure
no-split-detectionso that the VCF does not split when an NSSU upgrades a member.You can optionally configure NSSU upgrade groups with multiple linecard role members in each upgrade group.NSSU upgrades the members in each upgrade group simultaneously, which can shorten the NSSU process, and upgrades the upgrade groups in the order in which you configured them.
See Configuring Line-Card Upgrade Groups for Nonstop Software Upgrade for details.
Check that members are running the same version of the software:
user@switch>
show versionIf the Virtual Chassis Fabric members are not running the same version of the software, use the
request system software addCLI command to upgrade the software on the inconsistent members. If some member switches do not run the same software image, specify different package names using the set option:request system software add set [package-namepackage-name ...] reboot.For example:
user@switch> request system software add force-host set [jinstall-qfx5100.tgz jinstall-qfx3500.tgz jinstall-ex-4300.tgz] reboot
Verify that you enabled nonstop active routing (NSR) and graceful Routing Engine switchover (GRES).You only need to check the NSR state—if NSR is enabled, then GRES is also enabled:
user@switch> show task replication Stateful Replication: Enabled RE mode: Master Protocol Synchronization Status OSPF Complete BGP Complete PIM CompleteIf nonstop active routing is not enabled (
Stateful ReplicationisDisabled), see Example: Configuring Nonstop Active Routing on Switches for information on how to enable it.Verify that you enabled nonstop bridging (NSB), which ensures that all NSB-supported Layer 2 protocols operate seamlessly during the Routing Engine switchover that happens during NSSU.
(Optional) Back up the system software—Junos OS, the active configuration, and log files—on each member to an external storage device with the
request system snapshotcommand.
Upgrading the Software Using NSSU
This procedure describes how to upgrade the software running on all VCF members using NSSU. When the upgrade completes, all members are running the new version of the software. Because a graceful Routing Engine switchover occurs during the upgrade, the original VCF backup is the new primary at the end of the NSSU process.
During NSSU, the primary copies the new software image to all the members in the VCF and reboots them in turn. If copying the new software to a member fails or rebooting a member fails, NSSU terminates the upgrade process and logs the error. In this case, you must manually perform recovery measures for members left in an incompatible state, to restore all members to running the same version of the software. Starting in Junos OS Release 14.1X53-D40, NSSU automatically invokes recovery measures after either of these failures, as follows:
if NSSU terminates due to a copy error, the new image is removed from any members to which it was already copied.
If any member fails to reboot, NSSU automatically initiates a clean VCF restart by bringing down and rebooting the entire VCF. All members come up running the new software at the same time. This action cleanly recovers stable VCF operation more quickly than having an unstable VCF running different versions of the software trying to converge.
Junos OS software images with enhanced automation are only supported on a non-mixed VCF with QFX5100 switches. Also, performing an NSSU from a standard Junos OS software image to a Junos OS software image with enhanced automation, or from a Junos OS software image with enhanced automation to a standard Junos OS software image is not supported.
To upgrade all members of the VCF using NSSU:
Download the software package. If you are upgrading the software running on a mixed mode VCF, download the software packages for each switch type.
Copy the software package or packages to the VCF primary switch. We recommend that you copy the file to the
/var/tmpdirectory on the primary.Log in to the VCF using the console connection of the primary or the virtual management Ethernet (VME) interface. Without the console connection, you will not be able to view any CLI output during an NSSU reboot. The console connection enables you to view CLI output during an NSSU reboot and monitor the progress of the primary switch reboot.
Start the NSSU operation:
To perform NSSU on a VCF composed all of one type of switch, enter:
user@switch> request system software nonstop-upgrade force-host /var/tmp/package-name.tgz
where
package-name.tgzis, for example,jinstall-qfx5100.tgz.To perform NSSU on a mixed-mode QFX5100 VCF (QFX5100 switches with some combination of QFX3500,QFX3600, or EX4300 switches), enter:
user@switch> request system software nonstop-upgrade force-host set [package-name.tgz package-name.tgz package-name.tgz]
where
[package-name.tgz package-name.tgz package-name.tgz]is, for example,[jinstall-qfx5100.tgz jinstall-qfx3500.tgz jinsall-ex-4300.tgz].
As the upgrade proceeds, the VCF displays status messages similar to the following sample output for an NSSU operation on a QFX5100 VCF with four QFX5100 member switches:
user@switch> request system software nonstop-upgrade force-host /var/tmp/jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.tgz no-validate Chassis ISSU Check Done [Sep 24 11:53:16]:ISSU: Validating Image [Sep 24 11:53:16]:ISSU: Preparing Backup RE Installing image on other FPC's along with the backup [Sep 24 11:53:37]: Retrieving software images. This process can take several minutes. Please be patient.. Retrieving version and model information from /var/tmp/jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.tgz Starting with package /var/tmp/jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.tgz Download done for package /var/tmp/jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.tgz Pushing bundle to fpc1 Pushing bundle to fpc4 Pushing bundle to fpc5 fpc1 WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. Saving contents of boot area prior to installation WARNING: This package will load JUNOS 14.1-20160918_x141X53_vjqfd.0 software. WARNING: It will save JUNOS configuration files, and SSH keys WARNING: (if configured), but erase all other files and information WARNING: stored on this machine. It will attempt to preserve dumps WARNING: and log files, but this can not be guaranteed. This is the WARNING: pre-installation stage and all the software is loaded when WARNING: you reboot the system. POST-INSTALL... Saving the config files ... NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install Copying jpuppet pkg to /var/sw/pkg Copying chef pkg to /var/sw/pkg Pushing installation package to host... Upgrade Host OS to 14.1-20160918_x141X53_vjqfd.0... ./jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.img.gz: OK /boot/boot Extracting image... Host upgrade staging completed. Need reboot to complete upgrade installation. WARNING: Changing next boot to SSD0 WARNING: Changing boot device enable from 0x3F to 0x3F Install jinstall-vjunos completed WARNING: A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the WARNING: 'request system reboot' command when software installation is WARNING: complete. To abort the installation, do not reboot your system, WARNING: instead use the 'request system software delete jinstall' WARNING: command as soon as this operation completes. Saving state for rollback ... fpc4 WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. Saving contents of boot area prior to installation WARNING: This package will load JUNOS 14.1-20160918_x141X53_vjqfd.0 software. WARNING: It will save JUNOS configuration files, and SSH keys WARNING: (if configured), but erase all other files and information WARNING: stored on this machine. It will attempt to preserve dumps WARNING: and log files, but this can not be guaranteed. This is the WARNING: pre-installation stage and all the software is loaded when WARNING: you reboot the system. POST-INSTALL... Saving the config files ... NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install Copying jpuppet pkg to /var/sw/pkg Copying chef pkg to /var/sw/pkg Pushing installation package to host... Upgrade Host OS to 14.1-20160918_x141X53_vjqfd.0... ./jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.img.gz: OK /boot/boot Extracting image... Host upgrade staging completed. Need reboot to complete upgrade installation. WARNING: Changing next boot to SSD0 WARNING: Changing boot device enable from 0x3F to 0x3F Install jinstall-vjunos completed WARNING: A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the WARNING: 'request system reboot' command when software installation is WARNING: complete. To abort the installation, do not reboot your system, WARNING: instead use the 'request system software delete jinstall' WARNING: command as soon as this operation completes. Saving state for rollback ... fpc5 WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. Saving contents of boot area prior to installation WARNING: This package will load JUNOS 14.1-20160918_x141X53_vjqfd.0 software. WARNING: It will save JUNOS configuration files, and SSH keys WARNING: (if configured), but erase all other files and information WARNING: stored on this machine. It will attempt to preserve dumps WARNING: and log files, but this can not be guaranteed. This is the WARNING: pre-installation stage and all the software is loaded when WARNING: you reboot the system. POST-INSTALL... Saving the config files ... NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install Copying jpuppet pkg to /var/sw/pkg Copying chef pkg to /var/sw/pkg Pushing installation package to host... Upgrade Host OS to 14.1-20160918_x141X53_vjqfd.0... ./jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.img.gz: OK /boot/boot Extracting image... Host upgrade staging completed. Need reboot to complete upgrade installation. WARNING: Changing next boot to SSD0 WARNING: Changing boot device enable from 0x18 to 0x18 Install jinstall-vjunos completed WARNING: A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the WARNING: 'request system reboot' command when software installation is WARNING: complete. To abort the installation, do not reboot your system, WARNING: instead use the 'request system software delete jinstall' WARNING: command as soon as this operation completes. Saving state for rollback ... [Sep 24 12:05:28]: Backup upgrade done [Sep 24 12:05:28]: Rebooting Backup RE Rebooting fpc1 [Sep 24 12:05:28]:ISSU: Backup RE Prepare Done [Sep 24 12:05:28]: Waiting for Backup RE reboot [Sep 24 12:16:44]: GRES operational [Sep 24 12:16:44]: "Initiating Chassis In-Service-Upgrade" Chassis ISSU Started [Sep 24 12:16:50]:ISSU: Preparing Daemons . . [Sep 24 12:17:50]:ISSU: Daemons Ready for ISSU [Sep 24 12:17:54]:ISSU: Starting Upgrade for FRUs . . . . . . [Sep 24 12:21:09]: FPC 4 is undergoing a software upgrade . . . . . . . . . . . . . . . . [Sep 24 12:30:16]: FPC 4 has come up after a software upgrade . . . . . . . [Sep 24 12:34:42]: FPC 5 is undergoing a software upgrade . . . . . . . . . . . . . . . . [Sep 24 12:42:10]: FPC 5 has come up after a software upgrade . . . . . [Sep 24 12:45:42]:ISSU: Preparing for Switchover [Sep 24 12:45:47]:ISSU: Ready for Switchover [Sep 24 12:45:48]: Checking In-Service-Upgrade status Item Status Reason FPC 0 Online FPC 1 Online (ISSU) FPC 4 Online (ISSU) FPC 5 Online (ISSU) [Sep 24 12:45:48]: Going to install image on master WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. WARNING: The software that is being installed has limited support. WARNING: Run 'file show /etc/notices/unsupported.txt' for details. Saving contents of boot area prior to installation WARNING: This package will load JUNOS 14.1-20160918_x141X53_vjqfd.0 software. WARNING: It will save JUNOS configuration files, and SSH keys WARNING: (if configured), but erase all other files and information WARNING: stored on this machine. It will attempt to preserve dumps WARNING: and log files, but this can not be guaranteed. This is the WARNING: pre-installation stage and all the software is loaded when WARNING: you reboot the system. POST-INSTALL... Saving the config files ... NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install Copying jpuppet pkg to /var/sw/pkg Copying chef pkg to /var/sw/pkg Pushing installation package to host... Upgrade Host OS to 14.1-20160918_x141X53_vjqfd.0... ./jinstall-qfx-5-flex-14.1-20160918_x141X53_vjqfd.0-domestic.img.gz: OK /boot/boot Extracting image... Host upgrade staging completed. Need reboot to complete upgrade installation. WARNING: Changing next boot to SSD0 WARNING: Changing boot device enable from 0x3F to 0x3F Install jinstall-vjunos completed WARNING: A REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY. Use the WARNING: 'request system reboot' command when software installation is WARNING: complete. To abort the installation, do not reboot your system, WARNING: instead use the 'request system software delete jinstall' WARNING: command as soon as this operation completes. Saving state for rollback ... failover links [Sep 24 12:50:22]: Rebooting Old master Chassis NSSU Completed [Sep 24 12:50:22]:ISSU: IDLE
Log in after the reboot of the original primary switch completes. To verify that the software on all Routing Engines in the VCF members has been upgraded, enter the following command:
user@switch>
show version