Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Upgrading a Chassis Cluster Using In-Service Software Upgrade

This topic explains that In‑Service Software Upgrade (ISSU) enables a software upgrade from one Junos OS version to a later Junos OS version while ensuring minimal downtime.

Use Feature Explorer to confirm platform and release support for specific features.

Review the Platform-Specific In-Service Software Upgrade Behavior section for notes related to your platform.

See the Additional Platform Information section for more information.

Understanding ISSU for a Chassis Cluster

In-service software upgrade (ISSU) enables an upgrade from one Junos OS version to a later Junos OS version with little or no downtime. ISSU is performed only when devices are operating in chassis cluster mode .

The chassis cluster ISSU feature allows both devices in a cluster to be upgraded from supported Junos OS versions with minimal traffic disruption iand no interruptionto services, by coordinating the upgrade process across cluster nodes.

ISSU provides the following benefits:

  • Eliminates network downtime during software image upgrades

  • Reduces operating costs, while delivering higher service levels

  • Allows fast implementation of new features

ISSU has the following limitations:

  • ISSU is available only for Junos OS Release 10.4R4 or later.

  • ISSU does not support software downgrades.

  • If you upgrade from a Junos OS version that supports only IPv4 to a version that supports both IPv4 and IPv6, IPv4 traffic continues to function normally throughout the upgrade process. If you upgrade from a Junos OS version that supports both IPv4 and IPv6 to another version that also supports both protocols, both IPv4 and IPv6 traffic continue to function during the upgrade. Junos OS supported platforms support flow-based processing for IPv6 traffic.

  • During an ISSU, you cannot bring any PICs online. You cannot perform operations such as commit, restart, or halt.

  • During an ISSU, operations like fabric monitoring, control link recovery, and RGX preempt are suspended.

  • During an ISSU, you cannot commit any configurations.

For details about ISSU support status, see knowledge base article KB17946.

The following sequence describes the behavior of an ISSU for devices operating in a chassis cluster. The sequence applies when RG-0 is hosted on node 0, which isthe primary node. You must initiate the ISSU from the RG-0 primary node. If you attempt to start the ISSU from node 1 (the RG-0 secondary node), the system displays an error message and the upgrade does not proceed.

  1. Initial Redundancy Group Failover

    At the beginning of a chassis cluster ISSU, the system automatically fails over all RG-1 and higher (RG-1+) redundancy groups that are not already primary on the node from which the ISSU is initiated. This action ensures that all redundancy groups become active on the the RG-0 primary node before the software upgrade begins.

    The automatic failover of all RG-1+ redundancy groups is performed by the system. If you are running Junos OS release 18.1R1 or earlier, you must manually verify—before starting the ISSU—that all RG-1+ redundancy groups are active on the RG-0 primary node.

    After all RG-1+ redundancy groups fail over, the system sets the manual failover bit, preventing redundant group movement during the upgrade. The system changes the primary node priority of all RG‑1+ redundancy groups to 255, regardless of whether they failed over to the RG‑0 primary node during this step.

  2. During an ISSU, the primary node (node 0) validates the device configuration to ensure that it can be successfully committed using the new Junos OS version. As part of this validation process, the system performs the following checks on both nodes:

    • Disk space availability on the /var file system

    • Unsupported configuration statements

    • Unsupported Physical Interface Cards (PICs)

    If the available disk space on the /var file system of either Routing Engine is insufficient, the ISSU process fails and returns an error message, and the upgrade does not proceed.

    Unsupported PICs do not prevent the ISSU from continuing. However, the software generates a warning indicating that these PICs will restart during the upgrade.

    Similarly, the presence of an unsupported protocol configuration does not block the ISSU. In such cases, the software issues a warning that packet loss might occur for the affected protocol during the upgrade process.

  3. When the validation succeeds, the kernel state synchronization daemon (ksyncd) synchronizes the kernel on the secondary node (node 1) with the node 0.

  4. Node 1 is upgraded with the new software image. Before being upgraded, the node 1 gets the configuration file from node 0 and validates the configuration to ensure that it can be committed using the new software version. After being upgraded, it is resynchronized with node 0.

  5. The chassis cluster process (chassisd) on the node 0 prepares other software processes for the lSSU. When all the processes are ready, chassisd sends a message to the PICs installed in the device.

  6. The Packet Forwarding Engine on each Flexible PIC Concentrator (FPC) saves its state and downloads the new software image from node 1. Next, each Packet Forwarding Engine sends a message (unified-ISSU ready) to the chassisd.

  7. After receiving the message (unified-ISSU ready) from a Packet Forwarding Engine, the chassisd sends a reboot message to the FPC on which the Packet Forwarding Engine resides. The FPC reboots with the new software image. After the FPC is rebooted, the Packet Forwarding Engine restores the FPC state and a high-speed internal link is established with node 1 running the new software. The chassisd is also reestablished with node 0.

  8. After all Packet Forwarding Engines have sent a ready message using the chassisd on node 0, other software processes are prepared for a node switchover. The system is ready for a switchover at this point.

  9. Node switchover occurs and node 1 becomes the new primary node (hitherto secondary node 1).

  10. The new secondary node (hitherto primary node 0) is now upgraded to the new software image.

When both nodes are successfully upgraded, the ISSU is complete.

When upgrading a chassis cluster from a Junos OS version that does not support encryption to a version that supports encryption, perform the upgrade one node at a time:

  1. Upgrade the first node to the new Junos OS version.

    If encryption is not configured and enabled, cluster communication between the two nodes remains intact even though they are running different software versions, and services continue without interruption.

  2. Upgrade the second node to the same new Junos OS version.

After both nodes are successfully upgraded, you can choose to configure and enable encryption as required.

For downgrades to a Junos OS version that does not support encryption, ensure that encryption is deactivated before initiating the downgrade. Disabling encryption prior to the downgrade prevents communication failures between:

  • a node still running an encryption‑enabled Junos OS version, and
  • a node that has been downgraded to a version without encryption support.

By ensuring that encryption is disabled on both nodes, cluster communication remains unencrypted and operational throughout the downgrade process.

The policies in the Routing Engine and Packet Forwarding Engine must be in sync for the configuration to be committed. When the policy configurations are modified and the policies are out of sync, the system displays an error message.

As a workaround, you must use the request security policies resync command to synchronize the configuration of security policies in the Routing Engine and Packet Forwarding Engine, in case if you notice security policies are out of sync after an upgrade.

ISSU System Requirements

You can use ISSU to upgrade from an ISSU-capable software release to a later release.

To perform an ISSU, your device must be running a Junos OS release that supports ISSU for the specific platform.

See the ../concept/../topic-map/security-chassis-cluster-upgrading-using-in-service-software-upgrade.html#concept_yfw_3lw_t2c section for more information.

For additional details on ISSU support and limitations, see ISSU/ICU Upgrade Limitations on SRX Series Devices.

Following are the limitations when performing an ISSU:

  • The ISSU process is terminated if the Junos OS version specified for installation is earlier than the version currently running on the device.

  • ISSU is terminated if the specified upgrade conflicts with:

    • The current device configuration

    • Supported hardware components

    • Other platform or software dependencies

  • ISSU does not support extension application packages developed using the Junos OS SDK.

  • ISSU does not support version downgrading on all Firewalls.

  • ISSU may failunder heavy CPU load, and it is recommended to ensure adequate system resources before initiating the upgrade.

To downgrade from an ISSU-capable Junos OS release to an earlier release (whether ISSU-capable or not), use the request system software add command.

Unlike an ISSU upgrade, a downgrade may cause network disruptions.

There is also a risk of data loss during the downgrade process.

Plan downgrades carefully and ensure proper backups are taken before proceeding.

We strongly recommend that you perform ISSU under the following conditions:

  • When both the primary and secondary nodes are healthy

  • During system maintenance period

  • During the lowest possible traffic period

  • When the Routing Engine CPU usage is less than 40 percent

In scenarios, where ISSU is not supported or not recommended, but system upgrade downtime still needs to be minimized, the minimal downtime upgrade procedure can be used, see the relevant knowledge base articleKB17947.

Upgrade Both Devices in a Chassis Cluster Using ISSU

Before you begin the ISSU for upgrading both the devices, note the following guidelines:

  • Ensure the following ISSU pre-check requirements are met:

    • All redundancy groups priority is greater than 0

    • All redundancy groups are either primary or secondary in state

    • There exists enough (double the image size) space available in the /var/tmp

    • Usage of CPU is under 80% within 5 seconds period

    If the pre-check requirements are not met, ISSU will terminate at the beginning.

  • Back up the software using the request system snapshot command on each Routing Engine to back up the system software to the device’s hard disk.

  • If you are using Junos OS Release 18.1R1 or earlier, before starting the ISSU, set the failover for all redundancy groups so that they are all active on only one node (primary). See Initiating a Chassis Cluster Manual Redundancy Group Failover.

    If you are running Junos OS Release 18.1 or later, the system automatically fails over all RGs to the RG0 primary.

  • We recommend enabling graceful restart for all routing protocols before initiating an ISSU to ensure minimal traffic disruption.

On all supported Firewalls, the first recommended ISSU from release is Junos OS Release 18.1R1.

The chassis cluster ISSU feature allows both devices in a cluster to be upgraded from supported Junos OS releases with traffic impact comparable to that of redundancy group failovers.

To perform an ISSU from the CLI on Routing Engine2:

  1. Download the software package from the Juniper Networks Support website: https://www.juniper.net/support/downloads/
  2. Copy the package on primary node of the cluster. We recommend that you copy the package to the/var/tmp directory, which is a large file system on the hard disk. Note that the node from where you initiate the ISSU must have the software image.

    user@host>file copy ftp://username:prompt@ftp.hostname.net/filename /var/tmp/filename

  3. Verify the current software version running on both nodes by issuing the show version command on the primary node.
  4. Start the ISSU from the node that is primary for all the redundancy groups by entering the following command:

    Wait for both nodes to complete the upgrade (After which you are logged out of the device).

  5. Wait a few minutes, and then log in to the device again. Verify by using the show version command that both devices in the cluster are running the new Junos OS release.
  6. Verify that all policies, zones, redundancy groups, and other real-time objects (RTOs) return to their correct states.
  7. Make node 0 the primary node again by issuing the request chassis cluster failover node node-number redundancy-group group-number command.

If you want redundancy groups to automatically revert to node 0 as the primary after an in-service software upgrade (ISSU), you must configure the redundancy group priorities so that node 0 has the higher priority and enable the preempt option.

This approach applies to all redundancy groups except redundancy group 0 (RG0). For RG0, failover must be performed manually.

To set the redundancy group priority and enable the preempt option, see Example: Configuring Chassis Cluster Redundancy Groups.

To manually set the failover for a redundancy group, see Initiating a Chassis Cluster Manual Redundancy Group Failover.

During the upgrade, both devices may experience redundancy group failovers; however, traffic is not disrupted. Before initiating the upgrade, each device validates the upgrade package and verifies version compatibility. If the system detects that the new package version is incompatible with the currently installed version, the upgrade is either rejected or you are prompted to take corrective action. In some cases, a specific feature may be incompatible, in such situations, the upgrade software prompts you to either terminate the upgrade or disable the incompatible feature before proceeding.

If you plan to operate the Firewall as a standalone device or remove a node from a chassis cluster, ensure that the ISSU procedure has been fully terminated on both nodes (if an ISSU was initiated).

To start ISSU process on SRX5K devices with Routing Engine3 and on SRX1600, SRX2300, SRX4120, and SRX4300 devices:

  1. Run the following command to start ISSU:

See Also

Roll Back Devices in a Chassis Cluster After an ISSU

If an ISSU fails to complete and only one device in the cluster is upgraded, you can roll back to the previous configuration on the upgraded device alone by issuing one of the following commands on the upgraded device:

  • request chassis cluster in-service-upgrade abort

  • request system software rollback node node-id reboot

  • request system reboot

Enable an Automatic Chassis Cluster Node Failback After an ISSU

If you want redundancy groups to automatically return to node 0 as the primary after an in-service software upgrade (ISSU), you must configure the redundancy group priority so that node 0 has the higher priority and enable the preempt option.

This mechanism applies to all redundancy groups except redundancy group 0. Redundancy group 0 does not support automatic preemption and must be failed over manually.

To set redundancy group priorities and enable preempt option, see Example: Configuring Chassis Cluster Redundancy Groups. To manually initiate aredundancy group failover, see Initiating a Chassis Cluster Manual Redundancy Group Failover.

To complete the upgrade and make node 0 available in the chassis cluster after ISSU, you must manually reboot node 0. Node 0 does not reboot automatically as part of the ISSU process.

Log Error Messages used for Troubleshooting ISSU-Related Problems

The following problems might occur during an ISSU upgrade. You can identify the errors by using the details in the logs. For detailed information about specific system log messages, see System Log Explorer.

Chassisd Process Errors

Problem

Description

Errors related to chassisd.

Solution

Use the error messages to understand the issues related to chassisd.

When ISSU starts, a request is sent to chassisd to check whether there are any problems related to the ISSU from a chassis perspective. If there is a problem, a log message is created.

Common Error Handling for ISSU

Problem

Description

You might encounter some problems in the course of an ISSU. This section provides details on how to handle them.

Solution

Any errors encountered during an ISSU generate log messages, and the ISSU process continues without traffic impact. If a revert to a previous Junos OS version is required, the event is logged or the ISSU process is halted to prevent version mismatches between the nodes in the chassis cluster. Table 1 provides some of the common error conditions and their corresponding workarounds. The example log messages shown Table 1 are taken from the SRX1500 device, but they are also applicable to all supported Firewalls.

Table 1: ISSU-Related Errors and Solutions

Error Conditions

Solutions

Attempt to initiate an ISSU when previous instance of an ISSU is already in progress

The following message is displayed:

warning: ISSU in progress

You can abort the current ISSU process, and initiate the ISSU again using the request chassis cluster in-service-upgrade abort command.

Reboot failure on the secondary node

No service downtime occurs, because the primary node continues to provide required services. Detailed console messages are displayed requesting that you manually clear existing ISSU states and restore the chassis cluster.

error: [Oct  6 12:30:16]: Reboot secondary node failed (error-code: 4.1)

       error: [Oct  6 12:30:16]: ISSU Aborted! Backup node maybe in inconsistent state, Please restore backup node
       [Oct  6 12:30:16]: ISSU aborted. But, both nodes are in ISSU window.
       Please do the following:
       1. Rollback the node with the newer image using rollback command
          Note: use the 'node' option in the rollback command
          otherwise, images on both nodes will be rolled back
       2. Make sure that both nodes (will) have the same image
       3. Ensure the node with older image is primary for all RGs
       4. Abort ISSU on both nodes
       5. Reboot the rolled back node

Secondary node failed to complete the cold synchronization

The primary node times out if the secondary node fails to complete the cold synchronization. Detailed console messages are displayed that you manually clear existing ISSU states and restore the chassis cluster. No service downtime occurs in this scenario.

[Oct  3 14:00:46]: timeout waiting for secondary node node1 to sync(error-code: 6.1)
        Chassis control process started, pid 36707 

       error: [Oct  3 14:00:46]: ISSU Aborted! Backup node has been upgraded, Please restore backup node 
       [Oct  3 14:00:46]: ISSU aborted. But, both nodes are in ISSU window. 
       Please do the following: 
      1. Rollback the node with the newer image using rollback command 
          Note: use the 'node' option in the rollback command 
          otherwise, images on both nodes will be rolled back 
      2. Make sure that both nodes (will) have the same image 
      3. Ensure the node with older image is primary for all RGs 
      4. Abort ISSU on both nodes 
      5. Reboot the rolled back node

Failover of newly upgraded secondary failed

No service downtime occurs, because the primary node continues to provide required services. Detailed console messages are displayed requesting that you manually clear existing ISSU states and restore the chassis cluster.

[Aug 27 15:28:17]: Secondary node0 ready for failover.
[Aug 27 15:28:17]: Failing over all redundancy-groups to node0
ISSU: Preparing for Switchover
error: remote rg1 priority zero, abort failover.
[Aug 27 15:28:17]: failover all RGs to node node0 failed (error-code: 7.1)
error: [Aug 27 15:28:17]: ISSU Aborted!
[Aug 27 15:28:17]: ISSU aborted. But, both nodes are in ISSU window.
Please do the following:
1. Rollback the node with the newer image using rollback command
    Note: use the 'node' option in the rollback command
           otherwise, images on both nodes will be rolled back
2. Make sure that both nodes (will) have the same image
3. Ensure the node with older image is primary for all RGs
4. Abort ISSU on both nodes
5. Reboot the rolled back node
{primary:node1}

Upgrade failure on primary

No service downtime occurs, because the secondary node fails over as primary and continues to provide required services.

Reboot failure on primary node

Before the reboot of the primary node, devices being out of the ISSU setup, no ISSU-related error messages are displayed. The following reboot error message is displayed if any other failure is detected:

Reboot failure on     Before the reboot of primary node, devices will be out of ISSU setup and no primary node error messages will be displayed.
Primary node

ISSU Support-Related Errors

Problem

Description

Installation failure occurs because of unsupported software and unsupported feature configuration.

Solution

Use the following error messages to understand the compatibility-related problems:

Initial Validation Checks Failure

Problem

Description

The initial validation checks fail.

Solution

The validation checks fail if the image is not present or if the image file is corrupt. The following error messages are displayed when initial validation checks fail when the image is not present and the ISSU is aborted:

When Image Is Not Present

When Image File Is Corrupted

If the image file is corrupted, the following output displays:

The primary node validates the device configuration to ensure that it can be committed using the new software version. If anything goes wrong, the ISSU aborts and error messages are displayed.

Installation-Related Errors

Problem

Description

The install image file does not exist or the remote site is inaccessible.

Solution

Use the following error messages to understand the installation-related problems:

ISSU downloads the install image as specified in the ISSU command as an argument. The image file can be a local file or located at a remote site. If the file does not exist or the remote site is inaccessible, an error is reported.

Redundancy Group Failover Errors

Problem

Description

Problem with automatic redundancy group (RG) failure.

Solution

Use the following error messages to understand the problem:

Kernel State Synchronization Errors

Problem

Description

Errors related to ksyncd.

Solution

Use the following error messages to understand the issues related to ksyncd:

ISSU checks whether there are any ksyncd errors on the secondary node (node 1) and displays the error message if there are any problems and aborts the upgrade.

Platform-Specific In-Service Software Upgrade Behavior

Use Feature Explorer to confirm platform and release support for specific features.

Use the following table to review platform-specific behaviors for your platform.

Platform

Difference

SRX Series

  • SRX1500, SRX4100, and SRX4200 Firewalls support to upgrade from Junos OS 17.4 to successive 17.4 releases and cannot upgrade to 17.4 releases from previous Junos OS releases.

  • SRX5400, SRX5600, and SRX5800 Firewalls support to upgrade from Junos OS 17.3 to successive 17.3 releases and cannot upgrade to 17.3 and higher releases from earlier Junos OS releases.

  • SRX1500, SRX1600, SRX2300, SRX4120, SRX4100, SRX4200, SRX4300, and SRX4600, Firewalls do not support the request system snapshot command.

  • SRX1500, SRX4100, and SRX4200 Firewalls that support ISSU allow you to remove the original image file. Include unlink to the user@host> request system software in-service-upgrade image-name-with-full-path unlink command.

Additional Platform Information

Use Feature Explorer to confirm platform and release support for specific features.

Additional Platforms may be supported.

Table 2: ISSU Platform Support

Device

Junos OS Release

SRX5800 and SRX5600

10.4R4 or later

SRX5400

12.1X46-D20 or later

SRX1500

15.1X49-D70 or later

SRX1600 and SRX2300, SRX4120

23.4R1 or later

SRX4100 and SRX4200

15.1X49-D80 or later

SRX4300

24.2R1 or later

SRX4600

17.4R1 or later

Related Documentation