JUNOSe 9.1.x System Basics Configuration Guide > Configuring a Unified In-Service Software Upgrade > Unified ISSU Phases Overview

Unified ISSU Phases Overview

The JUNOSe software includes software modules that operate the following hardware components:

• SRP module
• Line module control plane
• Line module forwarding plane

A unified in-service software upgrade replaces the currently operating software on each of these components with a higher-numbered release. The unified ISSU also upgrades or re-creates the state and configuration data of the configured applications.

Before you begin the in-service software upgrade, you must first prepare the router for the upgrade. See Before You Begin a Unified In-Service Software Upgrade for more information.

The in-service software upgrade takes place in three phases:

1. Initialization Phase—When you issue the issu initialize command, unified ISSU verifies whether all prerequisites for the upgrade have been met. The router is prepared for the upgrade. The configuration that has been mirrored to the standby SRP module is upgraded according to the upgrade release. This phase can last from a few minutes up to 40 minutes depending on the number of software releases across which the router is being upgraded.
2. Upgrade Phase—When you issue the issu start command, unified ISSU again verifies whether all prerequisites for the upgrade have been met. During this phase the line module control plane and forwarding plane are upgraded and all three hardware components are resynchronized.
3. Service Restoration Phase—This phase automatically begins without user intervention when the upgrade phase has completed. During this final phase, the router is returned to a normal, runtime state.

The following sections describe these phases in more detail.

Unified ISSU Initialization Phase Overview

When you issue the issu initialize command, unified ISSU first verifies whether all requirements for the upgrade are met. The verification process examines the running release, the SRP modules, the line modules, line module redundancy, and the router configuration.

The issu initialize command does not interrupt or disrupt any of the runtime operations of the router. The command has no effect on changes of authorization, forwarding, or subscribers (except that the rate of logins might be affected). You cannot manually change the file system redundancy mode from high availability to file synchronization until the unified in-service software upgrade is completed.

NOTE: We recommend that you make no configuration changes after you have issued the issu initialization command. As a best practice, ensure that your configuration is complete before you start the software upgrade.

During the initialization phase, you can halt the in-service software upgrade at any time and revert either to a normal SRP module switchover or to the previous state of the router. To stop the unified ISSU process, you must issue the issu stop command. If instead you simply exit the CLI session, the unified ISSU initialization phase continues.

The action taken when a requirement is not met depends on the requirement. For some failed verifications, the CLI warns you of the issue and prompts you to proceed or quit the upgrade process. More serious failures cause the CLI to exit the command and provide a message describing the issue.

NOTE: We recommend that you issue the show issu command before beginning the in-service software upgrade. The output of the command lists any necessary conditions that are not currently met on the router. You can therefore correct these failures before entering into the upgrade. You can issue the show issu command at any time. NOTE: On E120 and E320 routers, you can hot swap an IOA during the initialization phase without affecting the in-service software upgrade. However, we strongly recommend that you perform any necessary hot swaps before you issue the issu initialize command.

If the standby SRP module reloads during the initialization phase, unified ISSU is halted. You must begin again by issuing the issu initialize command.

Application Data Upgrade on the Standby SRP Module

Synchronized modules can become unsynchronized because you can change the router configuration at any time until you issue the issu start command. When the verification process is completed, unified ISSU starts up the stateful SRP switchover process to maintain synchronization between the active SRP module and the standby SRP module during the SRP module upgrade phase.

NOTE: An SRP switchover from the active SRP module to the standby SRP module at this point in the in-service software upgrade causes a cold restart of the router because the SRP modules are running two different releases. The current release is on the active SRP module and the upgrade release is on the standby SRP module.

The application and configuration data that has been mirrored to the standby SRP module is upgraded as required by the new software release. The CLI displays the progress of the SRP module upgrade.

While data on the standby SRP module is upgraded, any new changes that are mirrored from the primary SRP module to the standby SRP module are also upgraded to the version required by the armed release.

NOTE: This process consumes significant CPU resources on the redundant SRP module and can take a considerable amount of time to complete. Performance of the active SRP module might be affected during the SRP module upgrade.

When the upgrade release has been synchronized to the standby SRP module, stateful SRP switchover is disabled until the in-service software upgrade is completed.

If you configure an application that does not support unified ISSU during the initialization phase, the initialization phase completes, but the issu start command subsequently fails.

Line Module Arming

When the upgrade of the application data on the standby SRP upgrade is completed, unified ISSU temporarily arms the line modules with the upgrade release in a backup region of the memory.

NOTE: If a line module reloads at this point, it it is held down for the duration of the upgrade and then undergoes a cold boot to the running release; the line module has no effect on the unified ISSU process.

SNMP Traps

When you issue the issu initialization command, the SNMP agent generates a juniSystemIssuStateChange trap with juniSystemIssuState set to initializing. When the unified ISSU initialization is completed, the SNMP agent generates a juniSystemIssuStateChange trap with juniSystemIssuState set to initialized.

Unified ISSU Upgrade Phase Overview

During the upgrade phase, the CLI supports only a reduced set of administrative commands. You cannot interrupt the upgrade phase. The upgrade phase cannot commence if any CLI commands outside of this set are executing when you issue the issu start command.

NOTE: Although you can use any CLI session to issue the issu start command, we recommend that you issue the command from a session to the management console port. When the standby SRP module switchover takes place, all management network connections through the Ethernet management port are dropped, and you can access the router only through a console port until the service restoration phase is completed.

When you issue the issu start command, unified ISSU performs the following operations:

1. Verifies that the unified ISSU requirements on the router are still met.
2. Verifies that the active and standby SRP modules are synchronized. If they are not synchronized, forces a synchronization to ensure that all configuration and file system changes are propagated to the standby SRP module before proceeding with the upgrade.
3. Copies the NVS configuration from a backup memory area to the flash card on the standby SRP module. During the initialization phase, this configuration data was mirrored from NVS on the active SRP module and upgraded as required by the armed release.
4. Upgrades the control plane on each line module at the same time.
5. Switches control from the primary SRP module (running the current release) to the standby SRP module (running the armed upgrade release).
6. Upgrades the forwarding plane on each line module at the same time. The fabric is switched and upgraded.

You can view the status of the router and the progress of the upgrade at any time by issuing the show issu command from another terminal session to the router.

NOTE: While a unified ISSU operation is in progress, do not remove the SRP modules or attempt to reset them. Removing the SRP modules anytime during unified ISSU has an adverse impact. After the unified ISSU operation is completed, issue the show version command. The output should indicate the following: The formerly active SRP module has rebooted and come up as the new standby SRP module. The newly active SRP module indicates that the formerly active SRP has rebooted and has come up as standby SRP module You can then remove an SRP module after issuing the halt command.

Exceptions During the Upgrade Phase

Table 47 describes the behavior of the router during the upgrade phase when certain exceptional events take place outside the context of the in-service software upgrade.

Table 47: Router Response to Undesirable Events During the Upgrade Phase

Event	Router Action
The router reloads.	The unified ISSU operation halts. The router undergoes a cold restart. The router boots with the armed upgrade release. The line modules reboot.
The primary SRP module switches over to the standby SRP module.	The unified ISSU operation halts. The router undergoes a cold restart. The router boots with the armed upgrade release. The line modules reboot.
The standby SRP module reloads.	The unified ISSU operation halts. The router undergoes a cold restart. The router boots with the armed upgrade release. The line modules reboot.
A line module reloads.	The line module is held down and prevented from rebooting until the service restoration phase is completed. The line module then undergoes a cold reboot to the running (pre-upgrade) release.
An IOA is hotswapped.	Hot swapping is disabled during the upgrade phase. The line module undergoes a cold reboot and hot swapping is reenabled when the service restoration phase is completed,
An application that does not support unified ISSU is configured.	This event can take place only before the issu start command is issued, because that command disables CLI configuration commands. When you issue the issu start command, after configuring such an application, the command exits and generates an error message.

Verification of Requirements

Because some time may have passed since unified ISSU verified the requirements for the upgrade during the initialization phase, unified ISSU reverifies all the same conditions.

Unified ISSU also verifies that no CLI configuration sessions are open, that no scripts or macros are running, and that any SNMP requests or CLI commands in progress complete within 5 seconds.

If any of the required conditions are not met, the CLI either exits the command with an error message or provides an informative message and prompts you to proceed or halt.

When all the conditions are met, the CLI prompts you to proceed. If you continue, then you can subsequently halt the upgrade only by reloading the router. If you exit the command, the router remains in the unified ISSU initialized state.

Upgrade Setup

At this stage all the preconditions have been met. The unified ISSU process shuts down all management interfaces to the router in order to prevent changes in the configuration.

Final preparation for the upgrade phase includes the following actions:

• SNMP—The SNMP agent generates a juniSystemIssuStateChange trap with juniSystemIssuState set to upgrading to indicate that the final phase of the operation has begun. When the trap is issued with this state value, the SNMP agent stops accepting any new SNMP gets or sets and does not issue any further traps.
• CLI—Most CLI commands are disabled. Only reload, show issu, and show version commands are available to you until the service restoration phase completes. These commands are available on the console and are not available to Telnet sessions.
• External events—Externally created events from sources other than the CLI are ignored. These events typically come from user connections, RADIUS servers, SRC software and SDX software, and SNMP, and include login requests, COA requests, multicast join requests, packet mirroring requests, and so on. Logout requests are cached and processed at a later stage.
• Routing protocols—The unified ISSU process prompts you to consider raising the link costs for each routing protocol that is configured on the router. Raising the link cost for routes through the upgrading router enables neighbors to recompute better routes to those destinations. If you choose to raise the link cost, the higher costs can take some time to propagate through the network. Because the router is unable to determine when this has completed, it waits for 2 minutes before proceeding to the next step in the upgrade.

The reason for raising the link cost is that once the upgrade of the line module control plane begins, routing protocol updates cannot be installed in the line modules until that upgrade completes. That period can be in the range 2–15 minutes. During the control plane upgrade, the routing protocols can still accept new routes and communicate with their neighbors but cannot install the routes.

• Routing protocols—The unified ISSU process prompts you to consider raising the link costs for each routing protocol that is configured on the router.

Once the upgrade of the line module control plane begins, routing protocol updates cannot be installed in the line modules until that upgrade completes. That period can be in the range 2–15 minutes. During the control plane upgrade, the routing protocols can still accept new routes and communicate with their neighbors but cannot install the routes.

Raising the link cost for routes through the upgrading router enables neighbors to recompute better routes to those destinations that do not pass through the upgrading router. If you choose to raise the link cost, the higher costs can take some time to propagate through the network. Because the router is unable to determine when this has completed, it waits for 2 minutes before proceeding to the next step in the upgrade.

• Unsupported line modules—Any unsupported line modules that are present are held down after the start of this phase when you can no longer gracefully exit from the unified ISSU process. The modules are held down for the duration of the in-service software upgrade and are then undergo a cold boot to the original running release.
• IGMP requests—The router cannot handle IGMP requests for channel changing for IPTV implementations.

Line Module Control Plane Upgrade

At this point, the upgrade release is preserved on each line module in some backup region. When signaled by the active SRP module, all supported line modules simultaneously reload and restart with the new release. Forwarding through the forwarding subsystem on the line modules—through the fabric of the system—is not be affected by the reload.

The line modules then simultaneously recover any application data preserved in memory on the line module and upgrade that data into a format that is understood by the applications running on the new release. This operation can take in the range of 1–10 minutes depending on the size of the data and the upgrade path of the data. Each line module restores its operational state, running the new release with all data upgraded to a version acceptable to the new software.

If the upgrade process fails for any line module, that module undergoes a cold restart, but none of the other line modules is affected.

SRP Module Switchover

At this stage the primary SRP module is running the current release, the redundant SRP module is running the armed release, and the control plane on each supported line module is running the armed release.

When the primary SRP module has verified that all line modules are up, the redundant SRP module takes over control of the router by becoming the active SRP module. The primary, and formerly active, SRP module reboots to the armed release and serves as the standby SRP module.

All applications on the newly active SRP module are restarted. Each application reconstructs itself from the mirrored data, restoring its state and configuration as it was before the switchover. Forwarding through the fabric is interrupted for about one second.

The SRP switchover restarts the routing protocols and triggers a graceful restart because the routes need to be recomputed. This recalculation can take up to 90 seconds. Data continues to be forwarded through routes that were learned before the upgrade of the line module control planes.

Line Module Forwarding Plane Upgrade

While the applications on the SRP module and the line modules reconstruct themselves, they also begin to build up the new state for the forwarding subsystem. All applications on the line module signal the system when they are ready to start the forwarding upgrade.

Because upgrading the forwarding plane affects forwarding through the chassis for up to 30 seconds, unified ISSU does not proceed until the routing protocols have signaled that route reconvergence has completed. Unified ISSU then informs all line modules to simultaneously upgrade their forwarding subsystems.

The line modules then perform the following steps:

1. Halt forwarding through the line modules. Although any links to external devices remain up, incoming data is dropped.
2. Update any changed programmable hardware devices.
3. Update the forwarding subsystem with the new release and upgraded configuration data.
4. Update the routing tables with the reconverged routes.
5. Resume forwarding.

Unified ISSU Service Restoration Phase Overview

This is the final unified ISSU phase. At this point, all three major components of the router—the SRP modules, the line module control planes, and the line module forwarding planes—have been upgraded and forwarding has resumed through the chassis. The following actions take place during this phase:

• The CLI is re-enabled. All commands are made available to users.
• The SNMP agent is restarted and bulk statistics are collected and available for review.
• New login requests and logout requests are processed. The router begins to accept externally created events from sources other than the CLI and SNMP. These events typically come from user connections, RADIUS servers, and SRC software and SDX software, and include login requests, COA requests, multicast join requests, and so on.
• Logout requests that were cached at the start of the in-service software upgrade are processed.
• After the flash memory on the newly active SRP module is updated, stateful SRP switchover is available to the router.

At this point the in-service software upgrade is completed, and the router is restored to normal operation. Any line modules that reloaded during the upgrade phase and were therefore held down are now rebooted to the original running release.