Workflow: Upgrading an IDP OS 4.1r4 Cluster to IDP OS 5.1

The upgrade from IDP OS 4.1r4 to IDP OS 5.1 is a major task. It is not a matter of simply pushing the new OS to the cluster members. You must perform the following steps in the order shown.

When you upgrade from IDP OS 4.1r4 to IDP OS 5.1, you are reimaging the disk with a new operating system. All partitions except /var/idp are rewritten.

The upgrade process restores your license and most of your previous settings. The following settings are not preserved:

The upgrade process preserves packet log files in /usr/idp/device/var/pktlogs/0/. Packet log files in other directories will be lost upon upgrade. If you have been using the option to maintain packet data locally and send to NSM on demand, copy logs from /usr/idp/device/var/pktlogs/1/ and higher numbered log directories to a remote location before you upgrade. This action is not required if you have been using the option to always include packet data when NSM sends the event log.

We have verified upgrade from an IDP OS 4.1r4 cluster where the IDP Series devices are deployed in transparent mode or sniffer mode. If the devices are deployed in bridge, proxy-arp, or router mode, you must redeploy them in IDP OS 4.1r4 transparent mode or sniffer mode before you can upgrade to IDP OS 5.1. Note, however, that HA is not applicable to sniffer mode deployments. Your target is a transparent mode deployment, so migration to IDP OS 5.1 might involve network topology replanning. In transparent mode, an HA cluster is an active-passive, failover pair. You deploy the IDP Series devices in redundant paths and rely on third-party link detection mechanisms to choose the active path.

To upgrade an active-passive, failover pair:

  1. Disconnect the HA interface cable.
  2. Upgrade the device deployed in the inactive path first.

    1. Log into the CLI and run the idp.sh stop command to stop the IDP engine.
    2. Upgrade the OS using either NSM or the CLI:

      • In NSM, push the software to the device, not the cluster. If you push to the cluster, you will disrupt traffic in the primary path.
      • From the CLI, execute the reimage shar file.

        During the upgrade, the console might display messages similar to the following:

        Critical: sc_flow.c:4020 sc_flow_ha_handle_msg: received SC_HA_MSG_TYPE_SYNC_FLOW_TABLE from major = 5 minor = 1 buid = 137260, my major = 4 my minor = 1  my build = 134028
        

        You can disregard these messages.

    3. Use ACM and the CLI to configure new and changed features:

      • On the ACM Configure High Availability page, enable Third-Party HA.
      • On the ACM Configure HA State-Sync page, assign the backup device ID 1.
      • On the ACM virtual routers page:

        • Assign interface pairs to virtual routers. Because HA was enabled on your IDP OS 4.1r4 configuration, the upgrade process creates a virtual router named vr0 that contains eth1 (the HA interface). One of the system requirements for HA is that a virtual router named vr0 contain eth1.
        • Specify transparent mode.
        • Specify Nics Off in case of failure or graceful shutdown.
        • Enable Layer 2 bypass.
        • (Optional) Enable PPM.
      • Log into the CLI and edit the user_funcs file:

        • Use the backup copy generated by the OS upgrade to add your custom settings back into the operative file: /usr/idp/device/bin/user_funcs.
        • (Optional) If you want to enable interface signaling, edit the user_funcs file and change the value of the ha_interface_signal setting to 1, as highlighted in the following example:
           #########################################################################
          #                             VARIABLES
          #########################################################################
          
          [...]
          #Enable or disable interface based third-party HA signaling
          
          #Enable or disable interface based third-party HA signaling
          #Setting this variable to 1,indicated that interface based
          #HA signaling should be used, and setting it to 1 indicates
          #to block STP and similar kind of traffic to enable traffic
          #switch-over by third-party HA devices.
          
          export ha_interface_signal=1
          
          # 'max_intf_recv_failed_cnt_nicbypass' - The maximum count value for any
          # data interface indicating the number of times the packet could not
          # be received by that interface. If the count for any interface reaches
          # this value nicBypass gets triggered.
          #  **WARNING**: Changing the value would require running 'idp.sh restart'.
          
          export max_intf_recv_failed_cnt_nicbypass=18
          
          # Define SCIO
          SCIO=/usr/idp/device/bin/scio
          
    4. Verify the device functions as expected:

      • Display interface statistics to verify traffic flow.
      • Run scio getsystem and verify that HA is enabled:

        [root@defaulthost ~]# scio getsystem
        Product Name:  NS-IDP-200
        Serial Number:  0146082005000328
        Software Version:  5.1.137260
        IDP Mode:  transparent
        HA Mode:  Enabled
        Detector Version:  5.1.110101209
        Software License:   Permanent
        Software Expiration Date:   never
        
      • Run idp.sh status and verify the status of the hasignal.sh process.
      • Run sctop to see if the eth1 interface is displayed as the sync interface.
      • Run scio vr list to verify that eth1 is listed in vr0.

        [root@defaulthost ~]# scio vr list
        Attached Virtual Routers:
        V-Router  V-Circuit  NIC
        --------  ---------  ----
        vr0       eth1       eth1     
                  eth3       eth3     
                  eth2       eth2
        
      • Check for the logs in /var/idp/device/sysinfo/logs/hasignal.timestamp:
        20101210030431:[WARN]:The UP & RUNNING nics are eth2 eth3 eth6 eth7
        20101210030431:[WARN]:The considered nics are eth2 eth3 eth4 eth5 eth6 eth7 eth8 eth9 eth10 eth11

      Every time the IDP engine is restarted, IDP device rebooted, ACM configuration changed, or HA interface signaling script restarted, the HA signaling script generates a pair of logs listing traffic interfaces.

      In the example above, the first line (UP & RUNNING nics) indicates the traffic interfaces monitored by the interface signaling script. The second line (considered nics) indicates all the traffic interfaces that have been enabled through the ACM Configure Virtual Routers page. If the first line does not include all the interfaces you expect to be monitored, run the hasignal.sh restart command and check the new logs.

  3. Upgrade the OS for the primary device.

    1. Log into the CLI and run the idp.sh stop command to stop the IDP engine. Stopping the IDP engine signals unavailability to firewalls or routers monitoring the links and cause traffic to cutover to the redundant path.
    2. Upgrade the OS using either NSM or the CLI:

      • In NSM, push the software to the device.
      • From the CLI, execute the installation shar file.
    3. Use ACM and the CLI to configure new and changed features:

      • On the ACM Configure High Availability page, enable Third-Party HA.
      • On the ACM Configure HA State-Sync page, assign the primary device ID 0.
      • On the ACM virtual routers page:

        • Assign interface pairs to virtual routers. Because HA was enabled on your IDP OS 4.1r4 configuration, the upgrade process creates a virtual router named vr0 that contains eth1 (the HA interface). One of the system requirements for HA is that a virtual router named vr0 contain eth1.
        • Specify transparent mode.
        • Specify Nics Off in case of failure or graceful shutdown.
        • Enable Layer 2 bypass.
        • (Optional) Enable PPM.
      • Log into the CLI and edit the user_funcs file:

        • Use the backup copy generated by the OS upgrade to add your custom settings back into the operative file: /usr/idp/device/bin/user_funcs.
        • (Optional) In the user_funcs file, change the value of the ha_interface_signal setting to 1 if you want to enable interface signaling.
    4. Verify the device functions as expected.
  4. Reconnect the HA interface cable.
  5. Use the sctop utility flow tables to verify state sync; and examine device logs to verify HA operations. Logs related to HA communication are written locally to /var/idp/device/sysinfo/logs/hasignal.timestamp.
  6. (If necessary), use NSM to push the latest detector engine to the cluster.
  7. Use NSM to push the same security policy to the cluster.

Related Documentation

The following related topics are included in IDP Series Deployment Scenarios:

The following additional related topics are included in the IDP Series Administration Guide: