Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

TechLibrary
Navigation
Table of Contents
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Verifying Nonstop Software Upgrade for QFabric Systems

    This topic discusses how you can monitor the progress of each of the three steps in a nonstop software upgrade. By identifying the key actions and events that define this process, you can track the status of the upgrade with confidence.

    Tip: When performing a nonstop software upgrade, open two SSH sessions to the QFabric CLI. Use one session to monitor the upgrade itself and use a second session to verify that the QFabric system components respond to operational mode commands as expected.

    Verifying a Director Group Nonstop Software Upgrade

    Purpose

    During the Director group portion of a nonstop software upgrade, you should expect to see the Director device that hosts the CLI session selected as the master device. When mastership of all processes moves to the master, the QFabric system upgrades the backup Director device and this Director device reboots. After the backup Director device comes back online, the master Director device suspends CLI operations for 15 minutes, upgrades itself, and reboots. At this point, the backup becomes the new master Director device and you can issue CLI operational commands. Finally, the former master comes back online as a backup and both devices are operational once again.

    Action

    1. In one SSH session to the QFabric CLI, verify the current status of the QFabric system by issuing the show fabric administration inventory, show fabric administration inventory director-group status, and show fabric session-host commands. In this case, Director device DG0 is the master device but DG1 hosts the CLI session.
      session1@qfabric> show fabric administration inventory
      Item                    Identifier              Connection      Configuration
Node group
  NW-NG-0                                       Connected       Configured    
    P1507-C                                     Connected                     
  RSNG                                          Connected       Configured    
    P1550-C                                     Connected                     
    P1571-C                                     Connected                     
Interconnect device
  IC-F4912                                      Connected       Configured    
    F4912/RE0                                   Connected                     
Fabric manager
  FM-0                                          Connected       Configured    
Fabric control
  FC-0                                          Connected       Configured    
  FC-1                                          Connected       Configured    
Diagnostic routing engine
  DRE-0                                         Connected       Configured
      session1@qfabric> show fabric administration inventory director-group status 
      Director Group Status Tue Jun  5 15:11:26 UTC 2012

 Member Status Role     Mgmt Address    CPU Free Memory VMs Up Time
 ------ ------ -------- --------------- --- ----------- --- -------------
 dg0    online master   10.49.215.38    8%  17363152k   4   3 days, 20:55 hrs
 dg1    online backup   10.49.215.39    6%  20157440k   3   3 days, 20:55 hrs

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg0    0281052011000001 online  master   

  Master Services
  ---------------
  Database Server                online    
  Load Balancer Director         online    
  QFabric Partition Address      online    

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:4                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.4G 395G  2%   /          
  99M  16M  79M   17%  /boot      
  93G  7.3G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     master                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up        

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg1    0281052011000032 online  backup   

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:8                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.5G 395G  2%   /          
  99M  16M  79M   17%  /boot            
  93G  7.3G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     backup                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up
      session1@qfabric> show fabric session-host
      Identifier: 0281052011000032
    2. In a second SSH session to the QFabric CLI, issue the request for the Director group nonstop software upgrade.
      root@qfabric> request system software nonstop-upgrade director-group jinstall-qfabric-12.2X50-D10.3.rpm
    3. If the CLI session is being hosted by the master Director device, skip to step 4. However, if the CLI session is hosted by the backup Director device, the Director group mastership switches to the backup device after you issue the nonstop software upgrade command. In this example, mastership switches to Director device DG1.
      session1@qfabric> show fabric administration inventory director-group status 
      Director Group Status Tue Jun  5 15:12:20 UTC 2012

 Member Status Role     Mgmt Address    CPU Free Memory VMs Up Time
 ------ ------ -------- --------------- --- ----------- --- -------------
 dg0    online backup   10.49.215.38    8%  31905924k   0   3 days, 21:16 hrs
 dg1    online master   10.49.215.39    6%  18010368k   3   3 days, 21:16 hrs

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg0    0281052011000001 online  backup   

  Director Group Managed Services
  -------------------------------
  Shared File System             offline   
  Network File System            offline   
  Virtual Machine Server         offline   
  Load Balancer/DHCP             offline   

  Hard Drive Status
  ----------------
  Volume ID:4                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.4G 395G  2%   /          
  99M  16M  79M   17%  /boot      

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            offline   
  DHCP Server master             offline    backup                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         offline   
  SNMP Trap Forwarder            offline   
  SNMP Process                   offline   
  Platform Management            online    
                                        
  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up        

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg1    0281052011000032 online  master   

  Master Services
  ---------------
  Database Server                online    
  Load Balancer Director         online    
  QFabric Partition Address      online    

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:8                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 6.0G 395G  2%   /          
  99M  16M  79M   17%  /boot      
  93G  7.3G 86G   8%   /pbdata          

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     master                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up
      session1@qfabric> show fabric session-host
      Identifier: 0281052011000032
    4. The Director group nonstop software upgrade process continues by downloading and installing software for the fabric manager Routing Engines and the Director devices.
      root@qfabric> 
      Validating update package jinstall-qfabric-12.2X50-D10.3.rpm
Installing update package jinstall-qfabric-12.2X50-D10.3.rpm
Installing fabric images version 12.2X50-D10.3
Performing cleanup
Package install complete
Installing update package jinstall-qfabric-12.2X50-D10.3.rpm on peer
Triggering Initial Stage of Fabric Manager Upgrade
Updating CCIF default image to 12.2X50-D10.3
Updating FM-0 to Junos version 12.2X50-D10.3
[Status   2012-06-05 15:25:29]: Fabric Manager: Upgrade Initial Stage started
[FM-0     2012-06-05 15:25:38]: FM-0 Master already running on LOCAL DG
[NW-NG-0  2012-06-05 15:25:45]: NW-NG-0 Master already running on LOCAL DG
[FM-0     2012-06-05 15:26:12]: Retrieving package
[FM-0     2012-06-05 15:27:11]: Pushing bundle to re0
[Status   2012-06-05 15:29:06]: Load completed with 0 errors...
[Status   2012-06-05 15:29:06]: Reboot is required to complete upgrade ...
[Status   2012-06-05 15:29:07]: Trying to Connect to Node: FM-0
[Status   2012-06-05 15:29:13]: Rebooting FM-0
[FM-0     2012-06-05 15:29:13]: Waiting for FM-0 to terminate ...
Starting Peer upgrade

Initiating rolling upgrade of Director peer:  version 12.2X50-D10.3

Inform CCIF regarding rolling upgrade
[Peer Update Status]: Validating install package jinstall-qfabric-12.2X50-D10.3.rpm
[Peer Update Status]: Cleaning up node for rolling phase one upgrade
[Peer Update Status]: Director group upgrade complete
[Peer Update Status]: COMPLETED
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to return after reboot and start phase one of rolling upgrade
[Peer Update Status]: Waiting for peer to complete phase one of rolling upgrade
[Peer Update Status]: Peer completed phase one of rolling upgrade
    5. When the system upgrades and reboots the backup Director device DG0, notice how this device is not displayed in the output of the show fabric administration inventory director-group status command. Because Director device DG1 appears, this means that the DG1 is operational and acts as the master device.

      Note: If your second SSH session is being hosted by the rebooting Director device, your session terminates and you need to log back in to establish a new session running on the active Director device.

      session1@qfabric> show fabric administration inventory director-group status 
      Director Group Status Tue Jun  5 15:41:14 UTC 2012

 Member Status Role     Mgmt Address    CPU Free Memory VMs Up Time
 ------ ------ -------- --------------- --- ----------- --- -------------
 dg1    online master   10.49.215.39    6%  8372272k    4   3 days, 21:25 hrs

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg1    0281052011000032 online  master   

  Master Services
  ---------------
  Database Server                online    
  Load Balancer Director         online    
  QFabric Partition Address      online    

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:8                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 6.0G 395G  2%   /          
  99M  16M  79M   17%  /boot      
  93G  7.3G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     master                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up
    6. The upgrade continues with master Director device DG1 suspending CLI services for 15 minutes, transferring mastership to Director device DG0, and then rebooting Director device DG1 (which terminates the CLI session).
      root@qfabric> 
      [Peer Update Status]: Setting peer DG node as the master SFC

Delaying start of local upgrade to allow peer services time to initialize [15 minutes]
Delaying start of local upgrade to allow peer services time to initialize [15 minutes]
Delaying start of local upgrade to allow peer services time to initialize [12 minutes]
Delaying start of local upgrade to allow peer services time to initialize [9 minutes]
Delaying start of local upgrade to allow peer services time to initialize [6 minutes]
Delaying start of local upgrade to allow peer services time to initialize [3 minutes]
[Peer Update Status]: Check for VMs on dg0
Triggering Final Stage of Fabric Manager Upgrade:
Updating FM-0 to Junos version 12.2X50-D10.3
[Status   2012-06-05 16:10:12]: Fabric Manager: Upgrade Final Stage started
[NW-NG-0  2012-06-05 16:10:22]: Transferring NW-NG-0 Mastership to REMOTE DG
[NW-NG-0  2012-06-05 16:11:44]: Finished NW-NG-0 Mastership switch
[Status   2012-06-05 16:11:45]: Upgrading FM-0 VM on worker DG to 12.2X50-D10.3
[DRE-0    2012-06-05 16:12:43]: Retrieving package
[DRE-0    2012-06-05 16:13:46]: ------- re0: -------
[Status   2012-06-05 16:15:17]: Load completed with 0 errors...
[Status   2012-06-05 16:15:17]: Reboot is required to complete upgrade ...
[DRE-0    2012-06-05 16:15:22]: Waiting for DRE-0 to terminate ...
[DRE-0    2012-06-05 16:15:34]: Waiting for DRE-0 to come back ...
[DRE-0    2012-06-05 16:18:44]: Running Uptime Test for DRE-0
[DRE-0    2012-06-05 16:18:51]: Uptime Test for DRE-0 Passed ...
[Status   2012-06-05 16:18:51]: DRE-0 booted successfully ...
Performing post install shutdown and cleanup

Broadcast message from root (Tue Jun  5 16:18:51 2012):

The system is going down for reboot NOW!
Director group upgrade complete

root@qfabric> Read from remote host qfabric-partition0: Connection reset by peer
Connection to qfabric-partition0 closed.
    7. Upon reopening the SSH session, notice that Director device DG0 is now the master device hosting the session and Director device DG1 does not appear in the QFabric system inventory while it is rebooting.
      session1@qfabric> show fabric session-host
      Identifier: 0281052011000001
      session1@qfabric> show fabric administration inventory director-group status 
      Director Group Status Tue Jun  5 16:21:23 UTC 2012

 Member Status Role     Mgmt Address    CPU Free Memory VMs Up Time
 ------ ------ -------- --------------- --- ----------- --- -------------
 dg0    online master   10.49.215.38    13% 20739560k   3   36:29 mins       

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg0    0281052011000001 online  master   

  Master Services
  ---------------
  Database Server                online    
  Load Balancer Director         online    
  QFabric Partition Address      online    

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:4                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.3G 396G  2%   /          
  99M  16M  79M   17%  /boot      
  93G  7.4G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     master                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up
    8. When Director device DG1 comes back online, it returns to the QFabric system inventory as a backup Director device and hosts some of the Routing Engine processes (which should appear load balanced between the master and backup Director devices).
      session1@qfabric> show fabric administration inventory director-group status 
      root@qfabric> show fabric administration inventory director-group status    
Director Group Status Tue Jun  5 16:41:02 UTC 2012

 Member Status Role     Mgmt Address    CPU Free Memory VMs Up Time
 ------ ------ -------- --------------- --- ----------- --- -------------
 dg0    online master   10.49.215.38    15% 14759920k   6   56:09 mins       
 dg1    online backup   10.49.215.39    8%  31486680k   0   07:51 mins       

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg0    0281052011000001 online  master   

  Master Services
  ---------------
  Database Server                online    
  Load Balancer Director         online    
  QFabric Partition Address      online    

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:4                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.3G 396G  2%   /          
  99M  16M  79M   17%  /boot      
  93G  7.4G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     master                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up        

 Member Device Id/Alias  Status  Role
 ------ ---------------- ------- ---------
 dg1    0281052011000032 online  backup   

  Director Group Managed Services
  -------------------------------
  Shared File System             online    
  Network File System            online    
  Virtual Machine Server         online    
  Load Balancer/DHCP             online    

  Hard Drive Status
  ----------------
  Volume ID:8                    optimal    
  Physical ID:1                  online     
  Physical ID:0                  online     
  SCSI ID:1                      100%       
  SCSI ID:0                      100%       


  Size  Used Avail Used% Mounted on 
  ----  ---- ----- ----- ----------
  423G 5.3G 396G  2%   /          
  99M  16M  79M   17%  /boot            
  93G  7.4G 86G   8%   /pbdata    

  Director Group Processes
  ------------------------
  Director Group Manager         online    
  Partition Manager              online    
  Software Mirroring             online    
  Shared File System master      online    
  Secure Shell Process           online    
  Network File System            online    
  DHCP Server master             online     backup                           
  FTP Server                     online    
  Syslog                         online    
  Distributed Management         online    
  SNMP Trap Forwarder            online    
  SNMP Process                   online    
  Platform Management            online    

  Interface Link Status
  ---------------------
  Management Interface           up        
  Control Plane Bridge           up        
  Control Plane LAG              up        
  CP Link [0/2]                  up        
  CP Link [0/1]                  up        
  CP Link [0/0]                  up        
  CP Link [1/2]                  down      
  CP Link [1/1]                  down      
  CP Link [1/0]                  down      
  Crossover LAG                  up        
  CP Link [0/3]                  up        
  CP Link [1/3]                  up
      session1@qfabric> show fabric administration inventory infrastructure
      dg0:
   Routing Engine Type                   Hostname              PID      CPU-Use(%)
------------------------- --------------------------------- ---------- -------------
 Fabric control              QFabric_default_FC-1_RE0         27906       2.5            
 Network Node group          QFabric_default_NW-NG-1_RE1      20421       1.8            
 Fabric manager              FM-0                             4211        1.8            
 Debug Routing Engine        QFabric_DRE                      1575        3.3            
dg1:
   Routing Engine Type                   Hostname              PID      CPU-Use(%)
------------------------- --------------------------------- ---------- -------------
 Fabric control              QFabric_default_FC-0_RE0         5686        2.3            
 Network Node group          QFabric_default_NW-NG-0_RE0      5866        1.9            
 Fabric manager              FM-1                             572         1.6

    Meaning

    Verifying a Fabric Nonstop Software Upgrade

    Purpose

    During the fabric portion of a nonstop software upgrade, you should expect to see both fabric control Routing Engines upgrade first, followed by the upgrade of each Interconnect device one at a time.

    Action

    1. In an SSH session to the QFabric CLI, issue the request for the fabric nonstop software upgrade.
      root@qfabric> request system software nonstop-upgrade fabric jinstall-qfabric-12.2X50-D10.3.rpm
      [FC-0     2012-06-05 16:48:53]: Retrieving package
[FC-1     2012-06-05 16:48:53]: Retrieving package
[IC-F4912 2012-06-05 16:48:59]: Retrieving package
[FC-0     2012-06-05 16:49:51]: ------- re0: -------
[FC-1     2012-06-05 16:49:52]: ------- re0: -------
[IC-F4912 2012-06-05 16:49:54]: ------- re0: -------
[IC-F4912 2012-06-05 16:50:42]: Step 1 of 20 Creating temporary file system
[IC-F4912 2012-06-05 16:50:42]: Step 2 of 20 Determining installation source
[IC-F4912 2012-06-05 16:50:43]: Step 3 of 20 Processing format options
[IC-F4912 2012-06-05 16:50:43]: Step 4 of 20 Determining installation slice
[IC-F4912 2012-06-05 16:50:43]: Step 5 of 20 Creating and labeling new slices
[IC-F4912 2012-06-05 16:50:44]: Step 6 of 20 Create and mount new file system
[IC-F4912 2012-06-05 16:50:53]: Step 7 of 20 Getting OS bundles
[IC-F4912 2012-06-05 16:50:53]: Step 8 of 20 Updating recovery media
[IC-F4912 2012-06-05 16:51:17]: Step 9 of 20 Extracting incoming image
[IC-F4912 2012-06-05 16:52:56]: Step 10 of 20 Unpacking OS packages
[IC-F4912 2012-06-05 16:52:59]: Step 11 of 20 Mounting jbase package
[IC-F4912 2012-06-05 16:53:28]: Step 12 of 20 Creating base OS symbolic links
[IC-F4912 2012-06-05 16:54:45]: Step 13 of 20 Creating fstab
[IC-F4912 2012-06-05 16:54:45]: Step 14 of 20 Creating new system files
[IC-F4912 2012-06-05 16:54:46]: Step 15 of 20 Adding jbundle package
[IC-F4912 2012-06-05 16:58:15]: Step 16 of 20 Backing up system data
[IC-F4912 2012-06-05 16:58:18]: Step 17 of 20 Setting up shared partition data
[IC-F4912 2012-06-05 16:58:18]: Step 18 of 20 Checking package sanity in installation
[IC-F4912 2012-06-05 16:58:18]: Step 19 of 20 Unmounting and cleaning up temporary file systems
[IC-F4912 2012-06-05 16:58:22]: Step 20 of 20 Setting da0s1 as new active partition
[Status   2012-06-05 16:58:34]: Load completed with 0 errors...
[Status   2012-06-05 16:58:34]: Reboot is required to complete upgrade ...
[Status   2012-06-05 16:58:34]: Trying to Connect to Node: FC-0
[Status   2012-06-05 16:58:39]: Rebooting FC-0
[Status   2012-06-05 16:58:39]: Trying to Connect to Node: FC-1
[Status   2012-06-05 16:58:44]: Rebooting FC-1
[Status   2012-06-05 16:58:44]: Trying to Connect to Node: IC-F4912
[Status   2012-06-05 16:58:50]: Rebooting IC-F4912
Success
    2. When the fabric components reboot, they appear as Disconnected in the output of the show fabric administration inventory infrastructure fabric-controls and show fabric administration inventory interconnect-devices commands.
      session1@qfabric> show fabric administration inventory infrastructure fabric-controls
      Item                    Identifier              Connection      Configuration
Fabric control
  FC-0                                          Disconnected
  FC-1                                          Disconnected
      session1@qfabric> show fabric administration inventory interconnect-devices IC-F4912
      Item                    Identifier              Connection      Configuration
Interconnect device
  IC-F4912                                      Disconnected
    F4912/RE0                                   Disconnected
    3. When the fabric components return to full service, they appear as Connected in the output of the show fabric administration inventory command.
      session1@qfabric> show fabric administration inventory
      Item                    Identifier              Connection      Configuration
Node group
  NW-NG-0                                       Connected       Configured    
    P1507-C                                     Connected                     
  RSNG                                          Connected       Configured    
    P1550-C                                     Connected                     
    P1571-C                                     Connected                     
Interconnect device
  IC-F4912                                      Connected       Configured    
    F4912/RE0                                   Connected                     
Fabric manager
  FM-0                                          Connected       Configured    
Fabric control
  FC-0                                          Connected       Configured    
  FC-1                                          Connected       Configured    
Diagnostic routing engine
  DRE-0                                         Connected       Configured

    Meaning

    Verifying a Redundant Server Node Group Nonstop Software Upgrade

    Purpose

    During the redundant server Node group portion of a nonstop software upgrade, you should expect to see the backup Node device upgrade first, followed by the upgrade of the master Node device. Server Node groups with a single device upgrade the device in the same way as a standalone switch.

    Action

    1. In an SSH session to the QFabric CLI, issue the request for the redundant server Node group nonstop software upgrade.
      root@qfabric> request system software nonstop-upgrade node-group RSNG jinstall-qfabric-12.2X50-D10.3.rpm 
      Upgrading target(s): RSNG

[RSNG     2012-06-05 17:26:44]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D10.3/jinstall-qfx.tgz
[RSNG     2012-06-05 17:26:44]: Retrieving package
[RSNG     2012-06-05 17:28:56]: Pushing bundle to fpc1
[RSNG     2012-06-05 17:29:26]: fpc1: Validate package...
[RSNG     2012-06-05 17:35:22]: fpc0: Validate package...
[RSNG     2012-06-05 17:35:49]: ------- fpc1 -------
[RSNG     2012-06-05 17:36:25]: Step 1 of 20 Creating temporary file system
[RSNG     2012-06-05 17:36:26]: Step 2 of 20 Determining installation source
[RSNG     2012-06-05 17:36:26]: Step 3 of 20 Processing format options
[RSNG     2012-06-05 17:36:26]: Step 4 of 20 Determining installation slice
[RSNG     2012-06-05 17:36:27]: Step 5 of 20 Creating and labeling new slices
[RSNG     2012-06-05 17:36:27]: Step 6 of 20 Create and mount new file system
[RSNG     2012-06-05 17:36:35]: Step 7 of 20 Getting OS bundles
[RSNG     2012-06-05 17:36:35]: Step 8 of 20 Updating recovery media
[RSNG     2012-06-05 17:36:56]: Step 9 of 20 Extracting incoming image
[RSNG     2012-06-05 17:38:07]: Step 10 of 20 Unpacking OS packages
[RSNG     2012-06-05 17:38:16]: Step 11 of 20 Mounting jbase package
[RSNG     2012-06-05 17:38:41]: Step 12 of 20 Creating base OS symbolic links
[RSNG     2012-06-05 17:39:41]: Step 13 of 20 Creating fstab
[RSNG     2012-06-05 17:39:42]: Step 14 of 20 Creating new system files
[RSNG     2012-06-05 17:39:42]: Step 15 of 20 Adding jbundle package
[RSNG     2012-06-05 17:42:16]: Step 16 of 20 Backing up system data
[RSNG     2012-06-05 17:42:32]: Step 17 of 20 Setting up shared partition data
[RSNG     2012-06-05 17:42:33]: Step 18 of 20 Checking package sanity in installation
[RSNG     2012-06-05 17:42:33]: Step 19 of 20 Unmounting and cleaning up temporary file systems
[RSNG     2012-06-05 17:42:36]: Step 20 of 20 Setting da0s2 as new active partition
[RSNG     2012-06-05 17:42:51]: ------- fpc0 - master -------
[RSNG     2012-06-05 17:42:51]: Step 1 of 20 Creating temporary file system
[RSNG     2012-06-05 17:42:51]: Step 2 of 20 Determining installation source
[RSNG     2012-06-05 17:42:51]: Step 3 of 20 Processing format options
[RSNG     2012-06-05 17:42:51]: Step 4 of 20 Determining installation slice
[RSNG     2012-06-05 17:42:51]: Step 5 of 20 Creating and labeling new slices
[RSNG     2012-06-05 17:42:51]: Step 6 of 20 Create and mount new file system
[RSNG     2012-06-05 17:42:51]: Step 7 of 20 Getting OS bundles
[RSNG     2012-06-05 17:42:51]: Step 8 of 20 Updating recovery media
[RSNG     2012-06-05 17:42:51]: Step 9 of 20 Extracting incoming image
[RSNG     2012-06-05 17:42:51]: Step 10 of 20 Unpacking OS packages
[RSNG     2012-06-05 17:42:51]: Step 11 of 20 Mounting jbase package
[RSNG     2012-06-05 17:42:51]: Step 12 of 20 Creating base OS symbolic links
[RSNG     2012-06-05 17:42:51]: Step 13 of 20 Creating fstab
[RSNG     2012-06-05 17:42:51]: Step 14 of 20 Creating new system files
[RSNG     2012-06-05 17:42:51]: Step 15 of 20 Adding jbundle package
[RSNG     2012-06-05 17:42:51]: Step 16 of 20 Backing up system data
[RSNG     2012-06-05 17:42:51]: Step 17 of 20 Setting up shared partition data
[RSNG     2012-06-05 17:42:51]: Step 18 of 20 Checking package sanity in installation
[RSNG     2012-06-05 17:42:51]: Step 19 of 20 Unmounting and cleaning up temporary file systems
[RSNG     2012-06-05 17:42:51]: Step 20 of 20 Setting da0s2 as new active partition
[RSNG     2012-06-05 17:43:36]: Rebooting Backup RE
[RSNG     2012-06-05 17:43:36]: ------- Rebooting fpc1 -------
[RSNG     2012-06-05 17:50:12]: Initiating Chassis In-Service-Upgrade
[RSNG     2012-06-05 17:50:33]: Upgrading group: 0 fpc: 0
[RSNG     2012-06-05 17:52:38]: Upgrade complete for group:0
[RSNG     2012-06-05 17:52:38]: Upgrading group: 1 fpc: 1
[RSNG     2012-06-05 17:54:42]: Upgrade complete for group:1
[RSNG     2012-06-05 17:54:42]: Finished processing all upgrade groups, last group :1
[RSNG     2012-06-05 17:54:48]: Preparing for Switchover
[RSNG     2012-06-05 17:55:38]: Switchover Completed
[Status   2012-06-05 17:55:41]: Upgrade completed with 0 errors
Success
    2. During the redundant server Node group upgrade, the backup Node device (in this case, P1571-C) is upgraded first and appears in the Disconnected state in the output of the show fabric administration inventory command.
      session1@qfabric> show fabric administration inventory
      Item                    Identifier              Connection      Configuration
Node group
  NW-NG-0                                       Connected       Configured    
    P1507-C                                     Connected                     
  RSNG                                          Connected       Configured    
    P1550-C                                     Connected                     
    P1571-C                                     Disconnected
Interconnect device
  IC-F4912                                      Connected       Configured    
    F4912/RE0                                   Connected                     
Fabric manager
  FM-0                                          Connected       Configured    
Fabric control
  FC-0                                          Connected       Configured    
  FC-1                                          Connected       Configured    
Diagnostic routing engine
  DRE-0                                         Connected       Configured
    3. After the backup Node device comes back online, the master Node device (in this case, P1550-C) appears in the Disconnected state in the output of the show fabric administration inventory command while the master Node device upgrades its software.
      session1@qfabric> show fabric administration inventory
      Item                    Identifier              Connection      Configuration
Node group
  NW-NG-0                                       Connected       Configured    
    P1507-C                                     Connected                     
  RSNG                                          Connected       Configured    
    P1550-C                                     Disconnected
    P1571-C                                     Connected                     
Interconnect device
  IC-F4912                                      Connected       Configured    
    F4912/RE0                                   Connected                     
Fabric manager
  FM-0                                          Connected       Configured    
Fabric control
  FC-0                                          Connected       Configured    
  FC-1                                          Connected       Configured    
Diagnostic routing engine
  DRE-0                                         Connected       Configured
    4. After both Node devices in the redundant server Node group come back online, both Node devices appear as Connected to indicate the successful completion of the Node group nonstop software upgrade step.
      session1@qfabric> show fabric administration inventory
      Item                    Identifier              Connection      Configuration
Node group
  NW-NG-0                                       Connected       Configured    
    P1507-C                                     Connected                     
  RSNG                                          Connected       Configured    
    P1550-C                                     Connected
    P1571-C                                     Connected                     
Interconnect device
  IC-F4912                                      Connected       Configured    
    F4912/RE0                                   Connected                     
Fabric manager
  FM-0                                          Connected       Configured    
Fabric control
  FC-0                                          Connected       Configured    
  FC-1                                          Connected       Configured    
Diagnostic routing engine
  DRE-0                                         Connected       Configured

    Verifying a Network Node Group Nonstop Software Upgrade

    Purpose

    During the network Node group portion of a nonstop software upgrade, you should expect to see the backup network Node group Routing Engine upgrade first, followed by the Node devices within the network Node group upgrading one at a time, and ending with the upgrade of the master network Node group Routing Engine.

    Note: If you configure an upgrade group for Node groups containing 2 or more Node devices, all Node devices within the upgrade group reboot at the same time.

    Action

    • In an SSH session to the QFabric CLI, issue the request for the network Node group nonstop software upgrade.
      root@qfabric> request system software nonstop-upgrade node-group NW-NG-0 jinstall-qfabric-12.2X50-D10.3.rpm
      Upgrading target(s): NW-NG-0

[NW-NG-0  2012-06-01 09:45:06]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D10.3/jinstall-qfx.tgz
[NW-NG-0  2012-06-01 09:45:06]: Retrieving package
[NW-NG-0  2012-06-01 09:46:18]: Pushing bundle to fpc0
[NW-NG-0  2012-06-01 09:46:52]: fpc0: Validate package...
[NW-NG-0  2012-06-01 09:53:26]: ------- fpc0 -------
[NW-NG-0  2012-06-01 09:54:01]: Step 1 of 20 Creating temporary file system
[NW-NG-0  2012-06-01 09:54:01]: Step 2 of 20 Determining installation source
[NW-NG-0  2012-06-01 09:54:02]: Step 3 of 20 Processing format options
[NW-NG-0  2012-06-01 09:54:02]: Step 4 of 20 Determining installation slice
[NW-NG-0  2012-06-01 09:54:02]: Step 5 of 20 Creating and labeling new slices
[NW-NG-0  2012-06-01 09:54:03]: Step 6 of 20 Create and mount new file system
[NW-NG-0  2012-06-01 09:54:10]: Step 7 of 20 Getting OS bundles
[NW-NG-0  2012-06-01 09:54:10]: Step 8 of 20 Updating recovery media
[NW-NG-0  2012-06-01 09:54:31]: Step 9 of 20 Extracting incoming image
[NW-NG-0  2012-06-01 09:55:43]: Step 10 of 20 Unpacking OS packages
[NW-NG-0  2012-06-01 09:55:46]: Step 11 of 20 Mounting jbase package
[NW-NG-0  2012-06-01 09:56:09]: Step 12 of 20 Creating base OS symbolic links
[NW-NG-0  2012-06-01 09:57:05]: Step 13 of 20 Creating fstab
[NW-NG-0  2012-06-01 09:57:05]: Step 14 of 20 Creating new system files
[NW-NG-0  2012-06-01 09:57:05]: Step 15 of 20 Adding jbundle package
[NW-NG-0  2012-06-01 09:59:30]: Step 16 of 20 Backing up system data
[NW-NG-0  2012-06-01 09:59:44]: Step 17 of 20 Setting up shared partition data
[NW-NG-0  2012-06-01 09:59:44]: Step 18 of 20 Checking package sanity in installation
[NW-NG-0  2012-06-01 09:59:44]: Step 19 of 20 Unmounting and cleaning up temporary file systems
[NW-NG-0  2012-06-01 09:59:47]: Step 20 of 20 Setting da0s1 as new active partition
[NW-NG-0  2012-06-01 09:59:55]: Starting with package ftp://169.254.0.3/pub/images/12.2X50-D10.3/jinstall-dc-re.tgz
[NW-NG-0  2012-06-01 09:59:55]: Retrieving package
[NW-NG-0  2012-06-01 10:01:04]: Pushing bundle to re1
[NW-NG-0  2012-06-01 10:01:35]: re1: Validate package...
[NW-NG-0  2012-06-01 10:02:56]: re0: Validate package...
[NW-NG-0  2012-06-01 10:04:45]: Rebooting Backup RE
[NW-NG-0  2012-06-01 10:08:31]: Initiating Chassis In-Service-Upgrade
[NW-NG-0  2012-06-01 10:08:52]: Upgrading group: 0 fpc: 0
[NW-NG-0  2012-06-01 10:18:33]: Upgrade complete for group:0
[NW-NG-0  2012-06-01 10:18:33]: Finished processing all upgrade groups, last group :0
[NW-NG-0  2012-06-01 10:18:37]: Preparing for Switchover
[NW-NG-0  2012-06-01 10:18:55]: Switchover Completed
[Status   2012-06-01 10:18:58]: Upgrade completed with 0 errors
Success
    • Verify the progress of the upgrade by issuing the show chassis nonstop-upgrade node-group, show fabric administration inventory, show fabric administration inventory infrastructure, and show fabric administration inventory node-groups NW-NG-0 commands. You should see the backup network Node group Routing Engine reboot first, followed by each Node device within the network Node group, and ending with the reboot of master network Node group Routing Engine. Restarting devices appear as Disconnected in the output of the show fabric administration inventory command and restarting Routing Engines do not appear in output of the show fabric administration inventory infrastructure command until they return to service.
     

    Related Documentation

     

    Published: 2012-09-26

    Previous Page Next Page