Performing a QFabric Switch Recovery Installation on the Director Group

If the software on your QFabric switch is damaged in some way that prevents the software from loading correctly, or you need to upgrade the software on your QFabric switch, you may need to perform a recovery installation on the Director group.

If possible, perform the following steps before you perform the recovery installation:

1. Ensure that you have an emergency boot device (for example, an external USB flash drive) for each of your Director devices to use during the recovery installation.

   You can either use the external USB flash drive containing the software supplied by Juniper Networks, or you can use an external USB flash drive supplied by Juniper Networks on which you install the QFabric switch install media.

2. Back up any configuration files and initial setup information on a different external USB flash drive and then restore this information as part of the recovery process. Use the request system software configuration-backup command to back up your configuration files and initial setup information:
   user@switch> request system software configuration-backup path

   Backing up the files to an external USB flash drive is the recommended method.

Note: To recover the Director group, you must upgrade both Director devices in parallel. If you are recovering only one Director device in a Director group, and the software version will remain the same between the two Director devices, make sure that the other Director device is powered on and operational. If the software version of the Director device you are recovering will be different, make sure that the other Director device is powered off and is not operational.

Note: The recovery installation process deletes the existing QFabric switch ID on the Director device and generates another one. If you encounter any licensing issues due to the changed QFabric switch ID, please contact the Juniper Technical Assistance Center (JTAC).

This topic describes the two ways to perform the recovery installation:

Performing a Recovery Installation Using a Juniper Networks External USB Flash Drive with Preloaded Software

1. Insert the external USB flash drive supplied by Juniper Networks into the Director device.
2. If you have access to the default partition, reboot the Director device by issuing the request system reboot director-group command.
3. If you do not have access to the default partition, power cycle the Director device.

   The following menu appears on the Director device console when the Director device boots up:

   Juniper Networks QFabric Director Install/Recovery Media
   -  To boot from the local disk, wait 10 seconds or press the Enter key.
   -  To reinstall the QFabric software on this Director device, type: install
   

4. Type install and then press Enter to install the software on the Director device.

   Once the installation process is complete, the Director device reboots, and the following menu appears on the Director device console:

   Juniper Networks QFabric Director Install/Recovery Media
   -  To boot from the local disk, wait 10 seconds or press the Enter key.
   -  To reinstall the QFabric software on this Director device, type: install
   

5. Press Enter.

   The Director device reboots from the local disk on which the software was just installed.

   Note: Since the recovery installation process completely overwrites the entire contents of the Director device, you will need to restore the required configuration files and initial setup information. Ensure that you have this information backed up on an external USB flash drive before you perform the following steps.

6. Log in as root on the Director device.

   The following menu appears on the Director device console:

   Before you can access the QFabric switch, you must complete the initial setup of the Director group by using the steps that follow.
   If the initial setup procedure does not complete successfully, log out of the Director device and then log back in to restart
    this setup menu.
   
   Continue?[y/n]

7. Enter n to bypass the initial setup script and enter the Director device root directory, where you can mount the external USB flash drive containing the configuration files and initial setup information.
8. Issue the ls /mnt command to list the mount directory.
   root@dg0 ~]# ls /mnt
9. Issue the mkdir command to create a directory within the mount directory.
   root@dg0 ~]# mkdir /mnt/myusb
10. Issue the mount /dev/sdb2 /mnt/myusb/ command to mount the external USB flash drive to the local drive of the Director device.
    root@dg0 ~]# mount /dev/sdb2 /mnt/myusb/
11. Issue the ls -la /mnt/mnt/myusb/ command to verify the contents of your mounted external USB flashdrive.
    root@dg0 ~]# ls -la /mnt/mnt/myusb/

    total 1770884
    drwxr-xr-x 2 root root       4096 Sep  7 05:16 .
    drwxr-xr-x 3 root root       4096 Sep  7 10:15 ..
    -rw-r--r-- 1 root root       4249 Sep  7 03:52 mybackup-20110907
    

12. Exit the Director device and log back in as root on the Director device.

    The following menu appears:

    Before you can access the QFabric switch, you must complete the initial setup of the Director group by using the steps that follow.
    If the initial setup procedure does not complete successfully, log out of the Director device and then log back in to restart
     this setup menu.
    
    Continue?[y/n] y
    Initial Configuration
    
    You may enter the configuration manually or restore from a backup.
    
    Specify a backup file? [y/n] : y
    Please specify the full path of the configuration backup file. : /mnt/myusb/mybackup-20110907

13. Enter y to continue.
14. Enter y and specify the path to the backup configuration file located on the external USB flash drive.
    /mnt/myusb/mybackup-20110907

    The following messages appear:

    Saving temporary configuration...
    Configuring peer...
    connect error for 1.1.1.2:9001
    Configuring local interfaces...
    Configuring interface eth0 with [10.49.213.163/24:10.49.213.254]
    Configured interface eth0 with [10.49.213.163/24:10.49.213.254]
    Configuring QFabric software with initial pool of 4000 MAC addresses [00:10:00:00:00:00 - 00:10:00:00:0f:3b]
    Configuring QFabric address [10.49.213.50]
    Reconfiguring QFabric software static configuration
    Applying the new Director Device password
    Applying the QFabric component password
    First install initial configuration, generating and sharing SSH keys.
    First install initial configuration, generating SSH keys.
    connect error for 1.1.1.2:9001
    Shared SSH keys.
    Configuration complete. Director Group services will auto start within 30 seconds.
    

    The Director device reboots from the local disk on which the software was just installed.

15. From the default partition, issue the request system reboot all command to reboot all of the components of the QFabric switch and to make sure that all components are running the same version of software.
    user@switch> request system reboot all
16. Log in to the default partition and issue the show version component all command to verify that all components are running the same version of software.
    user@switch> show version component all

    dg1:
    -
    Hostname: qfabric
    Model: qfx3100
    JUNOS Base Version [11.3X30.6]
    
    dg0:
    -
    Hostname: qfabric
    Model: qfx3100
    JUNOS Base Version [11.3X30.6]
    
    NW-NG-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FC-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FC-1:
    Hostname: qfabric                       
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    DRE-0:
    -
    Hostname: dre-0
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FM-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    nodedevice1:
    -
    Hostname: qfabric
    Model: QFX3500
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    interconnectdevice1:
    -
    Hostname: qfabric
    Model: QFX3108
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    warning:  from interconnectdevice0: Disconnected

Performing a Recovery Installation Using a Juniper Networks External Blank USB Flash Drive

If you do not have an external USB flash drive preloaded with the software from Juniper Networks to use as an emergency boot device, you can create your own, using a blank external USB flash drive provided by Juniper Networks. You need to download the install media from the Juniper Networks Support website onto your UNIX workstation, uncompress and untar the software, and then burn the software image onto your Juniper Networks external USB (4-gigabyte) flash drive. Make sure you create two emergency boot devices, one for each Director device, so you can perform a recovery installation in parallel.

Note: Because the recovery installation process completely overwrites the entire contents of the Director device, you will need to restore the required configuration files and initial setup information. Ensure that you have this information backed up on an external USB flash drive before you perform the following steps.

1. In a browser, go to http://www.juniper.net/support/downloads/junos.html.

   The Junos Platforms Download Software page appears.

2. In the QFX Series section of the Junos Platforms Download Software download page, select the QFX Series platform software you want to download.
3. Select the number of the software version that you want to download in the Release: pull-down window to the right of the tabs on the Download Software page.
4. Click the Software tab and select the complete install media you want to download in the QFabric Switch Install Package and Media section.

   A login screen appears.

5. Enter your name and password and press Enter.
6. Read the End User License Agreement, click the I agree radio button, and then click Proceed.
7. Log in and save the install media file to your UNIX workstation.
8. Use FTP to access the UNIX workstation where the install media resides.
   ftp ftp://hostname / pathname / install-media-qfabric-<version>.img.tgz
9. When prompted, enter your username and password.
10. Make sure you are in binary mode by entering binary at the prompt.
    binary
11. Use the get command to transfer the installation package from the FTP host to your UNIX workstation.
    get install-media-qfabric-<version>.img.tgz
12. Close the FTP session:
    bye
13. Untar the install-media-qfabric-<version> .img.tgz file on your UNIX workstation.
    tar -xvzf install-media-qfabric-11.3X30.6.img.tgz
14. Insert a blank external USB (4-gigabyte) flash drive supplied by Juniper Networks into your UNIX workstation.
15. Burn the software image you just downloaded to your UNIX workstation onto your external USB flash drive using the dd command:
    dd if=install-media-qfabric-11.3X30.6.img of=/dev/sdb bs=16k

    250880+0 records in
    250880+0 records out
    4110417920 bytes (4.1 GB) copied, 5.10768 seconds, 805 MB/s
    

16. If you have access to the default partition, reboot the Director device by issuing the request system reboot director-group command.
17. If you do not have access to the default partition, power cycle the Director device.

    The following menu appears on the Director device console when the Director device boots up:

    Juniper Networks QFabric Director Install/Recovery Media
    -  To boot from the local disk, wait 10 seconds or press the Enter key.
    -  To reinstall the QFabric software on this Director device, type: install
    

18. Type install and then press Enter to install the software on the Director device.

    Once the installation process is complete, the Director device reboots, and the following menu appears on the Director device console:

    Juniper Networks QFabric Director Install/Recovery Media
    -  To boot from the local disk, wait 10 seconds or press the Enter key.
    -  To reinstall the QFabric software on this Director device, type: install
    

19. Press Enter. The Director device reboots from the local disk on which the software was just installed.
20. Log in as root from the Director device.

    Note: Because the recovery installation process completely overwrites the entire contents of the Director device, you need to restore the required configuration files and initial setup information. Ensure that you have this information backed up on an external USB flash drive.

    The following menu appears on the Director device console:

    Before you can access the QFabric switch, you must complete the initial setup of the Director group by using the steps that follow.
    If the initial setup procedure does not complete successfully, log out of the Director device and then log back in to restart
     this setup menu.
    
    Continue?[y/n]

21. Enter n to bypass the initial setup script and enter the Director device root directory, where you can mount the external USB flash drive containing the configuration files and initial setup information.
22. Issue the ls /mnt command to list the mount directory.
    root@dg0 ~]# ls /mnt
23. Issue the mkdir command to create a directory within the mount directory.
    root@dg0 ~]# mkdir /mnt/myusb
24. Issue the mount /dev/sdb2 /mnt/myusb/ command to mount the external USB flash drive to the local drive of the Director device.
    root@dg0 ~]# mount /dev/sdb2 /mnt/myusb/
25. Issue the ls -la /mnt/mnt/myusb/ command to verify the contents of your mounted external USB flash drive.
    root@dg0 ~]# ls -la /mnt/mnt/myusb/

    total 1770884
    drwxr-xr-x 2 root root       4096 Sep  7 05:16 .
    drwxr-xr-x 3 root root       4096 Sep  7 10:15 ..
    -rw-r--r-- 1 root root       4249 Sep  7 03:52 mybackup-20110907
    

26. Exit the Director device and log back in as root on the Director device.

    The following menu appears:

    Before you can access the QFabric switch, you must complete the initial setup of the Director group by using the steps that follow.
    If the initial setup procedure does not complete successfully, log out of the Director device and then log back in to restart
     this setup menu.
    
    Continue?[y/n] y
    Initial Configuration
    
    You may enter the configuration manually or restore from a backup.
    
    Specify a backup file? [y/n] : y
    Please specify the full path of the configuration backup file. : /mnt/myusb/mybackup-20110907

27. Enter y to continue.
28. Enter y and specify the path to the backup configuration file located on the external USB flash drive.
    /mnt/myusb/mybackup-20110907

    The following messages appear:

    Saving temporary configuration...
    Configuring peer...
    connect error for 1.1.1.2:9001
    Configuring local interfaces...
    Configuring interface eth0 with [10.49.213.163/24:10.49.213.254]
    Configured interface eth0 with [10.49.213.163/24:10.49.213.254]
    Configuring QFabric software with initial pool of 4000 MAC addresses [00:10:00:00:00:00 - 00:10:00:00:0f:3b]
    Configuring QFabric address [10.49.213.50]
    Reconfiguring QFabric software static configuration
    Applying the new Director Device password
    Applying the QFabric component password
    First install initial configuration, generating and sharing SSH keys.
    First install initial configuration, generating SSH keys.
    connect error for 1.1.1.2:9001
    Shared SSH keys.
    Configuration complete. Director Group services will auto start within 30 seconds.
    

29. From the default partition, issue the show fabric administration inventory command to make sure all of the QFabric components are connected.
    user@switch> show fabric administration inventory

    Item                    Identifier              Connection      Configuration
    Node group
      BBAK8273                                      Connected       Configured    
        node6               BBAK8273                Connected                     
      BBAK8276                                      Connected       Configured    
        node5               BBAK8276                Connected                     
      BBAK8281                                      Connected       Configured    
        node7               BBAK8281                Connected                     
      BBAK8283                                      Connected       Configured    
        node2               BBAK8283                Connected                     
      BBAK8309                                      Connected       Configured    
        node1               BBAK8309                Connected                     
      BBAK8835                                      Connected       Configured    
        node8               BBAK8835                Connected                     
      NW-NG-0                                       Connected       Configured    
      service-group                                 Connected       Configured    
        node3               BBAK8891                Connected                     
        node4               BBAK8868                Connected                     
    Interconnect device
      IC-YW3781                                     Connected       Configured    
        YW3781/RE0                                  Connected                     
      IC-YW3798                                     Connected       Configured    
        YW3798/RE0                                  Connected                     
    Fabric manager
      FM-0                                          Connected       Configured    
    Fabric control
      FC-0                                          Connected       Configured    
      FC-1                                          Connected       Configured    
    Diagnostic routing engine
      DRE-0                                         Connected       Configured    
    

    Once all of the components are connected, the Director group checks the software version of each component to make sure that each component is running the same version of software as the Director group. If there is a difference in software version between a component and the Director group, the Director group will automatically upgrade or downgrade the software on the component to match the version of software installed on the Director group.

30. From the default partition, issue the show version component all command to verify that all components are running the same version of software.
    user@switch> show version component all

    dg1:
    -
    Hostname: qfabric
    Model: qfx3100
    JUNOS Base Version [11.3X30.6]
    
    dg0:
    -
    Hostname: qfabric
    Model: qfx3100
    JUNOS Base Version [11.3X30.6]
    
    NW-NG-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FC-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FC-1:
    Hostname: qfabric                       
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    DRE-0:
    -
    Hostname: dre-0
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    FM-0:
    -
    Hostname: qfabric
    Model: qfx-jvre
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    nodedevice1:
    -
    Hostname: qfabric
    Model: QFX3500
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    
    interconnectdevice1:
    -
    Hostname: qfabric
    Model: QFX3108
    JUNOS Base OS boot [11.3X30.6]
    JUNOS Base OS Software Suite [11.3X30.6]
    JUNOS Kernel Software Suite [11.3X30.6]
    JUNOS Crypto Software Suite [11.3X30.6]
    JUNOS Online Documentation [11.3X30.6]
    JUNOS Enterprise Software Suite [11.3X30.6]
    JUNOS Packet Forwarding Engine Support (QFX RE) [11.3X30.6]
    JUNOS Routing Software Suite [11.3X30.6]
    warning:  from interconnectdevice0: Disconnected