Troubleshooting: Software Upgrade Failure Due to Insufficient Free Disk Space on a Routing Matrix with a TX Matrix Plus Router

Problem

In a routing matrix with a TX Matrix Plus router, a successful upgrade of the Junos OS requires sufficient free disk space on all Routing Engines of the TX Matrix Plus router and connected T1600 routers.

If the amount of free disk space on a Routing Engine is insufficient to install the Junos OS, you might receive a warning that the /var filesystem is low on free disk space, similar to the following message:

WARNING: The /var filesystem is low on free disk space.
WARNING: This package requires 1075136k free, but there
WARNING: is only 666502k available.

Cause

The amount of free disk space necessary to upgrade a Routing Engine with a new version of the Junos OS can vary from one release to another. Check the Junos OS Release Notes for the software version you are installing.

To determine the amount of free disk space on each Routing Engine in the routing matrix, issue the show system storage command on the master Routing Engine and also on the backup Routing Engine on the TX Matrix Plus router. The command output is organized by chassis: sfc0, lcc0, lcc1, lcc2, and lcc3.

Note: When you issue the show system storage command on a TX Matrix Plus master Routing Engine, the command is broadcast to the master Routing Engines (only) on all T1600 routers in the routing matrix. Similarly, when you issue the show system storage command on a TX Matrix Plus backup Routing Engine, the command is broadcast to the backup Routing Engines (only) on all T1600 routers in the routing matrix.

The following example shows sample output for the show system storage command when it is issued on the master Routing Engine in slot 0 on the TX Matrix Plus router:

root@mysfc0> show system storage 
sfc0-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.4G       179M       3.2G        5%  /
devfs                   1.0K       1.0K         0B      100%  /dev
devfs                   1.0K       1.0K         0B      100%  /dev/
/dev/md0                 33M        33M         0B      100%  /packages/mnt/jbase
/dev/md1                217M       217M         0B      100%  /packages/mnt/jkernel-9.6-20090624.0
/dev/md2                 66M        66M         0B      100%  /packages/mnt/jpfe-T-9.6-20090624.0
/dev/md3                4.1M       4.1M         0B      100%  /packages/mnt/jdocs-9.6-20090624.0
/dev/md4                 57M        57M         0B      100%  /packages/mnt/jroute-9.6-20090624.0
/dev/md5                 15M        15M         0B      100%  /packages/mnt/jcrypto-9.6-20090624.0
/dev/md6                 34M        34M         0B      100%  /packages/mnt/jpfe-common-9.6-20090624.0
/dev/md7                2.0G       8.0K       1.8G        0%  /tmp
/dev/md8                2.0G       878K       1.8G        0%  /mfs
/dev/ad0s1e             383M        90K       379M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              24G       8.7G        13G       40%  /var

lcc0-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.4G       179M       2.9G        6%  /
devfs                   1.0K       1.0K         0B      100%  /dev
devfs                   1.0K       1.0K         0B      100%  /dev/
/dev/md0                 33M        33M         0B      100%  /packages/mnt/jbase
/dev/md1                217M       217M         0B      100%  /packages/mnt/jkernel-9.6-20090629.0
/dev/md2                 66M        66M         0B      100%  /packages/mnt/jpfe-T-9.6-20090629.0
/dev/md3                4.1M       4.1M         0B      100%  /packages/mnt/jdocs-9.6-20090629.0
/dev/md4                 57M        57M         0B      100%  /packages/mnt/jroute-9.6-20090629.0
/dev/md5                 15M        15M         0B      100%  /packages/mnt/jcrypto-9.6-20090629.0
/dev/md6                 34M        34M         0B      100%  /packages/mnt/jpfe-common-9.6-20090629.0
/dev/md7                2.0G       8.0K       1.8G        0%  /tmp
/dev/md8                2.0G       344K       1.8G        0%  /mfs
/dev/ad0s1e             383M        14K       352M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              24G       3.7G        18G       17%  /var

lcc2-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.4G       178M       2.9G        6%  /
devfs                   1.0K       1.0K         0B      100%  /dev
devfs                   1.0K       1.0K         0B      100%  /dev/
/dev/md0                 32M        32M         0B      100%  /packages/mnt/jbase
/dev/md1                217M       217M         0B      100%  /packages/mnt/jkernel-9.6I20090626
/dev/md2                 66M        66M         0B      100%  /packages/mnt/jpfe-T-9.6I20090626
/dev/md3                4.1M       4.1M         0B      100%  /packages/mnt/jdocs-9.6I20090626
/dev/md4                 57M        57M         0B      100%  /packages/mnt/jroute-9.6I20090626
/dev/md5                 15M        15M         0B      100%  /packages/mnt/jcrypto-9.6I20090626
/dev/md6                 34M        34M         0B      100%  /packages/mnt/jpfe-common-9.6I20090626
/dev/md7                2.0G      10.0K       1.8G        0%  /tmp
/dev/md8                2.0G       346K       1.8G        0%  /mfs
/dev/ad0s1e             383M        14K       352M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              24G        12G       9.4G       57%  /var

Solution

1. To free up space in the file system, you can log in to a particular Routing Engine and then issue the request system storage cleanup command. This command deletes rotating log files in /var/log that are not current, temporary files in /var/tmp that have not been modified within the last two days, and all crash files in /var/crash. The command output first lists the files proposed for deletion and then prompts you to proceed to delete these storage cleanup candidates.

   Note: Although the command prompts you before deleting files, you can issue the command with the dry-run option, which causes the command to only list the cleanup candidates without offering to remove them.

   The following example shows sample output for the request system storage cleanup command with the dry-run option included:

   root@mysfc0> request system storage cleanup dry-run

   Currently rotating log files, please wait.
   This operation can take up to a minute.
   
   List of files to delete:
   
            Size Date         Name
        2B Jun 16 12:46 /var/crash/bounds
     80.4M Jun 15 14:56 /var/crash/cores/kernel.0.090615.1455
     80.4M Jun 15 15:46 /var/crash/cores/kernel.1.090615.1546
     80.4M Jun 15 15:58 /var/crash/cores/kernel.2.090615.1558
     80.4M Jun 16 12:47 /var/crash/cores/kernel.3.090616.1246
     30.6M Jun 15 15:05 /var/crash/cores/vmcore.0.090615.1455.tgz
     33.5M Jun 15 16:33 /var/crash/cores/vmcore.1.090615.1546.tgz
     35.9M Jun 15 17:52 /var/crash/cores/vmcore.2.090615.1558.tgz
     48.0M Jun 16 13:47 /var/crash/cores/vmcore.3.090616.1246.tgz
      504B Jun 15 14:55 /var/crash/info.0
      504B Jun 15 15:46 /var/crash/info.1
      505B Jun 15 15:58 /var/crash/info.2
      505B Jun 16 12:46 /var/crash/info.3
    258.1K Jun 23 16:26 /var/log/chassisd.1.gz
    259.2K Jun 17 08:41 /var/log/chassisd.2.gz
     39.6K Jun 15 13:03 /var/log/dcd.0.gz
      716B Jun 15 13:43 /var/log/install.0.gz
     1669B Jun 13 18:05 /var/log/install.1.gz
     
     ...output_truncated...
     
        4B Jun 27 10:37 /var/tmp/idp_license_info
    263.6M Jun 11 16:25 /var/tmp/jbundle-9.6-20090611.1-domestic.tgz
    263.8M Jun 24 13:44 /var/tmp/jbundle-9.6-20090624.0-domestic.tgz
    268.7M Jun 11 15:14 /var/tmp/jinstall-9.6-20090611.1-domestic-signed.tgz
    268.8M Jun 23 23:04 /var/tmp/jinstall-9.6-20090624.0-domestic-signed.tgz
     22.9K Jun 22 10:33 /var/tmp/memprobe
   1024.0K Jun 16 16:16 /var/tmp/p737a_BIOS_0_A_D.ROM
   1024.0K Jun 16 16:17 /var/tmp/p737f_BIOS_0_0_S.ROM
   Delete these files ? [yes,no] (no) 

2. You can use the following file management operational commands to display information about the directories and files on a Routing Engine, but you must include the chassis and Routing Engine specifiers in the pathname:

   • To display detailed information about a file or directory contents, issue the file list command with the detail option.
   • To display the contents of a file, issue the file show command.
   • To compare the contents of two files, issue the file compare command.

3. In addition to or instead of using the request system storage cleanup command to delete unnecessary files, you can use the following file management commands to delete or move files on a Routing Engine, but you must include the chassis and Routing Engine specifiers in the pathname:

   • To delete files, issue the file delete command.
   • To copy files, issue the file copy command.
   • To rename files, issue the file rename command.
   If you elect to use the request system storage cleanup command to automatically delete files, issue the command without the dry-run option, and then type yes when the CLI prompts to proceed with deleting the candidate files. The following truncated example shows sample output for the request system storage cleanup command:

   root@mysfc0> request system storage cleanup dry-run 
   Currently rotating log files, please wait.
   This operation can take up to a minute.
   
   List of files to delete:
   
            Size Date         Name
        2B Jun 16 12:46 /var/crash/bounds
     80.4M Jun 15 14:56 /var/crash/cores/kernel.0.090615.1455
     80.4M Jun 15 15:46 /var/crash/cores/kernel.1.090615.1546
     
     ...output_truncated...
     
   1024.0K Jun 16 16:16 /var/tmp/p737a_BIOS_0_A_D.ROM
   1024.0K Jun 16 16:17 /var/tmp/p737f_BIOS_0_0_S.ROM
   Delete these files ? [yes,no] (no) yes

4. After sufficient free disk space is available on the Routing Engines on which you are upgrading the Junos OS, attempt the upgrade procedure again.