Troubleshooting the System

You can use log commands to discover and isolate problems with the system. For information about using the log commands, see the JunosE System Event Logging Reference Guide. Juniper Networks Customer Service can use core dump files to troubleshoot line module and SRP module failures.

Creating Core Dump Files

You can enable the system to create a core dump file if a module fails. You can choose to send the core dump file to an FTP server or save the file to NVS. Juniper Networks Customer Service can then access the core dump file and analyze it to determine what went wrong. Local core dumps—stored in NVS—are enabled by default. You can enable the core dump from Boot mode or Global Configuration mode.

Caution: Create a core dump file only under the direction of Juniper Networks Customer Service. Network function can be disrupted if you create a core dump file while the system is running in a network.

On the E120 and E320 routers, the failure of some components on a line module generates multiple core dumps to provide more complete information about system state at the time of the failure. Other E Series routers generate only a single core dump for line module failures. When you contact Juniper Networks Customer Service for assistance, send all of the generated core dump files.

Boot Mode

To enable the core dump from Boot mode:

Access Boot mode by reloading the SRP module; then press the mb key sequence (case insensitive) during the countdown.
Specify where the system should transfer the core dump file.
Set the IP address and mask of the system interface over which you want to send the core dump file.
Specify the gateway through which the system sends the core dump file to the FTP server.
(Optional) Set a username and password for FTP access to the server where you transferred the core dump file.
Reload the operating system.
Example
:boot##exception dump 192.168.56.7 CORE_DUMPS :boot##exception protocol ftp user_name user_password :boot##exception gateway 192.168.12.3 :boot##exception source 10.10.33.8 255.255.255.0 :boot##reload

Global Configuration Mode

To enable the core dump from Global Configuration mode:

Access Global Configuration mode.
Specify where the system should transfer the core dump file.
Set the IP address and mask of the system interface over which you want to send the core dump file.
Specify the gateway through which the system sends the core dump file to the FTP server.
(Optional) Set a username and password for FTP access to the server where you want to transfer the core dump file.
(Optional) View parameters associated with creating a core dump file.
Example
host1(config)#exception dump 192.168.56.7 CORE_DUMPS host1(config)#exception protocol ftp username userpassword host1(config)#exception gateway 192.168.12.3 host1(config)#exception source 10.10.33.8 255.255.255.0 host1(config)#reload

exception dump

Use to specify where the system should transfer the core dump file.
- To send the file to an FTP server, enter the IP address of the FTP server and the name of the directory on the server to which the system will transfer the file.
- To send the core dump file to NVS memory, use the local keyword.
- Local core dumps—stored in NVS—are enabled by default.
Example
host1(config)#exception dump 192.168.56.7 CORE_DUMPS
Use the no version to disable the core dump.
See exception dump.

exception gateway

Use to specify the gateway through which the system sends the core dump file to the FTP server.
Example
host1(config)#exception gateway 10.10.1.15
Use the no version to return the value to its default (null).
See exception gateway.

exception protocol ftp

Use to set a username and password for FTP access to the server where you transferred a core dump file. The default settings are the username anonymous and no password.
Specify the number 8 before the username and before the password to encrypt these values. By default, the username and password are not encrypted.
Example
host1(config)#exception protocol ftp 8 user_core 8 user_password
Use the no version to restore the default settings.
See exception protocol ftp.

exception source

Use to set the IP address and mask of the system interface over which you want to send the core dump file to the FTP server.
You can optionally include an IP address mask.
Example
host1(config)#exception source 192.168.1.33 255.255.255.0
Use the no version to return the value to its default, null.
See exception source.

reload

Use to reload the software on the router immediately.
Reloads the system software (.rel) file and the configuration (.cnf) file.
Reloading the standby SRP causes high availability to be temporarily disabled until the standby SRP reloads and resynchronizes with the active SRP.
Example
host1#reload
There is no no version.
See reload.

show exception dump

Use to display the parameters associated with the core dump operation.
Field descriptions
- Dump host IP address—Address of the host where the system is configured to transfer the dump file
- Dump directory—Name of directory on the host where the system is configured to transfer the dump file
- Dump protocol—Protocol used to send the core dump file; currently only FTP is supported
- User name—Name configured for access to the core dump file on the FTP server
- Password—Password configured for access to the core dump file on the FTP server
- Interface IP address—Address of the system interface configured to send the core dump file
- Interface netmask—Mask of the system interface configured to send the core dump file
- Gateway IP address—Address of gateway configured between the system and the FTP server

Example

host1#show exception dump
Dump host IP address: 192.168.56.7
Dump directory:CORE_DUMPS/
Dump protocol: FTP
User name: user_name
Password: user_password
Interface IP address:
Interface netmask:
Gateway IP address:

See show exception dump.

Managing Core Dump Files

When a core dump occurs on a redundant SRP and you have the router configured to store network core dumps, the SRP that experiences the trouble retains the management Ethernet port to perform the core dump. This prevents the standby SRP from taking over operations until the core dump is complete.

When a router uses local NVS to store a core dump, the SRP does not need the management Ethernet port. However, because of the immense size of local core dump files, using NVS to store core dumps is not practical.

The SRP-120 available on the E120 router and the SRP-320 available on the E120 and E320 routers has a second NVS card which is dedicated to storing core dump files.

The core dump monitor eliminates the impact that core dumps may have on redundant routers by allowing you to manage core dump files in NVS. The core dump monitor allows you to automatically transfer core dump files from NVS to an FTP server location for storage. The core dump monitor can also automatically delete transferred core dump files.

The core dump monitor attempts to delete transferred files when all of the following conditions have been met:

The router attempts to write a core dump file to NVS.
NVS contains insufficient space to hold the new core dump file.
The core dump files have already been transferred from NVS to an FTP server location using the automatic core dump monitor transfer process.

Only those core dump files that have already been transferred from NVS are considered for deletion. Of those, the oldest files are deleted first, and the router generates a log message for each core dump file it deletes.

Note: If the router NVS does not contain sufficient space to hold a new core dump file even after deleting all possible core dump files, the core dump fails and the router generates a log message for this condition.

Enabling and Disabling the Core Dump Monitor

The core dump monitor is disabled by default. To enable the core dump monitor, use the exception monitor command. Use the no version of this command to disable the core dump monitor.

exception monitor

Use to enable the router core dump monitor and specify the location to which you want the router to transfer core dump files.
To send the file to an FTP server, enter the IP address of the FTP server and the name of the directory on the server to which the system will transfer the file.
Enabling the core dump monitor specifies that future core dump files be saved to NVS. See exception dump command for details.
Example
host1(config)#exception monitor 192.168.56.7 CORE_DUMPS

Use the no version to disable the core dump monitor.

Note: You can use exception protocol ftp command to assign a username and password to the targeted FTP server. If you choose not to define a username or password, the router uses the values of “ anonymous” and “ null,” respectively.

See exception monitor.

Specifying the Core Dump Monitor Interval

To specify the length of time that the router waits between checking for core dump files, use the exception monitor interval command. Use the no version of this command to revert the core dump monitor interval to its default value of 60 minutes (1 hour).

exception monitor interval

Use to specify the interval (in minutes) at which you want the router to check NVS for core dump files.
Example
host1(config)#exception monitor interval 1000
Use the no version to revert the core dump monitor interval to its default value, 60 minutes.
See exception monitor interval.

Viewing Core Dump Monitor Status

To view information about core dump monitor status and configuration, use the show exception monitor command.

show exception monitor

Use to display information about the core dump monitor status and configuration.
Field descriptions
- Core dump monitor—Status (enabled or disabled) of the core dump monitor
- Next dump monitor check time—Time at which the core dump monitor will next check for any new core dump files
- Host—IP address of the FTP host on which the core dump monitor saves core dump files
- Directory—Directory or directory path on the host to where the core dump files are located
- Core dump monitor interval—Time interval (in minutes) at which the core dump monitor checks for any new core dump files
- Files on flash which have been transferred—A list of core dump files in the router NVS that have already been transferred to the FTP host
- Files on flash which have not been transferred—A list of core dump files in the router NVS that have not yet been transferred to the FTP host

Example

host1#show exception monitor
Core dump monitor is enabled
Next dump monitor check time: WED AUG 16 2003 15:50:38 UTC
Host: 10.10.120.99
Directory: monitor
Core dump monitor interval(minutes): 10

Files on flash which have been transferred
------------------------------------------
standby:OC12Atm(P2)_5_IC_ERX-10-16-5b_09_15_2002_11_59.dmp
SRP-5GPlus_1_SC_tImBo-lAb-3_09_18_2002_19_39.dmp

Files on flash which have not been transferred
----------------------------------------------
standby:SRP-10Ge_1_SC_ERX-10-24-36_09_24_2002_11_04.dmp
OC12-SERVER_5_FC1_E_ERX-10-24-36_03_28_2003_12_44.dmp
E3_1_IC_ERX-10-0f-ab_10_08_2002_16_10.dmp

See show exception monitor.

Accessing the Core Dump File

If a module fails and saves a core dump file to NVS memory (which can take several minutes), and you have not configured the Core Dump Monitor for automatic transfer, you must transfer the file to a network host before it can be examined. You can transfer the core dump file when the module is back online or has assumed a redundant status. For information about the status of modules, see ERX Hardware Guide, Chapter 9, Troubleshooting. To transfer the core dump file to a network host, use the copy command.

In a system with two SRP modules, the following behavior applies if you have configured the SRP modules to save core dump files to an FTP server:

If the primary SRP module fails, it saves the core dump file to the FTP server before the standby SRP module assumes control.
If the standby SRP module fails, it must save the core dump file to NVS because it has no access to any configured network host.

The show version command output indicates the failed SRP module state as not responding during the save process. Consequently, when the failed SRP module recovers and assumes the role of redundant module, the show version command output indicates the SRP module state as standby and displays output for the standby SRP. The standby SRP can notify the primary SRP during a core dump. Output from the show version command displays core dumping for the Standby SRP.

If the standby SRP boot image encounters a problem loading the diagnostics or operational image, the state of the standby SRP appears as disabled (image error). When standby SRP diagnostics encounter a test failure, the primary SRP is notified and the state is set to hardware error.

You can now transfer the core dump file to a network host for examination. For example, to transfer the file SRP_1_SC_05_24_2000_02_20.dmp from NVS of the failed SRP module to the host server1, enter the following command:

host1#copy SRP_1_SC_05_24_2000_02_20.dmp host:/public/server1/SRP-5G_1_SC_05_24_2000_02_20.dmp

copy

Use to copy a core dump file.
You cannot use wildcards.
You can copy core dump files only to network locations.
You cannot create or copy over files generated by the system; however, you can copy such files to an unreserved filename.
Example
host1#copy fault.dmp host:/public/server1/fault.dmp
There is no no version.
See copy.

Capturing and Writing Core Dumps

You can capture and write a core dump to a file for an active or a standby SRP module or the line modules. You can store the file on the file system or on a network host. The SRP core dump files are stored on the respective SRP flash memory. The line module core dump files are stored on the active SRP flash memory at the instance of the core dump event. The core dump files are not synchronized between the active and the standby SRP module. You can use the resulting information to help diagnose a problem or to verify whether the core settings are correct (primarily for the network settings).

write core

Use to reboot the active SRP module, the standby SRP module, or the module in a specified slot, and write the core dump to a file.
If you specify the force keyword, you are prompted for confirmation to reboot when the router is in a state that can lead to loss of configuration data or NVS corruption.
Note: The force keyword enables you to specify a slot only if that slot is an SRP module slot.
If you do not specify a reason, Write Core is the default reason recorded in the reboot history.
Example 1—Prompts for confirmation to reboot
host1#write core force
Example 2—Reboots the module in slot 7 and writes a core memory file
host1#write core slot 7
There is no no version.
See write core.

Understanding the Core Dump File

The dump file indicates which module has failed by referencing that module’s hardware slot number. The hardware slot number is the slot number designation on the systems’s backplane. This slot number is different from the chassis slot number that appears on the front of the chassis and in screen displays (for example, in the display resulting if you issue the show version command).

Table 38 shows how the chassis slot numbers relate to the hardware slot numbers.

Table 38: Chassis Slot Numbers Versus Hardware Slot Numbers

Slot Number on Chassis	ERX7xx Model Hardware Slot Number	ERX14xx Model Hardware Slot Number	E320 Model Hardware Slot Number
0	1	0	16
1	3	1	17
2	4	2	18
3	5	3	19
4	6	4	20
5	7	5	21
6	8	7	8
7	9	9	10
8	–	10	9
9	–	11	12
10	–	12	13
11	–	13	25
12	–	14	26
13	–	15	27
14	–	–	28
15	–	–	29
16	–	–	30

Tracking IP Prefix Reachability

You can use the track command to define an IPv4 prefix object and track its reachability. The show track command displays the tracked information for any specified objects.

show track

Use to display tracking details for the object you specify.
Field descriptions
- Track—Name of the object being tracked
- IP Route—IP prefix being tracked
- Virtual router—Virtual router on which the object resides
- First-hop interface—Outgoing interface to reach the prefix
- changes—Number of times the object has changed state
- Tracked by—Application that is doing the tracking

Example

host1(config)#show track ERX_Bangalore

Track ERX_Bangalore
 IP Route 1.1.1.0 255.255.255.0 reachability
 in virtual router 1
 Reachability is Up
 First-hop interface is FastEthernet3/0
 2 change(s)
Tracked by:
 Vrrp in virtual router 1

See show track.

show track brief

Use to display a one-line summary of all objects being tracked.
Field descriptions
- Object—Name of the object being tracked
- Type—Type of object being tracked
- Parameter—Parameter type being tracked
- Value—State of the object being tracked

Example

host1(config)#show track brief

Object  Type             Parameter        Value
ERX-WF  IP-route         reachability     Up
ERX-BNG IP-route         reachability     Up

See show track brief.

track

Use to create an IPv4 object and to track its reachability.
The name of the object must be unique for the chassis.
Use the vrf keyword to specify the VRF on which the IP prefix resides.
Example
host1(config)#track ERX_Bangalore vrf VR1 ip-route 10.10.24.6 255.255.0.0 reachability
Use the no version to delete the object and stop tracking for that object.
See track.

Gathering Information for Customer Support

When you report a problem with your router, customer support personnel from the Juniper Networks Technical Assistance Center (JTAC) may request that you issue the show tech-support command. This command was created to help streamline the information-gathering process by providing a large amount of router information from one command and avoiding the need to access certain diagnostic commands.

The show tech-support command functions like any other show command, and you can issue this command the same way you issue any other show commands on the router. This means that you can redirect the output from the command to a file. For information about redirecting show command output, see Redirection of show Command Output.

Another command that customer support personnel might ask you to use is the tech-support encoded-string command. Customer support will provide you with an encoded string of commands that this command then executes.

tech-support encoded-string

Use to execute an encoded command string provided by Juniper Networks customer support personnel.
This command requires privilege level 15 access.
Optionally, specify a slot number on the router.
Optionally, specify a reliable or fast connection type; fast does not work under some conditions. The default connection type is reliable.
Example 1
host1(config)#tech-support encoded-string debug 1
Example 2
host1(config)#tech-support slot 0 connection fast encoded-string debug1
There is no no version.
See tech-support encoded-string.

show tech-support

Use to display technical support information used by Juniper Networks customer support personnel to assist in troubleshooting the router.

Example

host1#show tech-support
Show Technical Support
---------------------------------------------------------------------------

System Name     : host1
Time            : THU JUL 15 2004 17:12:48 UTC
System up since : WED JUN 30 2004 16:07:51 UTC
Software release: 1088523900
Boot Flags      : 0x100663296
Slot Number     : 0
Serial Number   : 7100170293
Assembly Number : 3400003701
Assembly Rev    : A07
Description     :

Command List:
CLI:show version
CLI:show boot
CLI:show hardware
CLI:show redundancy
CLI:show environment
CLI:show users detail
CLI:show utilization
CLI:show process cpu
CLI:show process memory
....

See show tech-support.

Slot Number on Chassis	ERX7xx Model Hardware Slot Number	ERX14xx Model Hardware Slot Number	E320 Model Hardware Slot Number
0	1	0	16
1	3	1	17
2	4	2	18
3	5	3	19
4	6	4	20
5	7	5	21
6	8	7	8
7	9	9	10
8	–	10	9
9	–	11	12
10	–	12	13
11	–	13	25
12	–	14	26
13	–	15	27
14	–	–	28
15	–	–	29
16	–	–	30

Slot Number on Chassis	ERX7xx Model Hardware Slot Number	ERX14xx Model Hardware Slot Number	E320 Model Hardware Slot Number
0	1	0	16
1	3	1	17
2	4	2	18
3	5	3	19
4	6	4	20
5	7	5	21
6	8	7	8
7	9	9	10
8	–	10	9
9	–	11	12
10	–	12	13
11	–	13	25
12	–	14	26
13	–	15	27
14	–	–	28
15	–	–	29
16	–	–	30

Slot Number on Chassis	ERX7xx Model Hardware Slot Number	ERX14xx Model Hardware Slot Number	E320 Model Hardware Slot Number
0	1	0	16
1	3	1	17
2	4	2	18
3	5	3	19
4	6	4	20
5	7	5	21
6	8	7	8
7	9	9	10
8	–	10	9
9	–	11	12
10	–	12	13
11	–	13	25
12	–	14	26
13	–	15	27
14	–	–	28
15	–	–	29
16	–	–	30