Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Configuring Data Path Debugging and Trace Options

 

Understanding Data Path Debugging for SRX Series Devices

Data path debugging, or end-to-end debugging, support provides tracing and debugging at multiple processing units along the packet-processing path. The packet filter can be executed with minimal impact to the production system.

If your goal is to collect packet captures, we strongly recommend leveraging the Operational Mode Packet Capture introduced in Junos OS Release 19.3R1. See Packet Capture from Operational Mode.

On an SRX Series device, a packet goes through series of events involving different components from ingress to egress processing.

With the data path debugging feature, you can trace and debug (capture packets) at different data points along the processing path. The events available in the packet-processing path are: NP ingress, load-balancing thread (LBT), jexec, packet-ordering thread (POT), and NP egress. You can also enable flow module trace if the security flow trace flag for a certain module is set.

At each event, you can specify any of the four actions (count, packet dump, packet summary, and trace). Data path debugging provides filters to define what packets to capture, and only the matched packets are traced. The packet filter can filter out packets based on logical interface, protocol, source IP address prefix, source port, destination IP address prefix, and destination port.

Data path debugging is supported on SRX1400, SRX3400, SRX3600, SRX4600, SRX5400, SRX5600, and SRX5800.

To enable end-to-end debugging, you must perform the following steps:

  1. Define the capture file and specify the maximum capture size.
  2. Define the packet filter to trace only a certain type of traffic based on the requirement.
  3. Define the action profile specifying the location on the processing path from where to capture the packets (for example, LBT or NP ingress).
  4. Enable the data path debugging.
  5. Capture traffic.
  6. Disable data path debugging.
  7. View or analyze the report.

The packet-filtering behavior for the port and interface options is as follows:

  • The packet filter traces both IPv4 and IPv6 traffic if only port is specified.

  • The packet filter traces IPv4, IPV6, and non-IP traffic if only interface is specified.

Packet Capture from Operational Mode

Data path debugging or end-to-end debugging provides tracing and debugging at multiple processing units along the packet-processing path. Packet capture is one of the data path debug function. You can execute the packet capture from the operational mode with minimal impact to the production system without committing the configurations.

You can capture the packets using filters to define what packets to capture. The packet filter can filter out packets based on logical interface, protocol, source IP address prefix, source port, destination IP address prefix, and destination port. You can modify the file name, file type, file size, and capture size of the packet capture output. You can also extend the filters into two filters, and swap the values of filters.

To capture packets from the operational mode, you must perform the following steps:

  1. From the operational mode, define the packet filter to trace the type of traffic based on your requirement using the request packet-capture start CLI command. See request packet-capture start for the available packet capture filter options.
  2. Capture the required packets.
  3. You can use either the request packet-capture stop CLI command to stop the packet capture or after collecting the requested number of packets, the packet capturing stops automatically.
  4. View or analyze the captured packet data report.

Limitations of capturing packets from the operational mode are:

  1. The configuration mode packet capture and the operational mode packet capture cannot coexist.
  2. The operational mode packet capture is a one-time operation and the system does not store the history of this command.
  3. You should use the operational mode packet capture in low rate of traffic flow.

Understanding Security Debugging Using Trace Options

The Junos OS trace function allows applications to write security debugging information to a file. The information that appears in this file is based on criteria you set. You can use this information to analyze security application issues.

The trace function operates in a distributed manner, with each thread writing to its own trace buffer. These trace buffers are then collected at one point, sorted, and written to trace files. Trace messages are delivered using the InterProcess Communications (IPC) protocol. A trace message has a lower priority than that of control protocol packets such as BGP, OSPF, and IKE, and therefore delivery is not considered to be as reliable.

Understanding Flow Debugging Using Trace Options

For flow trace options, you can define a packet filter using combinations of destination-port, destination-prefix, interface, protocol, source-port, and source-prefix. If the security flow trace flag for a certain module is set, the packet matching the specific packet filter triggers flow tracing and writes debugging information to the trace file.

Debugging the Data Path (CLI Procedure)

Data path debugging is supported on SRX1400, SRX3400, SRX3600, SRX5400, SRX5600, and SRX5800.

To configure the device for data path debugging:

  1. Specify the following request command to set the data path debugging for the multiple processing units along the packet-processing path:

  2. Specify the trace options for data path-debug using the following command:

  3. Using the request security packet-filter command, you can set the packet filter to specify the related packets to perform data path-debug action. A maximum of four filters are supported at the same time. For example, the following command sets the first packet-filter:

  4. Using the request security action-profile command, you can set the action for the packet match for a specified filter. Only the default action profile is supported, which is the trace option for network processor ezchip ingress, ezchip egress, spu.lbt, and spu.pot:

Setting Flow Debugging Trace Options (CLI Procedure)

The following examples display the options you can set by using security flow traceoptions.

  • To match the imap destination port for the filter1 packet filter, use the following statement:

  • To set the 1.2.3.4 destination IPv4 prefix address for the filter1 packet filter, use the following statement:

  • To set the fxp0 logical interface for the filter1 packet filter, use the following statement:

  • To match the TCP IP protocol for the filter1 packet filter, use the following statement:

  • To match the HTTP source port for the filter1 packet filter, use the following statement:

  • To set the 5.6.7.8 IPv4 prefix address for the filter1 packet filter, use the following statement:

Setting Security Trace Options (CLI Procedure)

Use the following configuration statements to configure security trace options in the CLI configuration editor.

  • To disable remote tracing, enter the following statement:

  • To write trace messages to a local file, enter the following statement. The system saves the trace file in the /var/log/ directory.

  • To specify a name for the trace file, enter the following statement. Valid values range from 1 and 1024 characters. The name cannot include spaces, /, or % characters. The default filename is security.

  • To specify the maximum number of trace files that can accumulate, enter the following statement. Valid values range from 2 to 1000. The default value is 3.

  • To specify the match criteria that you want the system to use when logging information to the file, enter the following statement. Enter a regular expression. Wildcard (*) characters are accepted.

  • To allow any user to read the trace file, enter the world-readable statement. Otherwise, enter the no-world-readable statement.

  • To specify the maximum size to which the trace file can grow, enter the following statement. Once the file reaches the specified size, it is compressed and renamed filename0.gz, the next file is named filename1.gz, and so on. Valid values range from 10240 to 1,073,741,824.

  • To turn on trace options and to perform more than one tracing operation, set the following flags.

  • To specify the groups that these trace option settings do or do not apply to, enter the following statements:

Displaying Log and Trace Files

Enter the monitor start command to display real-time additions to system logs and trace files:

When the device adds a record to the file specified by filename, the record displays on the screen. For example, if you have configured a system log file named system-log (by including the syslog statement at the [edit system] hierarchy level), you can enter the monitor start system-log command to display the records added to the system log.

To display a list of files that are being monitored, enter the monitor list command. To stop the display of records for a specified file, enter the monitor stop filename command.

Displaying Output for Security Trace Options

Purpose

Display output for security trace options.

Action

Use the show security traceoptions command to display the output of your trace files. For example:

The output for this example is as follows:

Displaying Multicast Trace Operations

To monitor and display multicast trace operations, enter the mtrace monitor command:

This example displays only mtrace queries. However, when the device captures an mtrace response, the display is similar, but the complete mtrace response also appears (exactly as it is appears in the mtrace from-source command output).

Table 1 summarizes the output fields of the display.

Table 1: CLI mtrace monitor Command Output Summary

Field

Description

Mtrace operation-type at time-of-day

  • operation-type—Type of multicast trace operation: query or response.

  • time-of-day—Date and time the multicast trace query or response was captured.

by

IP address of the host issuing the query.

resp to address

address—Response destination address.

qid qid

qid—Query ID number.

packet from source to destination

  • source—IP address of the source of the query or response.

  • destination—IP address of the destination of the query or response.

from source to destination

  • source—IP address of the multicast source.

  • destination—IP address of the multicast destination.

via group address

address—Group address being traced.

mxhop=number

number—Maximum hop setting.

Displaying a List of Devices

To display a list of devices between the device and a specified destination host, enter the traceroute command with the following syntax:

Table 2 describes the traceroute command options.

Table 2: CLI traceroute Command Options

Option

Description

host

Sends traceroute packets to the hostname or IP address you specify.

interface interface-name

(Optional) Sends the traceroute packets on the interface you specify. If you do not include this option, traceroute packets are sent on all interfaces.

as-number-lookup

(Optional) Displays the autonomous system (AS) number of each intermediate hop between the device and the destination host.

bypass-routing

(Optional) Bypasses the routing tables and sends the traceroute packets only to hosts on directly attached interfaces. If the host is not on a directly attached interface, an error message is returned.

Use this option to display a route to a local system through an interface that has no route through it.

gateway address

(Optional) Uses the gateway you specify to route through.

inet

(Optional) Forces the traceroute packets to an IPv4 destination.

inet6

(Optional) Forces the traceroute packets to an IPv6 destination.

no-resolve

(Optional) Suppresses the display of the hostnames of the hops along the path.

routing-instance routing-instance-name

(Optional) Uses the routing instance you specify for the traceroute.

source address

(Optional) Uses the source address that you specify, in the traceroute packet.

tos number

(Optional) Sets the type-of-service (TOS) value in the IP header of the traceroute packet. Specify a value from 0 through 255.

ttl number

(Optional) Sets the time-to-live (TTL) value for the traceroute packet. Specify a hop count from 0 through 128.

wait seconds

(Optional) Sets the maximum time to wait for a response.

To quit the traceroute command, press Ctrl-C.

The following is sample output from a traceroute command:

The fields in the display are the same as those displayed by the J-Web traceroute diagnostic tool.

Example: Configuring End-to-End Debugging on SRX Series Device

This example shows how to configure and enable end-to-end debugging on an SRX Series device with an SRX5K-MPC.

Requirements

This example uses the following hardware and software components:

  • SRX5600 device with an SRX5K-MPC installed with 100-Gigabit Ethernet CFP transceiver

  • Junos OS Release 12.1X47-D15 or later for SRX Series devices

Before you begin:

No special configuration beyond device initialization is required before configuring this feature.

Overview

Data path debugging enhances troubleshooting capabilities by providing tracing and debugging at multiple processing units along the packet-processing path. With the data path debugging feature, you can trace and debug (capture packets) at different data points along the processing path. At each event, you can specify an action (count, packet dump, packet summary, and trace) and you can set filters to define what packets to capture.

In this example, you define a traffic filter, and then you apply an action profile. The action profile specifies a variety of actions on the processing unit. The ingress and egress are specified as locations on the processing path to capture the data for incoming and outgoing traffic.

Next, you enable data path debugging in operational mode, and finally you view the data capture report.

Note

Data path debugging is supported on SRX1400, SRX3400, SRX3600, SRX5400, SRX5600, and SRX5800.

Configuration

CLI Quick Configuration

To quickly configure this example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the Junos OS CLI User Guide .

To configure data path debugging:

  1. Edit the security datapath debugging option for the multiple processing units along the packet-processing path:
  2. Enable the capture file, file format, file size, and the number of files.
  3. Configure action profile, event type, and actions for the action profile.

Results

From configuration mode, confirm your configuration by entering the show security datapath-debug command. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

If you are done configuring the device, enter commit from configuration mode.

Enabling Data Path Debugging

Step-by-Step Procedure

After configuring data path debugging, you must start the process on the device from operational mode.

  1. Enable data path debugging.
  2. Before you verify the configuration and view the reports, you must disable data path debugging.
    Note

    You must stop the debug process after you have finished capturing the data. If you attempt to open the captured files without stopping the debug process, the files obtained cannot be opened through any third-party software (for example, tcpdump and wireshark).

Verification

Confirm that the configuration is working properly.

Verifying Data Path Debug Packet Capture Details

Purpose

Verify the data captured by enabling the data path debugging configuration.

Action

From operational mode, enter the show security datapath-debug capture command.

For brevity, the show command output is truncated to display only a few samples. Additional samples have been replaced with ellipses (...).

To view the results, from CLI operational mode, access the local UNIX shell and navigate to the directory /var/log/<file-name>. The result can be read by using the tcpdump utility.

Note

If you are finished with troubleshooting the data path debugging, remove all traceoptions (not limited to flow traceoptions) and the complete data path debug configuration, including the data path debug configuration for packet capturing (packet-dump), which needs to be started/stopped manually. If any part of the debugging configuration remains active, it will continue to use the resources of the device (CPU/memory).