Configuring Data Path Debugging and Trace Options
Understanding Data Path Debugging for SRX Series Devices
Data path debugging, or end-to-end debugging, support provides tracing and debugging at multiple processing units along the packet-processing path. The packet filter can be executed with minimal impact to the production system.
If your goal is to collect packet captures, we strongly recommend leveraging the Operational Mode Packet Capture introduced in Junos OS Release 19.3R1. See Packet Capture from Operational Mode.
On an SRX Series Firewall, a packet goes through series of events involving different components from ingress to egress processing.
With the data path debugging feature, you can trace and debug (capture packets) at different data points along the processing path. The events available in the packet-processing path are: NP ingress, load-balancing thread (LBT), jexec, packet-ordering thread (POT), and NP egress. You can also enable flow module trace if the security flow trace flag for a certain module is set.
At each event, you can specify any of the four actions (count, packet dump, packet summary, and trace). Data path debugging provides filters to define what packets to capture, and only the matched packets are traced. The packet filter can filter out packets based on logical interface, protocol, source IP address prefix, source port, destination IP address prefix, and destination port.
Data path debugging is supported on SRX4600, SRX5400, SRX5600, and SRX5800.
To enable end-to-end debugging, you must perform the following steps:
Define the capture file and specify the maximum capture size.
Define the packet filter to trace only a certain type of traffic based on the requirement.
Define the action profile specifying the location on the processing path from where to capture the packets (for example, LBT or NP ingress).
Enable the data path debugging.
Capture traffic.
Disable data path debugging.
View or analyze the report.
The packet-filtering behavior for the port and interface options is as follows:
The packet filter traces both IPv4 and IPv6 traffic if only port is specified.
The packet filter traces IPv4, IPV6, and non-IP traffic if only interface is specified.
Packet Capture from Operational Mode
Data path debugging or end-to-end debugging provides tracing and debugging at multiple processing units along the packet-processing path. Packet capture is one of the data path debug function. You can execute the packet capture from the operational mode with minimal impact to the production system without committing the configurations.
You can capture the packets using filters to define what packets to capture. The packet filter can filter out packets based on logical interface, protocol, source IP address prefix, source port, destination IP address prefix, and destination port. You can modify the file name, file type, file size, and capture size of the packet capture output. You can also extend the filters into two filters, and swap the values of filters.
Packet capture from operational mode is supported on SRX4600, SRX5400, SRX5600, and SRX5800.
To capture packets from the operational mode, you must perform the following steps:
- From the operational mode, define the packet filter to
trace the type of traffic based on your requirement using the
request packet-capture startCLI command. See request packet-capture start for the available packet capture filter options. - Capture the required packets.
- You can use either the
request packet-capture stopCLI command to stop the packet capture or after collecting the requested number of packets, the packet capturing stops automatically. - View or analyze the captured packet data report.
Limitations of capturing packets from the operational mode are:
The configuration mode packet capture and the operational mode packet capture cannot coexist.
The operational mode packet capture is a one-time operation and the system does not store the history of this command.
You should use the operational mode packet capture in low rate of traffic flow.
See Also
Understanding Security Debugging Using Trace Options
The Junos OS trace function allows applications to write security debugging information to a file. The information that appears in this file is based on criteria you set. You can use this information to analyze security application issues.
The trace function operates in a distributed manner, with each thread writing to its own trace buffer. These trace buffers are then collected at one point, sorted, and written to trace files. Trace messages are delivered using the InterProcess Communications (IPC) protocol. A trace message has a lower priority than that of control protocol packets such as BGP, OSPF, and IKE, and therefore delivery is not considered to be as reliable.
Understanding Flow Debugging Using Trace Options
For flow trace options, you can define a packet filter using combinations of destination-port, destination-prefix, interface, protocol, source-port, and source-prefix. If the security flow trace flag for a certain module is set, the packet matching the specific packet filter triggers flow tracing and writes debugging information to the trace file.
Debugging the Data Path (CLI Procedure)
Data path debugging is supported on SRX5400, SRX5600, and SRX5800.
To configure the device for data path debugging:
Setting Flow Debugging Trace Options (CLI Procedure)
The following examples display the options you can set
by using security flow traceoptions.
To match the imap destination port for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 destination-port imap
To set the 1.2.3.4 destination IPv4 prefix address for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 destination-prefix 1.2.3.4
To set the fxp0 logical interface for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 interface fxp0
To match the TCP IP protocol for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 protocol tcp
To match the HTTP source port for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 source-port http
To set the 5.6.7.8 IPv4 prefix address for the filter1 packet filter, use the following statement:
[edit] user@host# set security flow traceoptions packet-filter filter1 source-prefix 5.6.7.8
Setting Security Trace Options (CLI Procedure)
Use the following configuration statements to configure security trace options in the CLI configuration editor.
To disable remote tracing, enter the following statement:
[edit] user@host# set security traceoptions no-remote-trace
To write trace messages to a local file, enter the following statement. The system saves the trace file in the /var/log/ directory.
[edit] user@host# set security traceoptions use-local-files
To specify a name for the trace file, enter the following statement. Valid values range from 1 and 1024 characters. The name cannot include spaces, /, or % characters. The default filename is security.
[edit] user@host# set security traceoptions file filename
To specify the maximum number of trace files that can accumulate, enter the following statement. Valid values range from 2 to 1000. The default value is 3.
[edit] user@host# set security traceoptions file files 3
To specify the match criteria that you want the system to use when logging information to the file, enter the following statement. Enter a regular expression. Wildcard (*) characters are accepted.
[edit] user@host# set security traceoptions file match *thread
To allow any user to read the trace file, enter the
world-readablestatement. Otherwise, enter theno-world-readablestatement.[edit] user@host# set security traceoptions file world-readable user@host# set security traceoptions file no-world-readable
To specify the maximum size to which the trace file can grow, enter the following statement. Once the file reaches the specified size, it is compressed and renamed filename0.gz, the next file is named filename1.gz, and so on. Valid values range from 10240 to 1,073,741,824.
[edit] user@host# set security traceoptions file size 10240
To turn on trace options and to perform more than one tracing operation, set the following flags.
[edit] user@host# set security traceoptions flag all user@host# set security traceoptions flag compilation user@host# set security traceoptions flag configuration user@host# set security traceoptions flag routing-socket
To specify the groups that these trace option settings do or do not apply to, enter the following statements:
[edit] user@host# set security traceoptions apply-groups value user@host# set security traceoptions apply-groups-except value
Displaying Log and Trace Files
Enter the monitor start command to display real-time
additions to system logs and trace files:
user@host> monitor start filename
When the device adds a record to the file specified by filename, the record displays on the screen.
For example, if you have configured a system log file named system-log (by including the syslog statement at the [edit
system] hierarchy level), you can enter the monitor start
system-log command to display the records added to the system
log.
To display a list of files that are being monitored, enter the monitor list command. To stop the display of records for a
specified file, enter the monitor stop filename command.
Displaying Output for Security Trace Options
Purpose
Display output for security trace options.
Action
Use the show security traceoptions command
to display the output of your trace files. For example:
[edit] user@host # show security traceoptions file usp_trace user@host # show security traceoptions flag all user@host # show security traceoptions rate-limit 888
The output for this example is as follows:
Apr 11 16:06:42 21:13:15.750395:CID-906489336:FPC-01:PIC-01:THREAD_ID-01:PFE:now update 0x3607edf8df8in 0x3607e8d0 Apr 11 16:06:42 21:13:15.874058:CID-1529687608:FPC-01:PIC-01:THREAD_ID-01:CTRL:Enter Function[util_ssam_handler] Apr 11 16:06:42 21:13:15.874485:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default1: Rate limit changed to 888 Apr 11 16:06:42 21:13:15.874538:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default1: Destination ID set to 1 Apr 11 16:06:42 21:13:15.874651:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default2: Rate limit changed to 888 Apr 11 16:06:42 21:13:15.874832:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default2: Destination ID set to 1 Apr 11 16:06:42 21:13:15.874942:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default3: Rate limit changed to 888 Apr 11 16:06:42 21:13:15.874997:CID-00:FPC-01:PIC-01:THREAD_ID-01:CTRL:default3: Destination ID set to 1
Displaying Multicast Trace Operations
To monitor and display multicast trace operations, enter the mtrace monitor command:
user@host> mtrace monitor
Mtrace query at Apr 21 16:00:54 by 192.1.30.2, resp to 224.0.1.32, qid 2a83aa packet from 192.1.30.2 to 224.0.0.2 from 192.1.30.2 to 192.1.4.1 via group 224.1.1.1 (mxhop=60) Mtrace query at Apr 21 16:00:57 by 192.1.30.2, resp to 224.0.1.32, qid 25dc17 packet from 192.1.30.2 to 224.0.0.2 from 192.1.30.2 to 192.1.4.1 via group 224.1.1.1 (mxhop=60) Mtrace query at Apr 21 16:01:00 by 192.1.30.2, resp to same, qid 20e046 packet from 192.1.30.2 to 224.0.0.2 from 192.1.30.2 to 192.1.4.1 via group 224.1.1.1 (mxhop=60) Mtrace query at Apr 21 16:01:10 by 192.1.30.2, resp to same, qid 1d25ad packet from 192.1.30.2 to 224.0.0.2 from 192.1.30.2 to 192.1.4.1 via group 224.1.1.1 (mxhop=60)
This example displays only mtrace queries. However,
when the device captures an mtrace response, the display
is similar, but the complete mtrace response also appears
(exactly as it is appears in the mtrace from-source command output).
Table 1 summarizes the output fields of the display.
Field |
Description |
|---|---|
|
|
|
IP address of the host issuing the query. |
|
|
|
|
|
|
|
|
|
|
|
|
Displaying a List of Devices
To display a list of devices between the device and a specified
destination host, enter the traceroute command with the
following syntax:
user@host> traceroute host <interface interface-name> <as-number-lookup> <bypass-routing> <gateway address> <inet | inet6> <no-resolve> <routing-instance routing-instance-name> <source source-address> <tos number> <ttl number> <wait seconds>
Table 2 describes
the traceroute command options.
Option |
Description |
|---|---|
|
Sends traceroute packets to the hostname or IP address you specify. |
|
(Optional) Sends the traceroute packets on the interface you specify. If you do not include this option, traceroute packets are sent on all interfaces. |
|
(Optional) Displays the autonomous system (AS) number of each intermediate hop between the device and the destination host. |
|
(Optional) Bypasses the routing tables and sends the traceroute packets only to hosts on directly attached interfaces. If the host is not on a directly attached interface, an error message is returned. Use this option to display a route to a local system through an interface that has no route through it. |
|
(Optional) Uses the gateway you specify to route through. |
|
(Optional) Forces the traceroute packets to an IPv4 destination. |
|
(Optional) Forces the traceroute packets to an IPv6 destination. |
|
(Optional) Suppresses the display of the hostnames of the hops along the path. |
|
(Optional) Uses the routing instance you specify for the traceroute. |
|
(Optional) Uses the source address that you specify, in the traceroute packet. |
|
(Optional) Sets the type-of-service (TOS)
value in the IP header of the traceroute packet. Specify a value from |
|
(Optional) Sets the time-to-live (TTL) value
for the traceroute packet. Specify a hop count from |
|
(Optional) Sets the maximum time to wait for a response. |
To quit the traceroute command, press Ctrl-C.
The following is sample output from a traceroute command:
user@host> traceroute host2
traceroute to 173.24.232.66 (172.24.230.41), 30 hops max, 40 byte packets 1 173.18.42.253 (173.18.42.253) 0.482 ms 0.346 ms 0.318 ms 2 host4.site1.net (173.18.253.5) 0.401 ms 0.435 ms 0.359 ms 3 host5.site1.net (173.18.253.5) 0.401 ms 0.360 ms 0.357 ms 4 173.24.232.65 (173.24.232.65) 0.420 ms 0.456 ms 0.378 ms 5 173.24.232.66 (173.24.232.66) 0.830 ms 0.779 ms 0.834 ms
The fields in the display are the same as those displayed by the J-Web traceroute diagnostic tool.
Example: Configuring End-to-End Debugging on SRX Series Device
This example shows how to configure and enable end-to-end debugging on an SRX Series Firewall with an SRX5K-MPC.
Requirements
This example uses the following hardware and software components:
SRX5600 device with an SRX5K-MPC installed with 100-Gigabit Ethernet CFP transceiver
Junos OS Release 12.1X47-D15 or later for SRX Series Firewalls
Before you begin:
No special configuration beyond device initialization is required before configuring this feature.
Overview
Data path debugging enhances troubleshooting capabilities by providing tracing and debugging at multiple processing units along the packet-processing path. With the data path debugging feature, you can trace and debug (capture packets) at different data points along the processing path. At each event, you can specify an action (count, packet dump, packet summary, and trace) and you can set filters to define what packets to capture.
In this example, you define a traffic filter, and then you apply an action profile. The action profile specifies a variety of actions on the processing unit. The ingress and egress are specified as locations on the processing path to capture the data for incoming and outgoing traffic.
Next, you enable data path debugging in operational mode, and finally you view the data capture report.
Data path debugging is supported on SRX1400, SRX3400, SRX3600, SRX5400, SRX5600, and SRX5800.
Configuration
Procedure
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security datapath-debug traceoptions file e2e.trace size 10m set security datapath-debug capture-file e2e.pcap format pcap set security datapath-debug maximum-capture-size 1500 set security datapath-debug capture-file files 10 set security datapath-debug action-profile profile-1 preserve-trace-order set security datapath-debug action-profile profile-1 record-pic-history set security datapath-debug action-profile profile-1 event np-ingress trace set security datapath-debug action-profile profile-1 event np-ingress count set security datapath-debug action-profile profile-1 event np-ingress packet-summary set security datapath-debug action-profile profile-1 event np-egress trace set security datapath-debug action-profile profile-1 event np-egress count set security datapath-debug action-profile profile-1 event np-egress packet-summary
Step-by-Step Procedure
The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the Junos OS CLI User Guide .
To configure data path debugging:
Edit the security datapath debugging option for the multiple processing units along the packet-processing path:
[edit] user@host# edit security datapath-debug
Enable the capture file, file format, file size, and the number of files.
[edit security datapath-debug] user@host# set traceoptions file e2e.trace size 10m user@host# set capture-file e2e.pcap format pcap; user@host# set maximum-capture-size 1500 user@host# set capture-file files 10
Configure action profile, event type, and actions for the action profile.
[edit security datapath-debug] user@host# set action-profile profile-1 preserve-trace-order user@host# set action-profile profile-1 record-pic-history user@host# set action-profile profile-1 event np-ingress trace user@host# set action-profile profile-1 event np-ingress count user@host# set action-profile profile-1 event np-ingress packet-summary user@host# set action-profile profile-1 event np-egress trace user@host# set action-profile profile-1 event np-egress count user@host# set action-profile profile-1 event np-egress packet-summary
Results
From configuration mode, confirm your configuration
by entering the show security datapath-debug command. If
the output does not display the intended configuration, repeat the
configuration instructions in this example to correct it.
traceoptions {
file e2e.trace size 10m;
}
capture-file e2e.pcap format pcap;
maximum-capture-size 1500;
capture-file files 10;
action-profile {
profile-1 {
preserve-trace-order;
record-pic-history;
event np-ingress {
trace;
packet-summary;
packet-dump;
}
event np-egress {
trace;
packet-summary;
packet-dump;
}
}
}
If you are done configuring the device, enter commit from configuration mode.
Enabling Data Path Debugging
Procedure
Step-by-Step Procedure
After configuring data path debugging, you must start the process on the device from operational mode.
Enable data path debugging.
user@host> request security datapath-debug capture start
datapath-debug capture started on file datapcap
Before you verify the configuration and view the reports, you must disable data path debugging.
user@host> request security datapath-debug capture stop
datapath-debug capture succesfully stopped, use show security datapath-debug capture to view
Note:You must stop the debug process after you have finished capturing the data. If you attempt to open the captured files without stopping the debug process, the files obtained cannot be opened through any third-party software (for example, tcpdump and wireshark).
Verification
Confirm that the configuration is working properly.
Verifying Data Path Debug Packet Capture Details
Purpose
Verify the data captured by enabling the data path debugging configuration.
Action
From operational mode, enter the show security
datapath-debug capture command.
Packet 8, len 152: (C2/F2/P0/SEQ:57935:np-ingress) 00 10 db ff 10 02 00 30 48 83 8d 4f 08 00 45 00 00 54 00 00 40 00 40 01 9f c7 c8 07 05 69 c8 08 05 69 08 00 91 1f 8f 03 2a a2 ae 66 85 53 8c 7d 02 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 Packet 9, len 152: (C2/F2/P0/SEQ:57935:np-egress) 00 30 48 8d 1a bf 00 10 db ff 10 03 08 00 45 00 00 54 00 00 40 00 3f 01 a0 c7 c8 07 05 69 c8 08 05 69 08 00 91 1f 8f 03 2a a2 ae 66 85 53 8c 7d 02 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37....
For brevity, the show command output is truncated
to display only a few samples. Additional samples have been replaced
with ellipses (...).
To view the results, from CLI operational mode, access the local
UNIX shell and navigate to the directory /var/log/<file-name>. The result can be read by
using the tcpdump utility.
user@host>start shell %tcpdump -nr/var/log/e2e.pcap
21:50:04.288767 C0/F3 event:1(np-ingress) SEQ:1 IP 192.168.14.2 > 192.168.13.2: ICMP echo request, id 57627, seq 0, length 64 21:50:04.292590 C0/F3 event:2(np-egress) SEQ:1 IP 192.168.14.2 > 192.168.13.2: ICMP echo request, id 57627, seq 0, length 64 1:50:04.295164 C0/F3 event:1(np-ingress) SEQ:2 IP 192.168.13.2 > 192.168.14.2: ICMP echo reply, id 57627, seq 0, length 64 21:50:04.295284 C0/F3 event:2(np-egress) SEQ:2 IP 192.168.13.2 > 192.168.14.2: ICMP echo reply, id 57627, seq 0, length 64
If you are finished with troubleshooting the data path
debugging, remove all traceoptions (not limited to flow
traceoptions) and the complete data path debug configuration, including
the data path debug configuration for packet capturing (packet-dump),
which needs to be started/stopped manually. If any part of the debugging
configuration remains active, it will continue to use the resources
of the device (CPU/memory).