Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

Monitoring and Troubleshooting

SUMMARY This section describes the network monitoring and troubleshooting features of Junos OS.

Pinging Hosts

Purpose

Use the CLI ping command to verify that a host can be reached over the network. This command is useful for diagnosing host and network connectivity problems. The switch sends a series of Internet Control Message Protocol (ICMP) echo (ping) requests to a specified host and receives ICMP echo responses.

Action

To use the ping command to send four requests (ping count) to host3:

Sample Output

command-name

Meaning

  • The ping results show the following information:

    • Size of the ping response packet (in bytes).

    • IP address of the host from which the response was sent.

    • Sequence number of the ping response packet. You can use this value to match the ping response to the corresponding ping request.

    • Time-to-live (ttl) hop-count value of the ping response packet.

    • Total time between the sending of the ping request packet and the receiving of the ping response packet, in milliseconds. This value is also called round-trip time.

    • Number of ping requests (probes) sent to the host.

    • Number of ping responses received from the host.

    • Packet loss percentage.

    • Round-trip time statistics: minimum, average, maximum, and standard deviation of the round-trip time.

Monitoring Traffic Through the Router or Switch

To help with the diagnosis of a problem, display real-time statistics about the traffic passing through physical interfaces on the router or switch.

To display real-time statistics about physical interfaces, perform these tasks:

Displaying Real-Time Statistics About All Interfaces on the Router or Switch

Purpose

Display real-time statistics about traffic passing through all interfaces on the router or switch.

Action

To display real-time statistics about traffic passing through all interfaces on the router or switch:

Sample Output
command-name

Meaning

The sample output displays traffic data for active interfaces and the amount that each field has changed since the command started or since the counters were cleared by using the C key. In this example, the monitor interface command has been running for 15 seconds since the command was issued or since the counters last returned to zero.

Displaying Real-Time Statistics About an Interface on the Router or Switch

Purpose

Display real-time statistics about traffic passing through an interface on the router or switch.

Action

To display traffic passing through an interface on the router or switch, use the following Junos OS CLI operational mode command:

Sample Output
command-name

Meaning

The sample output shows the input and output packets for a particular SONET interface (so-0/0/1). The information can include common interface failures, such as SONET/SDH and T3 alarms, loopbacks detected, and increases in framing errors. For more information, see Checklist for Tracking Error Conditions.

To control the output of the command while it is running, use the keys shown in Table 1.

Table 1: Output Control Keys for the monitor interface Command

Action

Key

Display information about the next interface. The monitor interface command scrolls through the physical or logical interfaces in the same order that they are displayed by the show interfaces terse command.

N

Display information about a different interface. The command prompts you for the name of a specific interface.

I

Freeze the display, halting the display of updated statistics.

F

Thaw the display, resuming the display of updated statistics.

T

Clear (zero) the current delta counters since monitor interface was started. It does not clear the accumulative counter.

C

Stop the monitor interface command.

Q

See the CLI Explorer for details on using match conditions with the monitor traffic command.

Dynamic Ternary Content Addressable Memory Overview

Understanding Dynamic Ternary Content Addressable Memory

In ACX Series routers, Ternary Content Addressable Memory (TCAM) is used by various applications like firewall, connectivity fault management, PTPoE, RFC 2544, etc. The Packet Forwarding Engine (PFE) in ACX Series routers uses TCAM with defined TCAM space limits. The allocation of TCAM resources for various filter applications are statically distributed. This static allocation leads to inefficient utilization of TCAM resources when all the filter applications might not use this TCAM resource simultaneously.

The dynamic allocation of TCAM space in ACX routers efficiently allocates the available TCAM resources for various filter applications. In the dynamic TCAM model, various filter applications (such as inet-firewall, bridge-firewall, cfm-filters, etc.) can optimally utilize the available TCAM resources as and when required. Dynamic TCAM resource allocation is usage driven and is dynamically allocated for filter applications on a need basis. When a filter application no longer uses the TCAM space, the resource is freed and available for use by other applications. This dynamic TCAM model caters to higher scale of TCAM resource utilization based on application’s demand.

Applications using Dynamic TCAM Infrastructure

The following filter application categories use the dynamic TCAM infrastructure:

  • Firewall filter—All the firewall configurations

  • Implicit filter—Routing Engine (RE) demons using filters to achieve its functionality. For example, connectivity fault management, IP MAC validation, etc.

  • Dynamic filters—Applications using filters to achieve the functionality at the PFE level. For example, logical interface level fixed classifier, RFC 2544, etc. RE demons will not know about these filters.

  • System-init filters—Filters that require entries at the system level or fixed set of entries at router's boot sequence. For example, Layer 2 and Layer 3 control protocol trap, default ARP policer, etc.

    Note:

    The System-init filter which has the applications for Layer 2 and Layer 3 control protocols trap is essential for the overall system functionality. The applications in this control group consume a fixed and minimal TCAM space from the overall TCAM space. The system-init filter will not use the dynamic TCAM infrastructure and will be created when the router is initialized during the boot sequence.

Features Using TCAM Resource

Applications using the TCAM resource is termed tcam-app in this document. For example, inet-firewall, bridge-firewall, connectivity fault management, link fault management, etc., are all different tcam-apps.

Table 2 describes the list of tcam-apps that use TCAM resources.

Table 2: Features Using TCAM Resource

TCAM Apps/TCAM Users

Feature/Functionality

TCAM Stage

bd-dtag-validate

Bridge domain dual-tagged validate

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Egress

bd-tpid-swap

Bridge domain vlan-map with swap tpid operation

Egress

cfm-bd-filter

Connectivity fault management implicit bridge-domain filters

Ingress

cfm-filter

Connectivity fault management implicit filters

Ingress

cfm-vpls-filter

Connectivity fault management implicit vpls filters

Note:

This feature is supported only on ACX5048 and ACX5096 routers.

Ingress

cfm-vpls-ifl-filter

Connectivity fault management implicit vpls logical interface filters

Note:

This feature is supported only on ACX5048 and ACX5096 routers.

Ingress

cos-fc

Logical interface level fixed classifier

Pre-ingress

fw-ccc-in

Circuit cross-connect family ingress firewall

Ingress

fw-family-out

Family level egress firewall

Egress

fw-fbf

Firewall filter-based forwarding

Pre-ingress

fw-fbf-inet6

Firewall filter-based forwarding for inet6 family

Pre-ingress

fw-ifl-in

Logical interface level ingress firewall

Ingress

fw-ifl-out

Logical interface level egress firewall

Egress

fw-inet-ftf

Inet family ingress firewall on a forwarding-table

Ingress

fw-inet6-ftf

Inet6 family ingress firewall on a forwarding-table

Ingress

fw-inet-in

Inet family ingress firewall

Ingress

fw-inet-rpf

Inet family ingress firewall on RPF fail check

Ingress

fw-inet6-in

Inet6 family ingress firewall

Ingress

fw-inet6-family-out

Inet6 Family level egress firewall

Egress

fw-inet6-rpf

Inet6 family ingress firewall on a RPF fail check

Ingress

fw-inet-pm

Inet family firewall with port-mirror action

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

fw-l2-in

Bridge family ingress firewall on Layer 2 interface

Ingress

fw-mpls-in

MPLS family ingress firewall

Ingress

fw-semantics

Firewall sharing semantics for CLI configured firewall

Pre-ingress

fw-vpls-in

VPLS family ingress firewall on VPLS interface

Ingress

ifd-src-mac-fil

Physical interface level source MAC filter

Pre-ingress

ifl-statistics-in

Logical level interface statistics at ingress

Ingress

ifl-statistics-out

Logical level interface statistics at egress

Egress

ing-out-iff

Ingress application on behalf of egress family filter for log and syslog

Ingress

ip-mac-val

IP MAC validation

Pre-ingress

ip-mac-val-bcast

IP MAC validation for broadcast

Pre-ingress

ipsec-reverse-fil

Reverse filters for IPsec service

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

irb-cos-rw

IRB CoS rewrite

Egress

lfm-802.3ah-in

Link fault management (IEEE 802.3ah) at ingress

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

lfm-802.3ah-out

Link fault management (IEEE 802.3ah) at egress

Egress

lo0-inet-fil

Looback interface inet filter

Ingress

lo0-inet6-fil

Looback interface inet6 filter

Ingress

mac-drop-cnt

Statistics for drops by MAC validate and source MAC filters

Ingress

mrouter-port-in

Multicast router port for snooping

Ingress

napt-reverse-fil

Reverse filters for network address port translation (NAPT) service

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

no-local-switching

Bridge no-local-switching

Ingress

ptpoe

Point-to-Point-Over-the-Ethernet traps

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

ptpoe-cos-rw

CoS rewrite for PTPoE

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Egress

rfc2544-layer2-in

RFC2544 for Layer 2 service at ingress

Pre-ingress

rfc2544-layer2-out

RFC2544 for Layer 2 service at egress

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Egress

service-filter-in

Service filter at ingress

Note:

This feature is not supported on ACX5048 and ACX5096 routers.

Ingress

Monitoring TCAM Resource Usage

You can use the show and clear commands to monitor and troubleshoot dynamic TCAM resource usage.

Table 3 summarizes the command-line interface (CLI) commands you can use to monitor and troubleshoot dynamic TCAM resource usage.

Table 3: Show and Clear Commands to Monitor and Troubleshoot Dynamic TCAM

Task

Command

Display the shared and the related applications for a particular application

show pfe tcam app

Display the TCAM resource usage for an application and stages (egress, ingress, and pre-ingress)

show pfe tcam usage

Display the TCAM resource usage errors for applications and stages (egress, ingress, and pre-ingress)

show pfe tcam errors

Clears the TCAM resource usage error statistics for applications and stages (egress, ingress, and pre-ingress)

clear pfe tcam-errors

Example: Monitoring and Troubleshooting the TCAM Resource

This section describes a use case where you can monitor and troubleshoot TCAM resources using show commands. In this use case scenario, you have configured Layer 2 services and the Layer 2 service-related applications are using TCAM resources. The dynamic approach, as shown in this example, gives you the complete flexibility to manage TCAM resources on a need basis.

The service requirement is as follows:

  • Each bridge domain has one UNI and one NNI interface

  • Each UNI interface has:

    • One logical interface level policer to police the traffic at 10 Mbps.

    • Multifield classifier with four terms to assign forwarding class and loss-priority.

  • Each UNI interface configures CFM UP MEP at the level 4.

  • Each NNI interface configures CFM DOWN MEP at the level 2

Let us consider a scenario where there are 100 services configured on the router. With this scale, all the applications are configured successfully and the status shows OK state.

  1. Viewing TCAM resource usage for all stages.

    To view the TCAM resource usage for all stages (egress, ingress, and pre-ingress), use the show pfe tcam usage all-tcam-stages detail command.

  2. Configure additional Layer 2 services on the router.

    For example, add 20 more services on the router, thereby increasing the total number of services to 120. After adding more services, you can check the status of the configuration by verifying either the syslog message using the command show log messages, or by running the show pfe tcam errors command.

    The following is a sample syslog message output showing the TCAM resource shortage for Ethernet-switching family filters for newer configurations by running the show log messages CLI command.

    If you use the show pfe tcam errors all-tcam-stages detail CLI command to verify the status of the configuration, the output will be as shown below:

    The output indicates that the fw-l2-in application is running out of TCAM resources and moves into a FAILED state. Although there are two TCAM slices available at the ingress stage, the fw-l2-in application is not able to use the available TCAM space due to its mode (DOUBLE), resulting in resource shortage failure.

  3. Fixing the applications that have failed due to the shortage of TCAM resouces.

    The fw-l2-in application failed because of adding more number of services on the routers, which resulted in shortage of TCAM resources. Although other applications seems to work fine, it is recommended to deactivate or remove the newly added services so that the fw-l2-in application moves to an OK state. After removing or deactivating the newly added services, you need to run the show pfe tcam usage and show pfe tcam error commands to verify that there are no more applications in failed state.

    To view the TCAM resource usage for all stages (egress, ingress, and pre-ingress), use the show pfe tcam usage all-tcam-stages detail command.

    To view TCAM resource usage errors for all stages (egress, ingress, and pre-ingress), use the show pfe tcam errors all-tcam-stages command.

    You can see that all the applications using the TCAM resources are in OK state and indicates that the hardware has been successfully configured.

Note:

As shown in the example, you will need to run the show pfe tcam errors and show pfe tcam usage commands at each step to ensure that your configurations are valid and that the applications using TCAM resource are in OK state.

Service Scaling on ACX5048 and ACX5096 Routers

In ACX5048 and ACX5096 routers, a typical service (such as ELINE, ELAN and IP VPN) that is deployed might require applications (such as policers, firewall filters, connectivity fault management IEEE 802.1ag, RFC2544) that uses the dynamic TCAM infrastructure.

Note:

Service applications that uses TCAM resources is limited by the TCAM resource availability. Therefore, the scale of the service depends upon the consumption of the TCAM resource by such applications.

A sample use case for monitoring and troubleshooting service scale in ACX5048 and ACX5096 routers can be found at the Dynamic Ternary Content Addressable Memory Overview section.

Understanding and Configuring the Unified Forwarding Table

Using the Unified Forwarding Table to Optimize Address Storage

ACX5048 and ACX5096 routers support the use of a unified forwarding table to optimize address storage. This feature gives you the flexibility to configure your router to match the needs of your particular network environment. You can control the allocation of forwarding table memory available to store the following entries:

  • MAC addresses

  • Layer 3 host entries

  • Longest prefix match (LPM) table entries

You can use five predefined profiles (l2-profile-one, l2-profile-two, l2-profile-three, l3-profile, lpm-profile) to allocate the table memory space differently for each of these entries. The sizes of the Layer 2 MAC address table, Layer 3 host entry table, and Layer 3 LPM table are decided based on the selected profile. You can configure and select the profiles that best suits your network environment needs.

Table 4 illustrates the predefined profiles in the unified forwarding table and the respective table sizes.

Table 4: Unified Forwarding Table Profiles

Profile

Layer 2 MAC Address Table

Layer 3 Host Table

Layer 3 LPM Table

l2-profile-one

288 K

16 K

16 K

l2-profile-two

224 K

80 K

16 K

l2-profile-three (default)

160 K

144 K

16 K

l3-profile

96 K

208 K

16 K

lpm-profile

32 K

16 K

128 K

IPv4 unicast, IPv6 unicast, IPv4 multicast, and IPv6 multicast route addresses share the Layer 3 host entry table. If the host table stores the maximum number of entries for any given type, the entire table is full and is unable to accommodate any entries of any other type. The IPv4 multicast and IPv6 unicast addresses occupy double the space as that occupied by IPv4 unicast entries, and IPv6 multicast addresses occupy four times the space of the IPv4 unicast addresses. Table 5 shows the Layer 3 host table size for each profile.

Table 5: Layer 3 Host Table

Profile

Layer 3 Host Table

IPv4 Unicast

IPv4 Multicast

IPv6 Unicast

IPv6 Multicast

l2-profile-one

16 K

8 K

8 K

4 K

l2-profile-two

80 K

40 K

40 K

20 K

l2-profile-three (default)

144 K

72 K

72 K

36 K

l3-profile

208 K

104 K

104 K

52 K

lpm-profile

16 K

8 K

8 K

4 K

The Layer 3 LPM table is shared between IPv4 route prefixes and IPv6 route prefixes. Table 6 illustrates the size of the table for different profiles of the IPv4 and IPv4 addresses in the Layer 3 LPM table. When unicast reverse-path forwarding (unicast RPF) is enabled, the table size reduces to half.

Table 6: Layer 3 LPM Table

Profile

Layer 3 LPM Table

IPv4 Unicast

IPv6 Unicast (Prefix <= /64)

IPv6 Unicast (Prefix > /64)

l2-profile-one

16 K

8 K

4 K

l2-profile-two

16 K

8 K

4 K

l2-profile-three (default)

16 K

8 K

4 K

l3-profile

16 K

8 K

4 K

lpm-profile

128 K

40 K

8 K

By default, there is no space allocated for IPv6 prefix address longer than /64 in the LPM table. Therefore, prefix address longer than /64 are not allowed in the table by default. The entire table is available for IPv4 addresses and for IPv6 addresses that have prefixes shorter than /64. You can provide space in the table for addresses with prefixes longer than /64 by using CLI configuration. The number of entries reserved for these prefixes is configured in multiples of 16.

Configuring the Unified Forwarding Table to Optimize Address Storage Using Profiles

You can use five predefined profiles (l2-profile-one, l2-profile-two, l2-profile-three, l3-profile, lpm-profile) to allocate the table memory space. The sizes of the Layer 2 MAC address table, Layer 3 host entry table, and Layer 3 LPM table are decided based on the selected profile. You can configure and select the profiles that best suits your network environment needs.

  1. To configure the profile that you want, enter the following statement:
  2. Commit the profile.
Note:

When you configure and commit a profile, the Packet Forwarding Engine (PFE) process restarts and all the data interfaces on the router go down and come back up.

The settings for l2-profile-three are configured by default. That is, if you do not configure the forwarding–options chassis profile-name statement, the l2-profile-three profile settings are configured.

Troubleshooting and Monitoring TCAM Resource in ACX Series Routers

The dynamic allocation of Ternary Content Addressable Memory (TCAM) space in ACX Series efficiently allocates the available TCAM resources for various filter applications. In the dynamic TCAM model, various filter applications (such as inet-firewall, bridge-firewall, cfm-filters, etc.) can optimally utilize the available TCAM resources as and when required. Dynamic TCAM resource allocation is usage driven and is dynamically allocated for filter applications on a need basis. When a filter application no longer uses the TCAM space, the resource is freed and available for use by other applications. This dynamic TCAM model caters to higher scale of TCAM resource utilization based on application’s demand. You can use the show and clear commands to monitor and troubleshoot dynamic TCAM resource usage in ACX Series routers.

Note:

Applications using the TCAM resource is termed tcam-app in this document.

Table 7 shows the task and the commands to monitor and troubleshoot TCAM resources in ACX Series routers

Table 7: Commands to Monitor and Troubleshoot TCAM Resource in ACX Series

How to

Command

View the shared and the related applications for a particular application.

show pfe tcam app (list-shared-apps | list-related-apps)

View the number of applications across all tcam stages.

show pfe tcam usage all-tcam-stages

View the number of applications using the TCAM resource at a specified stage.

show pfe tcam usage tcam-stage (ingress | egress | pre-egress)

View the TCAM resource used by an application in detail.

show pfe tcam usage app <application-name> detail

View the TCAM resource used by an application at a specified stage.

show pfe tcam usage tcam-stage (ingress | egress | pre-egress) app <application-name>

Know the number of TCAM resource consumed by a tcam-app

show pfe tcam usage app <application-name>

View the TCAM resource usage errors for all stages.

show pfe tcam errors all-tcam-stages detail

View the TCAM resource usage errors for a stage

show pfe tcam errors tcam-stage (ingress | egress | pre-egress)

View the TCAM resource usage errors for an application.

show pfe tcam errors app <application-name>

View the TCAM resource usage errors for an application along with its other shared application.

show pfe tcam errors app <application-name> shared-usage

Clear the TCAM resource usage error statistics for all stages.

clear pfe tcam-errors all-tcam-stages

Clear the TCAM resource usage error statistics for a specified stage

clear pfe tcam-errors tcam-stage (ingress | egress | pre-egress)

Clear the TCAM resource usage error statistics for an application.

clear pfe tcam-errors app <application-name>

To know more about dynamic TCAM in ACX Series, see Dynamic Ternary Content Addressable Memory Overview.