Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Troubleshooting System Performance with Resource Monitoring Methodology

 

Resource Monitoring Usage Computation Overview

You can configure the resource monitoring capability using both the CLI and SNMP MIB queries. You can employ this utility to provision sufficient headroom (memory space limits that are set for the application or virtual router) for monitoring the health and operating efficiency of DPCs and MPCs. You can also analyze and view the usage or consumption of memory for the jtree memory type and for contiguous pages, double words, and free memory pages. The jtree memory on all MX Series router Packet Forwarding Engines has two segments: one segment primarily stores routing tables and related information, and the other segment primarily stores firewall-filter-related information. As the allocation of more memory for routing tables or firewall filters might disrupt the forwarding operations of a Packet Forwarding Engine, the Junos OS CLI displays a warning to restart all affected FPCs when you commit a configuration that includes the memory-enhanced route statement.

The following sections describe the computation equations and the interpretation of the different memory regions for I-chip-based and Trio-based line cards:

Resource Monitoring and Usage Computation For Trio-Based Line Cards

In Trio-based line cards, memory blocks for next-hop and firewall filters are allocated separately. Also, an expansion memory is present, which is used when the allocated memory for next-hop or firewall filter is fully consumed. Both next-hop and firewall filters can allocate memory from the expansion memory. The encapsulation memory region is specific to I-chip-based line cards and it is not applicable to Trio-based line cards. Therefore, for Trio-based line cards, the percentage of free memory space can be interpreted as follows:

% Free (NH) = (1- (Used NH memory + Used Expansion memory ) / (Total NH memory+Total Expansion memory)) × 100

% Free (Firewall or Filter) = (1-(Used FW memory+Used Expansion memory ) / (Total FW memory+Total Expansion memory)) × 100

Encapsulation memory is I-chip-specific and is not applicable for Trio-based line cards.

% Free (Encap memory) = Not applicable

Resource Monitoring and Usage Computation For I-Chip-Based Line Cards

I-chip-based line cards contain 32 MB of static RAM (SRAM) memory associated with the route lookup block and 16 MB of SRAM memory associated with the output WAN block.

The route-lookup memory is a single pool of 32 MB memory that is divided into two segments of 16 MB each. In a standard configuration, segment 0 is used for NH and prefixes, and segment 1 is used for firewall or filter. This allocation can be modified by using the route-memory-enhanced option at the [edit chassis] hierarchy level. In a general configuration, NH application can be allocated memory from any of the two segments. Therefore, the percentage of free memory for NH is calculated on 32 MB memory. Currently, firewall applications are allotted memory only from segment 1. As a result, the percentage of free memory to be monitored for firewall starts from the available 16 MB memory in segment 1 only.

For I-chip-based line cards, the percentage of free memory space can be interpreted as follows:

% Free (NH) = (32-(Used NH memory+Used FW memory+Used Other application)) / 32×100

% Free (Firewall or Filter)=(16-(Used NH memory+Used FW memory+Used Other application)) / 16×100

The memory size for Output WAN (Iwo) SRAM is 16 MB and stores the Layer 2 descriptors that contain the encapsulation information. This entity is a critical resource and needs to be monitored. This memory space is displayed in the output of the show command as “Encap mem”. The percentage of free memory for the encapsulation region is calculated as follows:

% Free (Encapsulation memory) = (16-(Iwo memory used ( L2 descriptors +other applications))) / 16×100

The watermark level configured for next-hop memory is also effective for encapsulation memory. Therefore, if the percentage of free memory for encapsulation region falls below the configured watermark, logs are generated.

If the free memory percentage is lower than the free memory watermark of a specific memory type, the following error message is recorded in the syslog:

“Resource Monitor: FPC <slot no> PFE <pfe inst> <“JNH memory” or “FW/ Filter memory”> is below set watermark <configured watermark>”.

You can configure resource-monitoring tracing operations by using the traceoptions file <filename> flag flag level level size bytes statement at the [edit system services resource-monitor] hierarchy level. By default, messages are written to /var/log/rsmonlog. The error logs associated with socket communication failure (between the Routing Engine and the Packet Forwarding Engine) are useful in diagnosing the problems in the communication between the Routing Engine and the Packet Forwarding Engine.

From the Ukern perspective, MPC5E contains only one Packet Forwarding Engine instance. The show chassis fabric plane command output displays the state of fabric plane connections to the Packet Forwarding Engine. Because two Packet Forwarding Engines exist, you notice PFE-0 and PFE-1 in the output.

Because only one Packet Forwarding Engine instance for MPC5E exists, the output of the show system resource-monitor fpc command displays only one row corresponding to Packet Forwarding Engine instance 0.

The configured watermark is retained across GRES and unified ISSU procedures.

Diagnosing and Debugging System Performance by Configuring Memory Resource Usage Monitoring on MX Series Routers

Junos OS supports a resource monitoring capability using both the CLI and SNMP MIB queries. You can employ this utility to provision sufficient headroom (memory space limits that are set for the application or virtual router) for ensuring system stability, especially the health and operating efficiency of I-chip-based line cards and Trio-based FPCs on MX Series routers. When the memory utilization, either the ukernel memory or ASIC memory reaches a certain threshold, the system operations compromise on the health and traffic-handling stability of the line card and such a trade-off on the system performance can be detrimental for supporting live traffic and protocols.

To configure the properties of the memory resource-utilization functionality:

  1. Specify that you want to configure the monitoring mechanism for utilization of different memory resource regions.

    This feature is enabled by default and you cannot disable it manually.

  2. Specify the high threshold value, exceeding which warnings or error logs are generated, for all the regions of memory, such as heap or ukernel, next-hop and encapsulation, and firewall filter memory.
  3. Specify the percentage of free memory space used for next-hops to be monitored with a watermark value.
  4. Specify the percentage of free memory space used for ukernel or heap memory to be monitored with a watermark value.
  5. Specify the percentage of free memory space used for firewall and filter memory to be monitored with a watermark value.
    Note

    The default value and the configured value of the watermark value for the percentage of free next-hop memory also applies to encapsulation memory. The default watermark values for the percentage of free ukernel or heap memory, next-hop memory, and firewall filter memory are 20 percent.

  6. Disable the generation of error log messages when the utilization of memory resources exceeds the threshold or checkpoint levels. By default, messages are written to /var/log/rsmonlog.
  7. Define the resource category that you want to monitor and analyze for ensuring system stability, especially the health and operating efficiency of I-chip-based line cards and Trio-based FPCs on MX Series routers. The resource category includes detailed CPU utilization, session rate, and session count statistics. You use the resource category statistics to understand the extent to which new attack objects or applications affect performance.
    Note

    The jtree memory on all MX Series router Packet Forwarding Engines has two segments: one segment primarily stores routing tables and related information, and the other segment primarily stores firewall-filter-related information. The Junos OS provides the memory-enhanced statement to reallocate the jtree memory for routes, firewall filters, and Layer 3 VPNs.

  8. Configure the type of resource as contiguous pages for which you want to enable the monitoring mechanism to provide sufficient headroom for ensuring effective system performance and traffic-handling capacity. Specify the high and low threshold value, exceeding which warnings or error logs are generated, for the specified type or region of memory, which is contiguous page in this case.
  9. Configure the type of resource as free double words (dwords) for which you want to enable the monitoring mechanism to provide sufficient headroom for ensuring effective system performance and traffic-handling capacity. Specify the high and low threshold value, exceeding which warnings or error logs are generated, for the specified type or region of memory, which is free dwords in this case.
  10. Configure the type of resource as free memory pages for which you want to enable the monitoring mechanism to provide sufficient headroom for ensuring effective system performance and traffic-handling capacity. Specify the high and low threshold value, exceeding which warnings or error logs are generated, for the specified type or region of memory, which is free memory pages in this case.
  11. View the utilization of memory resources on the Packet Forwarding Engines of an FPC by using the show system resource-monitor fpc command. The filter memory denotes the filter counter memory used for firewall filter counters. The asterisk (*) displayed next to each of the memory regions denotes the ones for which the configured threshold is being currently exceeded.

Troubleshooting the Mismatch of jnxNatObjects Values for MS-DPC and MS-MIC

Problem

Description: When both MS-DPC and MS-MIC are deployed in a network and the Network Address Translation (NAT) type is configured as napt-44, the output of the snmp mib walk command for jnxNatObjects displays different values for MS-DPC and MS-MIC.

Resolution

Configure SNMP to Match jnxNatObjects Values for MS-DPC and MS-MIC

To configure SNMP to match jnxNatObjects values for MS-DPC and MS-MIC:

  1. Run the set services service-set service-set-name nat-options snmp-value-match-msmic configuration mode command. The following configuration example shows how to configure SNMP to match the values for MS-MIC-specific objects in the jnxNatObjects MIB table with the values for MS-DPC objects.
  2. Issue the commit command to confirm the changes.
  3. (Optional) Run the show snmp mib walk jnxNatObjects command to verify that the values for MS-MIC-specific objects in the jnxNatObjects MIB table match the values for MS-DPC objects. For example, the following output shows that the values for MS-MIC-specific objects and MS-DPC objects match.
    Note

    You can use the delete services service-set service-set-name nat-options snmp-value-match-msmic configuration mode command to disable this feature.

Managed Objects for Ukernel Memory for a Packet Forwarding Engine in an FPC Slot

The jnxPfeMemoryUkernTable, whose object identifier is {jnxPfeMemory 1}, contains the JnxPfeMemoryUkernEntry that retrieves the global ukernel or heap memory statistics for the specified Packet Forwarding Engine slot. Each JnxPfeMemoryUkernEntry, whose object identifier is {jnxPfeMemoryUkernTable 1}, contains the objects listed in the following table. The jnxPfeMemoryUkernEntry denotes the memory utilization, such as the total available memory and the percentage of memory used.

Table 1: jnxPfeMemoryUKernTable

Object

Object ID

Description

jnxPfeMemoryUkernFreePercent

jnxPfeMemoryUkernEntry 3

Denotes the percentage of free Packet Forwarding Engine memory within the ukern heap.

Managed Objects for Packet Forwarding Engine Memory Statistics Data

The jnxPfeMemory table, whose object identifier is {jnxPfeMib 2} contains the objects listed in Table 2

Table 2: jnxPfeMemory Table

Object

Object ID

Description

jnxPfeMemoryUkernTable

jnxPfeMemory 1

Provides global ukern memory statistics for the specified Packet Forwarding Engine slot.

jnxPfeMemoryForwardingTable

jnxPfeMemory 2

Provides global next-hop (for Trio-based line cards) or Jtree (for I-chip-based line cards) memory utilization and firewall filter memory utilization statistics for the specified Packet Forwarding Engine slot.

Managed Objects for Next-Hop, Jtree, and Firewall Filter Memory for a Packet Forwarding Engine in an FPC Slot

The jnxPfeMemoryForwardingTable, whose object identifier is {jnxPfeMemory 2}, contains JnxPfeMemoryForwardingEntry that retrieves the next-hop memory for Trio- based line cards, jtree memory for I-chip-based line cards, and firewall or filter memory statistics for the specified Packet Forwarding Engine slot for both I- chip and Trio-based line cards. Each jnxPfeMemoryForwardingEntry, whose object identifier is {jnxPfeMemoryForwardingTable 1}, contains the objects listed in the following table.

The jnxPfeMemoryForwardingEntry represents the ASIC instance, ASIC memory used, and ASIC free memory. The jtree memory on all MX Series router Packet Forwarding Engines has two segments: one segment primarily stores routing tables and related information, and the other segment primarily stores firewall-filter-related information. As the allocation of more memory for routing tables or firewall filters might disrupt the forwarding operations of a Packet Forwarding Engine, the Junos OS CLI displays a warning to restart all affected FPCs when you commit a configuration that includes the memory-enhanced route statement. The configuration does not become effective until you restart the FPC or DPC (on MX Series routers).

Table 3: jnxPfeMemoryForwardingTable

Object

Object ID

Description

jnxPfeMemoryForwardingChipSlot

jnxPfeMemoryForwardingEntry 1

Indicates the ASIC instance number in the Packet Forwarding Engine complex.

jnxPfeMemoryType

jnxPfeMemoryForwardingEntry 2

Indicates the Packet Forwarding Engine memory type, where nh = 1, fw = 2, encap = 3.

jnxPfeMemoryForwardingPercentFree

jnxPfeMemoryForwardingEntry 3

Indicates the percentage of memory free for each memory type.

jnxPfeMemoryErrorsTable

The Juniper Networks enterprise-specific Packet Forwarding Engine MIB, whose object ID is {jnxPfeMibRoot 1}, supports a new MIB table, jnxPfeMemoryErrorsTable, to display Packet Forwarding Engine memory error counters. The jnxPfeMemoryErrorsTable, whose object identifier is jnxPfeNotification 3, contains the JnxPfeMemoryErrorsEntry. Each JnxPfeMemoryErrorsEntry, whose object identifier is { jnxPfeMemoryErrorsTable 1 }, contains the objects listed in the following table.

Table 4: jnxPfeMemoryErrorsTable

Object

Object ID

Description

jnxPfeFpcSlot

jnxPfeMemoryErrorsEntry 1

Signifies the FPC slot number for this set of PFE notification

jnxPfeSlot

jnxPfeMemoryErrorsEntry 2

Signifies the PFE slot number for this set of errors

jnxPfeParityErrors

jnxPfeMemoryErrorsEntry 3

Signifies the parity error count

jnxPfeEccErrors

jnxPfeMemoryErrorsEntry 4

Signifies the error-checking code (ECC) error count

pfeMemoryErrors

The pfeMemoryErrorsNotificationPrefix, whose object identifier is {jnxPfeNotification 0}, contains the pfeMemoryErrors attribute. The pfeMemoryErrors object, whose identifier is {pfeMemoryErrorsNotificationPrefix 1} contains the jnxPfeParityErrors and jnxPfeEccErrors objects.

Table 5: pfeMemoryErrors

Object

Object ID

Description

pfeMemoryErrors

pfeMemoryErrorsNotificationPrefix 1

A pfeMemoryErrors notification is sent when the value of jnxPfeParityErrors or jnxPfeEccErrors increases.