Monitoring SNMP Activity and Tracking Problems That Affect SNMP Performance on a Device Running Junos OS

This section contains information on monitoring the SNMP activity on devices running Junos OS and identifying problems that may impact the SNMP performance on devices running Junos OS.

For more information, see the following sections:

Checking for MIB Objects Registered with SNMPD

For the SNMP process to be able to access data related to a MIB object, the MIB object must be registered with snmpd. When an SNMP subagent comes up online, it tries to register the associated MIB objects with snmpd. Snmpd maintains a mapping of objects and the subagents with which the objects are associated. However, the registration attempt fails occasionally, and the objects remain unregistered with snmpd until the next time the subagent restarts and successfully registers the objects.

When a network management system polls for data related to objects that are not registered with snmpd, snmpd returns either a noSuchName error (for SNMPv1 objects) or a noSuchObject error (for SNMPv2 objects).

You can use the following commands to check for MIB objects that are registered with snmpd:

show snmp registered-objects—Creates /var/log/snmp_reg_objs file that contains the list of registered objects and their mapping to various subagents.
file show /var/log/snmp_reg_objs—Displays the content of /var/log/snmp_reg_objs file.

The following example shows the steps for creating and displaying the /var/log/snmp_reg_objs file:

user@host> show snmp registered-objects

user@host> file show /var/log/snmp_reg_objs

--------------------------------------------------------------
 Registered MIB Objects
 root_name =
--------------------------------------------------------------
.1.2.840.10006.300.43.1.1.1.1.2 (dot3adAggMACAddress) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.3 (dot3adAggActorSystemPriority) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.4 (dot3adAggActorSystemID) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.5 (dot3adAggAggregateOrIndividual) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.6 (dot3adAggActorAdminKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.7 (dot3adAggActorOperKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.8 (dot3adAggPartnerSystemID) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.9 (dot3adAggPartnerSystemPriority) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.10 (dot3adAggPartnerOperKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.1.1.11 (dot3adAggCollectorMaxDelay) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.1.2.1.1 (dot3adAggPortListPorts) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.2 (dot3adAggPortActorSystemPriority) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.3 (dot3adAggPortActorSystemID) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.4 (dot3adAggPortActorAdminKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.5 (dot3adAggPortActorOperKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.6 (dot3adAggPortPartnerAdminSystemPriority) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.7 (dot3adAggPortPartnerOperSystemPriority) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.8 (dot3adAggPortPartnerAdminSystemID) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.9 (dot3adAggPortPartnerOperSystemID) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.10 (dot3adAggPortPartnerAdminKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.11 (dot3adAggPortPartnerOperKey) (/var/run/mib2d-11)
.1.2.840.10006.300.43.1.2.1.1.12 (dot3adAggPortSelectedAggID) (/var/run/mib2d-11)
---(more)---

Note: The /var/log/snmp_reg_objs file contains only those objects that are associated with Junos OS processes that are up and running, and are registered with snmpd, at the time of executing the show snmp registered-objects command. If a MIB object related to a Junos OS process that is up and running is not shown in the list of registered objects, you may want to restart the software process to retry object registration with SNMPD.

Tracking SNMP Activity

SNMP tracing operations track activity of SNMP agents and record the information in log files. The logged event descriptions provide detailed information to help you solve problems faster. By default, Junos OS does not trace any SNMP activity. To enable tracking of SNMP activities on a device running Junos OS, include the traceoptions statement at the [edit snmp] hierarchy level.

A sample traceoptions configuration may look like:

[edit snmp]set traceoptions flag all;

When the traceoptions flag all statement is included at the [edit snmp] hierarchy level, the following log files are created:

snmpd
mib2d
rmopd

You can use the show log log-filename operational mode command to view the contents of the log file. In the snmpd log file (see the following example), a sequence of >>>> represents an incoming packet, whereas a sequence of <<<< represents an outgoing packet. Note that the request response pair may not follow any sequence if there are multiple network management systems polling the device at the same time. You can use the source and request ID combinations to match requests and responses. However, note that no response log is created in the log file if the SNMP agent, the master agent or the subagent, has not responded to a request.

A careful analysis of the request-response time may help you identify and understand delayed responses.

Reviewing a Log File

The following example shows the output for the show log snmpd command:

user@host> show log snmpd

Apr 12 06:40:03 snmpd[7ee783df] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Apr 12 06:40:03 snmpd[7ee783df] >>> Get-Bulk-Request
Apr 12 06:40:03 snmpd[7ee783df] >>>  Source:      10.209.63.42
Apr 12 06:40:03 snmpd[7ee783df] >>>  Destination: 10.209.2.242
Apr 12 06:40:03 snmpd[7ee783df] >>>  Version:     SNMPv2
Apr 12 06:40:03 snmpd[7ee783df] >>>  Request_id:  0x7ee783df
Apr 12 06:40:03 snmpd[7ee783df] >>>  Community:   public
Apr 12 06:40:03 snmpd[7ee783df] >>>  Non-repeaters:   0
Apr 12 06:40:03 snmpd[7ee783df] >>>  Max-repetitions: 10
Apr 12 06:40:03 snmpd[7ee783df] >>>   OID  : jnxContentsType.6.1.2.0
Apr 12 06:40:03 snmpd[7ee783df] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Apr 12 06:40:03 snmpd[7ee783df]  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Apr 12 06:40:03 snmpd[7ee783df]  <<< Get-Response
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Source:      10.209.63.42
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Destination: 10.209.2.242
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Version:     SNMPv2
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Request_id:  0x7ee783df
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Community:   public
Apr 12 06:40:03 snmpd[7ee783df]  <<<  Error:       status=0 / vb_index=0
Apr 12 06:40:03 snmpd[7ee783df]  <<<
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.7.1.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iFPC.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.7.1.1.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxChassisTempSensor.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.7.2.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iFPC.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.7.2.1.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxChassisTempSensor.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.9.1.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iRE.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.9.1.1.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxPCMCIACard.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.9.2.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iRE.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.9.2.1.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxPCMCIACard.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.12.1.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iHCM.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<    
Apr 12 06:40:03 snmpd[7ee783df]  <<<   OID  : jnxContentsType.12.2.0.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<   type : Object
Apr 12 06:40:03 snmpd[7ee783df]  <<<   value: jnxM10iHCM.0
Apr 12 06:40:03 snmpd[7ee783df]  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Apr 12 06:40:03 snmpd[7ee783e0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Apr 12 06:40:03 snmpd[7ee783e0] >>> Get-Bulk-Request
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Source:      10.209.63.42
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Destination: 10.209.2.242
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Version:     SNMPv2
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Request_id:  0x7ee783e0
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Community:   public
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Non-repeaters:   0
Apr 12 06:40:03 snmpd[7ee783e0] >>>  Max-repetitions: 10
Apr 12 06:40:03 snmpd[7ee783e0] >>>   OID  : jnxContentsType.12.2.0.0
Apr 12 06:40:03 snmpd[7ee783e0] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
……
……

Monitoring SNMP Statistics

The show snmp statistics extensive operational mode command provides you with an option to review SNMP traffic, including traps, on a device. Output for the show snmp statistics extensive command shows real-time values and can be used to monitor values such as throttle drops, currently active, max active, not found, time out, max latency, current queued, total queued, and overflows. You can identify slowness in SNMP responses by monitoring the currently active count as a constant increase in the currently active count is directly linked to slow or no response to SNMP requests.

Sample Output for the show snmp statistics extensive Command

user@host> show snmp statistics extensive

  SNMP statistics:
   Input:
     Packets: 226656, Bad versions: 0, Bad community names: 0,
     Bad community uses: 0, ASN parse errors: 0,
     Too bigs: 0, No such names: 0, Bad values: 0,
     Read onlys: 0, General errors: 0,
     Total request varbinds: 1967606, Total set varbinds: 0,
     Get requests: 18478, Get nexts: 75794, Set requests: 0,
     Get responses: 0, Traps: 0,
     Silent drops: 0, Proxy drops: 0, Commit pending drops: 0,
     Throttle drops: 27084, Duplicate request drops: 0
   V3 Input:
     Unknown security models: 0, Invalid messages: 0
     Unknown pdu handlers: 0, Unavailable contexts: 0
     Unknown contexts: 0, Unsupported security levels: 0
     Not in time windows: 0, Unknown user names: 0
     Unknown engine ids: 0, Wrong digests: 0, Decryption errors: 0
   Output:
     Packets: 226537, Too bigs: 0, No such names: 0,
     Bad values: 0, General errors: 0,
     Get requests: 0, Get nexts: 0, Set requests: 0,
     Get responses: 226155, Traps: 382
   SA Control Blocks:
     Total: 222984, Currently Active: 501, Max Active: 501,
     Not found: 0, Timed Out: 0, Max Latency: 25
   SA Registration:
     Registers: 0, Deregisters: 0, Removes: 0
   Trap Queue Stats:
     Current queued: 0, Total queued: 0, Discards: 0, Overflows: 0
   Trap Throttle Stats:
     Current throttled: 0, Throttles needed: 0
   Snmp Set Stats:
     Commit pending failures: 0, Config lock failures: 0
     Rpc failures: 0, Journal write failures: 0
     Mgd connect failures: 0, General commit failures: 0

Checking CPU Utilization

High CPU usage of the software processes that are being queried, such as snmpd or mib2d, is another factor that may lead to slow response or no response. You can use the show system processes extensive operational mode command to check the CPU usage levels of the Junos OS processes.

Sample Output of show system processes extensive Command

user@host> show system processes extensive

last pid:  1415;  load averages:  0.00,  0.00,  0.00  up 0+02:20:54    10:26:25
117 processes: 2 running, 98 sleeping, 17 waiting

Mem: 180M Active, 54M Inact, 39M Wired, 195M Cache, 69M Buf, 272M Free
Swap: 1536M Total, 1536M Free


  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root        1 171   52     0K    12K RUN    132:09 95.21% idle
 1184 root        1  97    0 35580K  9324K select   4:16  1.61% chassisd
  177 root        1  -8    0     0K    12K mdwait   0:51  0.00% md7
  119 root        1  -8    0     0K    12K mdwait   0:20  0.00% md4
   13 root        1 -20 -139     0K    12K WAIT     0:16  0.00% swi7: clock sio
 1373 root        1  96    0 15008K 12712K select   0:09  0.00% snmpd
 1371 root        1  96    0  9520K  5032K select   0:08  0.00% jdiameterd
   12 root        1 -40 -159     0K    12K WAIT     0:07  0.00% swi2: net
 1375 root        2  96    0 15016K  5812K select   0:06  0.00% pfed
   49 root        1  -8    0     0K    12K mdwait   0:05  0.00% md0
 1345 root        1  96    0 10088K  4480K select   0:05  0.00% l2ald
 1181 root        1  96    0  1608K   908K select   0:05  0.00% bslockd
   23 root        1 -68 -187     0K    12K WAIT     0:04  0.00% irq10: fxp1
   30 root        1 171   52     0K    12K pgzero   0:04  0.00% pagezero
 1344 root        1   4    0 39704K 11444K kqread   0:03  0.00% rpd
1205 root        1  96    0  3152K   912K select   0:03  0.00% license-check
 1372 root        1  96    0 28364K  6696K select   0:03  0.00% dcd
 1374 root        1  96    0 11764K  7632K select   0:02  0.00% mib2d
 1405 regress     1  96    0 15892K 11132K select   0:02  0.00% cli
  139 root        1  -8    0     0K    12K mdwait   0:02  0.00% md5
   22 root        1 -80 -199     0K    12K WAIT     0:02  0.00% irq9: cbb1 fxp0
 1185 root        1  96    0  4472K  2036K select   0:02  0.00% alarmd
    4 root        1  -8    0     0K    12K -        0:02  0.00% g_down
    3 root        1  -8    0     0K    12K -        0:02  0.00% g_up
   43 root        1 -16    0     0K    12K psleep   0:02  0.00% vmkmemdaemon
 1377 root        1  96    0  3776K  2256K select   0:01  0.00% irsd
   48 root        1 -16    0     0K    12K -        0:01  0.00% schedcpu
   99 root        1  -8    0     0K    12K mdwait   0:01  0.00% md3
  953 root        1  96    0  4168K  2428K select   0:01  0.00% eventd
 1364 root        1  96    0  4872K  2808K select   0:01  0.00% cfmd
   15 root        1 -16    0     0K    12K -        0:01  0.00% yarrow
 1350 root        1  96    0 31580K  7248K select   0:01  0.00% cosd
 1378 root        1  96    0 19776K  6292K select   0:01  0.00% lpdfd
---(more)---

Checking Kernel and Packet Forwarding Engine Response

As mentioned in Understanding SNMP Implementation in Junos OS , some of the SNMP MIB data are maintained by the kernel or Packet Forwarding Engine. For such data to be available for the network management system, the kernel has to provide the required information to the SNMP subagent in mib2d. A slow response from the kernel can cause a delay in mib2d returning the data to the network management system. The Junos OS adds an entry in the mib2d log file every time when an interface takes more than 10,000 microseconds to respond to a request for interface statistics. You can use the show log log-filename | grep “kernel response time” command to find out the response time taken by the kernel.

Checking the Kernel Response Time

user@host> show log mib2d | grep “kernel
response time”

 Aug 17 22:39:37  == kernel response time for
 COS_IPVPN_DEFAULT_OUTPUT-t1-7/3/0:10:27.0-o: 9.126471 sec, range
 (0.000007, 11.000806)
 
 Aug 17 22:39:53  == kernel response time for
 COS_IPVPN_DEFAULT_INPUT-t1-7/2/0:5:15.0-i: 5.387321 sec, range
 (0.000007, 11.000806)
  
 Aug 17 22:39:53  == kernel response time for ct1-6/1/0:9:15: 0.695406
 sec, range (0.000007, 11.000806)
  
 Aug 17 22:40:04  == kernel response time for t1-6/3/0:6:19: 1.878542
 sec, range (0.000007, 11.000806)
  
 Aug 17 22:40:22  == kernel response time for lsq-7/0/0: 2.556592 sec,
 range (0.000007, 11.000806)